Three-dimensional object reconstruction method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of computers, and provides a three-dimensional object reconstruction method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: determining a three-dimensional point cloud, camera parameters and a mask image according to a multi-view image of a to-be-reconstructed object; performing neural radiation field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image to obtain a signed distance function of the object to be reconstructed; performing three-dimensional model segmentation based on the signed distance function to generate texture coordinates of a grid model corresponding to the to-be-reconstructed object; performing linear interpolation based on the texture coordinates to obtain a texture map of the to-be-reconstructed object; and constructing a three-dimensional model of the to-be-reconstructed object based on the grid model and the texture map. According to the method, even an object model with a texture map of a complex structure can be well reconstructed, so that the quality of a three-dimensional object reconstructed for a robot simulation environment can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a three-dimensional object reconstruction method and device, electronic equipment, storage medium and computer program product. BACKGROUND

[0002] With the rapid development of robot technology, robots are increasingly widely used in industrial production, home service, medical assistance and other fields. Robot simulation environment provides a risk-free and controllable virtual space to simulate the behavior and interaction of robots in the real world, which can be used to train the perception, decision-making and control capabilities of robots in a virtual environment, thereby improving the intelligent level of robots in the real environment. In the robot simulation environment, constructing realistic scene data, including the three-dimensional model of the object, can reduce the difference between the virtual environment and the real environment, and help to migrate the key algorithms of robot navigation, identification and operation constructed in the simulation environment to the real world (Simulation to Reality, Sim2Real) at low cost.

[0003] Traditional three-dimensional object reconstruction methods use a multi-view camera array system to collect multi-view data of an object, and then reconstruct the geometric information and texture information of the object. However, multi-view reconstruction usually includes more than a dozen separate steps, each of which is complex, resulting in a long reconstruction process and poor robustness. Figure Three Neural Radiance Field (NeRF) is currently the most popular three-dimensional reconstruction method, which is gradually replacing traditional multi-view reconstruction in various application scenarios. NeRF uses an implicit neural network to represent a three-dimensional object, and both the expression and the rendering are differentiable, converting the entire three-dimensional reconstruction process into a neural network optimization problem, making the reconstruction process highly integrated and automated, thereby improving robustness. Figure Three

[0004] However, neural radiance field can only generate object models with simple and repetitive texture maps, such as grass, tree bark, and solid-colored carpets. For object models with complex texture maps, such as musical instruments, food, fruits, and furniture, it cannot be well reconstructed, resulting in low quality of three-dimensional objects reconstructed for robot simulation environments. SUMMARY

[0005] The present application aims to at least solve one of the technical problems in the related art. To this end, the present application proposes a three-dimensional object reconstruction method and device, electronic equipment, storage medium and computer program product to solve the problem that neural radiance field cannot well reconstruct object models with complex texture maps, and to improve the quality of three-dimensional objects reconstructed for robot simulation environments. ​

[0006] The three-dimensional object reconstruction method according to an embodiment of the first aspect of the present application comprises: According to the multi-view images of the object to be reconstructed, a three-dimensional point cloud, camera parameters and a mask image are determined respectively; Based on the three-dimensional point cloud, the camera parameters and the mask image, neural radiance field three-dimensional reconstruction is performed to obtain a signed distance function of the object to be reconstructed; Based on the signed distance function, three-dimensional model segmentation is performed to generate texture coordinates of a mesh model corresponding to the object to be reconstructed; Based on the texture coordinates, linear interpolation is performed to obtain a texture map of the object to be reconstructed; Based on the mesh model and the texture map, a three-dimensional model of the object to be reconstructed is constructed.

[0007] According to an embodiment of the present application, the neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image to obtain the signed distance function of the object to be reconstructed comprises: The three-dimensional point cloud, the camera parameters and the mask image are input into a surface reconstruction model to obtain a signed distance function of the object to be reconstructed output by the surface reconstruction model; wherein the surface reconstruction model is obtained by neural surface reconstruction network training based on sample three-dimensional point clouds, sample camera parameters and sample mask images.

[0008] According to an embodiment of the present application, the three-dimensional model segmentation based on the signed distance function to generate the texture coordinates of the mesh model corresponding to the object to be reconstructed comprises: Based on the signed distance function, a mesh model of the object to be reconstructed is determined; The mesh model is subjected to mesh segmentation to obtain at least two mesh model blocks; Each mesh model block is subjected to two-dimensional parameterization to obtain texture coordinates of the corresponding mesh model block.

[0009] According to an embodiment of the present application, the determination of the mesh model of the object to be reconstructed based on the signed distance function comprises: The mesh model implicitly stored in the signed distance function is converted to an explicit form; Based on the camera parameters, coordinate mapping is performed on the explicit mesh model to obtain an initial mesh model; Each vertex in the initial mesh model is traversed, and a ray is projected along the normal direction of each vertex to the signed distance function field to obtain a color value corresponding to the vertex, thereby obtaining a mesh model with color of the object to be reconstructed.

[0010] According to one embodiment of the present application, the linear interpolation based on the texture coordinates comprises: According to each of the texture coordinates, a color value is assigned to each two-dimensional vertex corresponding to each three-dimensional vertex of the mesh model of the object to be reconstructed in each mesh model patch, to obtain a corresponding mesh model patch with colorized vertices; According to each of the texture coordinates, a color value is assigned to each two-dimensional vertex corresponding to each three-dimensional vertex of the mesh model of the object to be reconstructed in each mesh model patch, to obtain a corresponding mesh model patch with colorized vertices; Based on each of the mesh model patches with colorized domains, the texture map of the object to be reconstructed is determined.

[0011] According to one embodiment of the present application, the texture map of the object to be reconstructed is determined based on each of the mesh model patches with colorized domains, comprising: Each of the mesh model patches with colorized domains is subjected to an expansion operation and a corrosion operation in sequence to obtain a corresponding first mesh model patch; Each of the first mesh model patches is subjected to color value clustering to obtain a corresponding second mesh model patch; According to each of the second mesh model patches, the texture map of the object to be reconstructed is constructed.

[0012] According to the third aspect of the present application, an electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the three-dimensional object reconstruction method as described above. The determination module is configured to determine a three-dimensional point cloud, camera parameters, and a mask image based on multi-view images of an object to be reconstructed; The reconstruction module is configured to perform neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters, and the mask image to obtain a signed distance function of the object to be reconstructed; The segmentation module is configured to perform three-dimensional model segmentation based on the signed distance function to generate texture coordinates of a mesh model corresponding to the object to be reconstructed; The interpolation module is configured to perform linear interpolation based on the texture coordinates to obtain a texture map of the object to be reconstructed; The construction module is configured to construct a three-dimensional model of the object to be reconstructed based on the mesh model and the texture map.

[0013] According to the third aspect of the present application, an electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the three-dimensional object reconstruction method as described above.

[0014] The storage medium according to the fourth aspect of the present application is a non-transitory computer-readable storage medium, and a computer program is stored on the storage medium. The computer program is executed by a processor to implement the three-dimensional object reconstruction method according to any one of the above aspects.

[0015] The computer program product according to the fifth aspect of the present application comprises a computer program. The computer program is executed by a processor to implement the three-dimensional object reconstruction method according to any one of the above aspects.

[0016] The one or more technical solutions described above in the embodiments of the present application have at least the following technical effects: After the three-dimensional point cloud, the camera parameter, and the mask image are determined based on the multi-view images of the object to be reconstructed, the neural radiance field three-dimensional reconstruction is performed based on the three-dimensional point cloud, the camera parameter, and the mask image, so that the signed distance function containing more realistic image information of the object to be reconstructed is obtained. Then, the three-dimensional model segmentation is performed based on the signed distance function, so that the distortion and deformation of the model partition are reduced to generate the texture coordinates of the grid model corresponding to the object to be reconstructed. Further, the linear interpolation is performed based on the texture coordinates to realize the color value allocation, so that the texture map of the object to be reconstructed with accurate color mapping is obtained. Thus, the three-dimensional model of the object to be reconstructed with more reality can be constructed based on the grid model and the texture map. Based on the above process, even the object model with a complex structure of the texture map can be well reconstructed, and thus the quality of the three-dimensional object for the robot simulation environment reconstruction can be improved.

[0017] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a flowchart of the three-dimensional object reconstruction method provided by the embodiments of the present application.

[0020] Figure 2 is a data preprocessing flowchart of the three-dimensional object reconstruction method provided by the embodiments of the present application.

[0021] Figure 3 is a texture coordinate generation flowchart of the three-dimensional object reconstruction method provided by the embodiments of the present application.

[0022] Figure 4 is a texture mapping generation flowchart in a three-dimensional object reconstruction method provided by an embodiment of the present application.

[0023] Figure 5 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application will be further described below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0025] In the description of the embodiments of the present application, it should be noted that the terms "center", "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the embodiments of the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0026] In the description of the embodiments of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0027] In the embodiments of the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature "above", "above" and "above" the second feature can be that the first feature is directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "below" and "below" the second feature can be that the first feature is directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0028] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0029] The present application provides a three-dimensional object reconstruction method, device, electronic equipment, storage medium and computer program product.

[0030] Figure 1 is a flowchart of a three-dimensional object reconstruction method provided by an embodiment of the present application, as shown in Figure 1 The three-dimensional object reconstruction method comprises: Step 110, determining a three-dimensional point cloud, camera parameters and a mask image according to a multi-view image of the object to be reconstructed.

[0031] Step 120, performing neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, camera parameters and mask image to obtain a signed distance function of the object to be reconstructed.

[0032] Step 130, performing three-dimensional model segmentation based on the signed distance function to generate texture coordinates of a mesh model corresponding to the object to be reconstructed.

[0033] Step 140, performing linear interpolation based on the texture coordinates to obtain a texture map of the object to be reconstructed.

[0034] Step 150, constructing a three-dimensional model of the object to be reconstructed based on the mesh model and the texture map.

[0035] It should be noted that the three-dimensional object reconstruction method provided by the embodiments of the present application can be applied to the scene of three-dimensional object reconstruction, more specifically, can be used for three-dimensional object texture reconstruction for robot simulation environment, and the execution subject can be a server, a computer device, etc., such as a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an Ultra-mobile Personal Computer (UMPC), a netbook or a Personal Digital Assistant (PDA) and the like.

[0036] The server or computer device of the present application can be provided with or connected to a three-dimensional object reconstruction device, so as to control the three-dimensional object reconstruction device to perform the three-dimensional object reconstruction method of the present application.

[0037] The present application can obtain images of an object needing three-dimensional reconstruction under multiple viewing angles, thereby obtaining multi-view images of the object to be reconstructed. The number of specific viewing angles is not limited, for example, it can be 2-30 viewing angles, and the viewing angles should be uniformly distributed around the object to cover all angles.

[0038] After obtaining the multi-view images, data preprocessing can be performed on the multi-view images. In the data preprocessing process, a sparse three-dimensional point cloud can be generated from the multi-view images of the object using a classic three-dimensional reconstruction algorithm library colmap. Further, the user is prompted to manually delete noise points in the sparse three-dimensional point cloud that are not in the region of interest using a three-dimensional model editing software. Further, a camera parameter file containing camera parameters is generated from the remaining three-dimensional point cloud using colmap. The generation of the camera parameter file containing camera parameters from the remaining three-dimensional point cloud using colmap can use traditional techniques or other feasible techniques, which are not specifically limited in the present application.

[0039] The colmap is a general, end-to-end image-based three-dimensional reconstruction algorithm library.

[0040] In addition, a mask image is generated for each image in the multi-view images according to a color threshold.

[0041] Figure 2 is a data preprocessing flowchart of the three-dimensional object reconstruction method provided by the present application, as shown in Figure 2 The color threshold is used for image segmentation to generate a mask for each image. These masks are used to provide a supervision signal in the subsequent network training of the neural radiance field three-dimensional reconstruction algorithm Neus to ensure that the model can accurately learn the features of the target object. Then, the colmap library is used to sequentially perform feature detection, feature extraction, feature matching, geometric verification, and sparse reconstruction, thereby generating a sparse point cloud and a camera parameter file. In this process, the default configuration of colmap is used, and the reconstruction quality is set to "high".

[0042] The generated sparse point cloud may contain noise points, which can mislead the network, leading to inaccurate geometric fitting and decreased convergence speed. In order to solve this problem, a three-dimensional model editing software is used to remove noise points in the non-interest region.

[0043] Further, the application can utilize the high-precision neural radiance field three-dimensional reconstruction algorithm NeuS to perform neural radiance field three-dimensional reconstruction according to the sparse point cloud obtained in the data preprocessing step, the camera parameters, and each mask image, and reconstruct a signed distance function (SDF) of the object to be reconstructed.

[0044] Further, since NeuS stores the mesh model Mesh of the object to be reconstructed in an implicit manner, the application can convert the Mesh stored in the SDF in an implicit manner into an explicit Mesh.

[0045] Further, the color values of each vertex in the explicit Mesh can be obtained to form a colored Mesh, and the colored Mesh is further subjected to mesh segmentation to obtain a plurality of mesh model patches chart. The two-dimensional texture coordinates are then generated by performing two-dimensional parameterization on each chart.

[0046] Further, after obtaining the plurality of mesh model patches, the color values (which can be RGB values, RGB representing red (Red), green (Green), and blue (Blue)) of the two-dimensional points corresponding to each three-dimensional vertex of the mesh model in each mesh model patch can be assigned according to the corresponding texture coordinates of each mesh model patch. In addition, according to the corresponding texture coordinates of each mesh model patch, the triangles in the texture space of the corresponding mesh model patch are rasterized by performing linear interpolation, so as to assign color values to each pixel point inside the triangle.

[0047] Through each mesh model patch after assigning the color values, the texture map of the object to be reconstructed can be obtained.

[0048] Further, the three-dimensional model of the object to be reconstructed can be constructed by the texture map and the mesh model.

[0049] Specifically, the application can store the mesh model Mesh as an obj format file, store the texture map as an mtl and png format file, standardize the three-dimensional object according to the related files, generate a collision body model and a visual model, and finally import it into the simulation platform by using the interface program provided by the simulation platform. The obj format, the mtl format, and the png format are commonly used file formats in three-dimensional graphics and image processing.

[0050] According to the three-dimensional object reconstruction method of the embodiment of the application, after the three-dimensional point cloud, the camera parameters and the mask image are determined according to the multi-view images of the object to be reconstructed, the signed distance function containing more real image information of the object to be reconstructed can be obtained by performing neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image; then, the three-dimensional model segmentation is performed based on the signed distance function, so as to reduce the distortion and deformation of the model segmentation, and to generate the texture coordinates of the grid model corresponding to the object to be reconstructed; further, the linear interpolation is performed based on the texture coordinates, the color value allocation is realized, and the texture map of the object to be reconstructed with accurate color mapping is obtained; thus, the three-dimensional model of the object to be reconstructed can be constructed based on the grid model and the texture map. Based on the above process, even the object model with a complex texture map can be well reconstructed, and thus the quality of the three-dimensional object facing the simulation environment reconstruction of the robot can be improved.

[0051] According to the above embodiment, the signed distance function of the object to be reconstructed is obtained by performing neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image, which includes: The three-dimensional point cloud, the camera parameters and the mask image are input into the surface reconstruction model, and the signed distance function of the object to be reconstructed output by the surface reconstruction model is obtained; wherein the surface reconstruction model is obtained by training the neural surface reconstruction network based on the sample three-dimensional point cloud, the sample camera parameters and the sample mask image.

[0052] When the neural radiance field three-dimensional reconstruction is performed based on the three-dimensional point cloud, the camera parameters and the mask image, the NeuS network is adopted in the application, the core technology of which is to use SDF to represent the object, to generate the image of each view through volume rendering, to compare the input real multi-view image, and to optimize the color loss function and the mask loss function of each view, so as to significantly improve the quality of the generated Mesh. When training, the default parameters provided by the NeuS algorithm are adopted.

[0053] Specifically, the workflow of the NeuS network includes several steps. First, a light ray is defined: r(t)=o+td, Where o represents the origin of the light ray, that is, the position where the light ray starts. t is a parameter, the value range of which determines the propagation range of the light ray in space. When t=0, the light ray is located at the origin o; as t increases, the light ray continuously propagates along the direction vector. It can be understood as the distance of the light ray from the origin. d represents the direction vector of the light ray.

[0054] A plurality of sampling points are generated on the light ray.

[0055] Further, the color and volume density of each sampling point are queried by implicit neural field. The final color observable along the ray path is calculated by volume rendering formula. Finally, gradient descent algorithm is executed to optimize neural radiance field by comparing the difference between rendering result and actual picture.

[0056] Therefore, based on the above, the neural surface reconstruction network is trained by the sample three-dimensional point cloud, the sample camera parameter and the sample mask image, and the surface reconstruction model can be obtained.

[0057] In the training process, the key parameter settings include: the number of rays is 512, the number of iterations is 20w times, and the learning rate is 5e -4 The total loss of the network is composed of RGB loss , mask loss , SDF loss , and the definitions are as follows: ; Among them, and respectively represent the weight of the corresponding loss.

[0058] The color loss is defined as follows: ; Among them, is the number of rays, is the RGB color of each pixel point of the ground truth image, is the RGB color of each pixel point of the rendered image. is loss function.

[0059] The SDF loss is defined as follows: ; Among them, is the number of sampling points, is the Eikonal term, represents the L2 norm of the gradient of the function at the i-th sampling point in the k-th view.

[0060] The mask loss is defined as follows: ; Among them, BCE is binary cross entropy loss, is the pixel value of the ground truth mask image, is the sum of weights along the camera ray, represents the pixel value of the sampling point along the camera ray of the k-th view, a weight representing a sample point pixel value of a camera ray along the kth view, k represents the kth view, that is, the index of the view.

[0061] Therefore, inputting the three-dimensional point cloud, the camera parameters and the mask image into the surface reconstruction model can obtain the signed distance function of the object to be reconstructed output by the surface reconstruction model.

[0062] Based on the neural radiance field technology, the real texture features of the object can be efficiently obtained, and a more realistic three-dimensional model of the object to be reconstructed can be constructed according to the obtained information, which helps to improve the quality of the three-dimensional object facing the simulation environment reconstruction of the robot.

[0063] Based on the above embodiment, the three-dimensional model is segmented based on the signed distance function to generate the texture coordinates of the mesh model corresponding to the object to be reconstructed, which includes: determining the mesh model of the object to be reconstructed based on the signed distance function; performing mesh segmentation on the mesh model to obtain at least two mesh model blocks; performing two-dimensional parameterization on each mesh model block to obtain the texture coordinates of the corresponding mesh model block.

[0064] Figure 3 is a texture coordinate generation flowchart provided in the three-dimensional object reconstruction method of the embodiment of the present application, as shown in Figure 3 When the mesh model of the object to be reconstructed is determined based on the signed distance function, the mesh model implicitly stored in the signed distance function can be converted to be explicit, and the explicit mesh model is further mapped based on the camera parameters to obtain an initial mesh model; then each vertex in the initial mesh model is traversed, and a ray is projected along the normal direction of each vertex to the signed distance function field to obtain the color value of the corresponding vertex, and a mesh model with color of the object to be reconstructed is obtained.

[0065] Further, the mesh model with color is segmented into a plurality of blocks chart to obtain a plurality of mesh model blocks charts. Specifically, by analyzing the geometric shape of the model, natural gaps or areas that are not easy to observe are identified for cutting to minimize the visual seams. In order to achieve the above effect, a mesh segmentation algorithm is used to effectively decompose the three-dimensional surface into smaller and more manageable pieces. These pieces are called charts, which are carefully designed during the cutting process to ensure as little distortion and deformation as possible. After cutting, a flattening algorithm is used to flatten these cut mesh pieces into two-dimensional space to form charts.

[0066] The specific number and manner of segmentation can be set according to actual needs, which is not limited in the present application.

[0067] Further, each chart is individually two-dimensionally parameterized (also referred to as 2D space parameterization), generating two-dimensional texture coordinates. Finally, the charts are selected and combined onto a texture map. The parameterization step ensures maximum utilization of texture space and reduces distortion and waste by optimizing the arrangement and adjusting the positions of the charts.

[0068] Ultimately, the charts are selected and reorganized onto a fixed-size (e.g., 4096*4096) texture map, ensuring efficient utilization and accurate mapping of the texture. The charts are selected to ensure that the generated texture is optimal in terms of quality, mapping accuracy, and rendering efficiency. By selecting, distortion, resource waste, and rendering performance degradation caused by overly complex or inefficient charts can be avoided. Selection can be achieved through some ready-made algorithms, such as area-based selection, quality evaluation based on similarity metrics, and so on.

[0069] The generation step of the texture coordinates in this application can be completed using the classic open-source texture coordinate generation library xatlas.

[0070] According to the real texture characteristics of the object obtained based on the neural radiation field technology, the three-dimensional model is segmented to generate the texture coordinates of the corresponding mesh model of the object to be reconstructed. The texture characteristics obtained based on the neural radiation field technology can provide more accurate basis for segmentation, so that the segmentation result is more consistent with the real structure of the object. The segmented model part can be more convenient for generating texture coordinates and texture mapping, which helps to improve the quality of the three-dimensional object facing the robot simulation environment reconstruction.

[0071] Further, the mesh model of the object to be reconstructed is determined based on the signed distance function, comprising: Converting the mesh model implicitly stored in the signed distance function to an explicit mesh; Performing coordinate mapping on the explicit mesh model based on camera parameters to obtain an initial mesh model; Traversing each vertex in the initial mesh model, and projecting a ray along the normal direction of each vertex to the signed distance function field to obtain the color value of the corresponding vertex, thereby obtaining a colored mesh model of the object to be reconstructed.

[0072] Specifically, in this application, because NeuS stores Mesh in an implicit way (SDF), it can be converted to an explicit Mesh using the classic Marching Cubes algorithm. This algorithm constructs a cubic grid and gradually approximates the implicit surface to form a clearly defined geometry. The resolution parameter of the Marching Cubes algorithm can be set to 512. The Marching Cubes algorithm is a classic algorithm for extracting isosurfaces from three-dimensional discrete scalar fields.

[0073] Secondly, according to the camera parameters obtained from the multi-view reconstruction process, the extracted explicit Mesh is rotated, translated and scaled to ensure that the Mesh is correctly mapped from the local coordinate system to the world coordinate system.

[0074] In addition, in order to enhance the detail performance of the three-dimensional model, the application can perform a Loop mesh subdivision process on the Mesh. This algorithm effectively increases the number of vertices and smoothness of the mesh by adding new vertices in each triangular edge and adjusting the positions of the original vertices, so that the Mesh can better capture subtle geometric changes, thereby laying the foundation for more accurate color mapping. Loop subdivision is a geometric processing method for subdividing triangular meshes. Its main purpose is to increase the resolution of the mesh, making the mesh more fine, so that it can better approximate complex geometric shapes or be used for high-quality rendering.

[0075] Finally, by traversing each vertex in the refined Mesh and projecting a ray along the normal direction of each vertex towards the SDF field, the corresponding RGB color value is obtained, and the colored mesh model of the object to be reconstructed is obtained.

[0076] The texture features obtained by the neural radiance field technology can provide more accurate basis for segmentation, so that the grid model of the object to be reconstructed determined based on the signed distance function can improve the accuracy of the grid model of the object to be reconstructed, and thus help to improve the quality of the three-dimensional object facing the robot simulation environment reconstruction.

[0077] In one embodiment, the texture map of the object to be reconstructed is obtained by linear interpolation based on the texture coordinates, including: According to each texture coordinate, a color value is assigned to each two-dimensional vertex corresponding to each three-dimensional vertex of the grid model of the object to be reconstructed in each grid model block, to obtain a corresponding vertex-colored grid model block; According to each texture coordinate, a color value is assigned to each two-dimensional vertex corresponding to each three-dimensional vertex of the grid model of the object to be reconstructed in each grid model block, to obtain a corresponding vertex-colored grid model block; Based on the globally colored grid model blocks, the texture map of the object to be reconstructed is determined.

[0078] Specifically, the application can assign an RGB color value to each two-dimensional vertex corresponding to each three-dimensional vertex of the Mesh in the charts according to the texture coordinates of the charts in the Mesh, to obtain a corresponding vertex-colored grid model block.

[0079] Further, each triangle in the texture space corresponding to the texture coordinates of the charts is rasterized, which can be specifically implemented by calculating the RGB value of each pixel point inside the triangle through linear interpolation. Specifically, for each triangle, the two-dimensional coordinates of its vertices in the texture space are determined, and linear interpolation is performed to assign a color value to each pixel point inside the triangle, thereby obtaining a corresponding mesh model patch with color in the whole domain.

[0080] Further, the texture map of the object to be reconstructed is determined based on the mesh model patches with color in the whole domain.

[0081] The present application performs linear interpolation according to the texture coordinates to obtain the texture map of the object to be reconstructed, which can more smoothly transition the texture on the model surface, reduce obvious seams and boundaries, make the texture map more natural in vision, and effectively allocate the texture on different levels of detail, so that the high-detail area can use higher-resolution texture, while the low-detail area can use lower-resolution texture, and also help maintain the continuity of the texture between different parts of the model, avoid the visual abruptness caused by uneven texture resolution or sampling, and help improve the quality of the three-dimensional object reconstructed for the robot simulation environment.

[0082] Further, based on the mesh model patches with color in the whole domain, the texture map of the object to be reconstructed is determined, including: The mesh model patches with color in the whole domain are respectively subjected to an expansion operation and a corrosion operation in sequence to obtain corresponding first mesh model patches. The color value of each first mesh model patch is clustered to obtain a corresponding second mesh model patch. The texture map of the object to be reconstructed is constructed according to the second mesh model patches.

[0083] It should be noted that linear interpolation ensures that the colors of most areas can be correctly mapped, but since the triangle does not cover the entire texture image, some pixel points may not have colors.

[0084] In order to fill in these missing pixel points, expansion and corrosion operations are adopted. First, 32 expansion operations are performed to expand the area with existing colors, so that the colors can cover more pixel points. Then, 3 corrosion operations are performed to refine and correct the over-expansion possibly introduced in the expansion process. For a small number of pixel points that still have no color, the K nearest neighbor (KNN) algorithm is used to query the nearest pixel point with color, and the pixel points are assigned the corresponding color value. Finally, the texture map of the object to be reconstructed is constructed through the mesh model patches after the above operations.

[0085] The application ensures that the finally generated texture map has complete color coverage and high-precision detail performance, and helps to improve the quality of the three-dimensional object facing the simulation environment reconstruction of the robot.

[0086] Figure 4 is a texture map generation process schematic diagram in the three-dimensional object reconstruction method provided by the application, as Figure 4 shown, in the production process of the texture map, rasterization of the triangular patch can be performed first, which can be implemented by pixel interpolation, but linear interpolation only ensures that the colors of most areas can be correctly mapped, since the triangle does not cover the entire texture image, there may still be some pixel points without color. In order to fill in these missing pixel points, inflation and corrosion selection operations are adopted. For a small number of pixel points still without color, the KNN algorithm is used to query the nearest pixel point with color, and the pixel points are assigned corresponding color values. Finally, the texture map of the object to be reconstructed is constructed by each mesh model block after the above operations.

[0087] The three-dimensional object reconstruction device provided by the application is described below, and the three-dimensional object reconstruction device described below can be correspondingly referred to the three-dimensional object reconstruction method described above.

[0088] Further, the application also provides a three-dimensional object reconstruction device.

[0089] The three-dimensional object reconstruction device comprises: A determination module is configured to determine a three-dimensional point cloud, camera parameters and a mask image respectively according to multi-view images of an object to be reconstructed; A reconstruction module is configured to perform neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image to obtain a signed distance function of the object to be reconstructed; A segmentation module is configured to perform three-dimensional model segmentation based on the signed distance function to generate texture coordinates of a mesh model corresponding to the object to be reconstructed; An interpolation module is configured to perform linear interpolation based on the texture coordinates to obtain a texture map of the object to be reconstructed; A construction module is configured to construct a three-dimensional model of the object to be reconstructed based on the mesh model and the texture map.

[0090] The three-dimensional object reconstruction device of the present application, after determining the three-dimensional point cloud, camera parameters and mask image according to the multi-view image of the object to be reconstructed, performing neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, camera parameters and mask image, can obtain a signed distance function containing more realistic image information of the object to be reconstructed; then performing three-dimensional model segmentation based on the signed distance function can reduce the distortion and deformation of the model blocks to generate the texture coordinates of the corresponding grid model of the object to be reconstructed; further performing linear interpolation based on the texture coordinates can realize color value allocation and obtain the texture map of the object to be reconstructed with accurate color mapping; thus, a more realistic three-dimensional model of the object to be reconstructed can be constructed based on the grid model and the texture map. Based on the above process, even the object model with complex texture map can be well reconstructed, so the quality of the three-dimensional object facing the simulation environment reconstruction of the robot can be improved.

[0091] In one embodiment, the reconstruction module is specifically used for: inputting the three-dimensional point cloud, the camera parameters and the mask image into a surface reconstruction model to obtain a signed distance function of the object to be reconstructed output by the surface reconstruction model; wherein the surface reconstruction model is obtained by neural surface reconstruction network training based on sample three-dimensional point cloud, sample camera parameters and sample mask image.

[0092] In one embodiment, the segmentation module is specifically used for: determining a grid model of the object to be reconstructed based on the signed distance function; performing grid segmentation on the grid model to obtain at least two grid model blocks; performing two-dimensional parameterization on each of the grid model blocks to obtain the texture coordinates of the corresponding grid model block.

[0093] In one embodiment, the segmentation module comprises a first determination unit, and the first determination unit is used for: converting the grid model implicitly stored in the signed distance function to an explicit form; performing coordinate mapping on the explicit grid model based on the camera parameters to obtain an initial grid model; traversing each vertex in the initial grid model and projecting a light ray along the normal direction of each vertex to the signed distance function field to obtain the color value of the corresponding vertex, thereby obtaining a colored grid model of the object to be reconstructed.

[0094] In one embodiment, the interpolation module is specifically used for: assigning a color value to each two-dimensional vertex in each grid model block corresponding to each three-dimensional vertex of the grid model of the object to be reconstructed according to each of the texture coordinates, thereby obtaining a corresponding grid model block with vertices colored. Based on the texture coordinates, linear interpolation is performed on each mesh model block with colored vertices to obtain the corresponding mesh model block with colored global domain. The texture map of the object to be reconstructed is determined based on the color-coded mesh model blocks across the entire domain.

[0095] In one embodiment, the interpolation module includes a second determining unit, the second determining unit being configured to: Dilation and erosion operations are performed sequentially on each colored mesh model block to obtain the corresponding first mesh model block; Cluster the color values ​​of each first grid model block to obtain the corresponding second grid model block; The texture map of the object to be reconstructed is constructed based on each second mesh model block.

[0096] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the following method: determining the 3D point cloud, camera parameters, and mask image based on the multi-view images of the object to be reconstructed; Based on the three-dimensional point cloud, the camera parameters, and the mask image, a three-dimensional reconstruction of the neural radiation field is performed to obtain the signed distance function of the object to be reconstructed. The three-dimensional model is segmented based on the signed distance function to generate the texture coordinates of the mesh model corresponding to the object to be reconstructed; Linear interpolation is performed based on the texture coordinates to obtain the texture map of the object to be reconstructed; Based on the mesh model and the texture map, a three-dimensional model of the object to be reconstructed is constructed.

[0097] Further, the logic instructions in the memory 530 described above can be implemented in the form of software functional units and sold or used as independent products, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the related art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0098] In another aspect, the embodiments of the present application also provide a non-transitory computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, a method provided by any of the above embodiments is implemented, for example, including: determining a three-dimensional point cloud, camera parameters and a mask image respectively according to multi-view images of an object to be reconstructed; performing neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image to obtain a signed distance function of the object to be reconstructed; performing three-dimensional model segmentation based on the signed distance function to generate a texture coordinate of a mesh model corresponding to the object to be reconstructed; performing linear interpolation based on the texture coordinate to obtain a texture map of the object to be reconstructed; constructing a three-dimensional model of the object to be reconstructed based on the mesh model and the texture map.

[0099] In another aspect, the embodiments of the present application also provide a computer program product, which stores a computer program. When the computer program is executed by a processor, a method provided by any of the above embodiments is implemented, for example, including: determining a three-dimensional point cloud, camera parameters and a mask image respectively according to multi-view images of an object to be reconstructed; performing neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image to obtain a signed distance function of the object to be reconstructed; performing three-dimensional model segmentation based on the signed distance function to generate a texture coordinate of a mesh model corresponding to the object to be reconstructed; performing linear interpolation based on the texture coordinate to obtain a texture map of the object to be reconstructed; constructing a three-dimensional model of the object to be reconstructed based on the mesh model and the texture map.

[0100] The apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0101] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the related art, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the present application, and not to limit the present application. Although the present application is described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications or equivalent replacements of the technical solutions of the present application do not deviate from the spirit and scope of the present application.

Claims

1. A method of reconstructing a three-dimensional object, characterized by, The method comprises the following steps: determining a three-dimensional point cloud, camera parameters and a mask image based on multi-view images of an object to be reconstructed; performing neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image to obtain a signed distance function of the object to be reconstructed; performing three-dimensional model segmentation based on the signed distance function to generate texture coordinates of a mesh model corresponding to the object to be reconstructed; performing linear interpolation based on the texture coordinates to obtain a texture map of the object to be reconstructed; constructing a three-dimensional model of the object to be reconstructed based on the mesh model and the texture map.

2. The method of claim 1, wherein, The method further comprises the following steps: inputting the three-dimensional point cloud, the camera parameters and the mask image into a surface reconstruction model to obtain a signed distance function of the object to be reconstructed output by the surface reconstruction model; wherein the surface reconstruction model is obtained by training a neural surface reconstruction network based on sample three-dimensional point clouds, sample camera parameters and sample mask images.

3. The method of claim 1, wherein, The method further comprises the following steps: determining a mesh model of the object to be reconstructed based on the signed distance function; performing mesh segmentation on the mesh model to obtain at least two mesh model patches; performing two-dimensional parameterization on each mesh model patch to obtain texture coordinates of the corresponding mesh model patch.

4. The method of claim 3, wherein, The method further comprises the following steps: converting an implicitly stored mesh model in the signed distance function to an explicit mesh model; performing coordinate mapping on the explicit mesh model based on the camera parameters to obtain an initial mesh model; traversing each vertex in the initial mesh model and projecting a ray along the normal direction of each vertex towards the signed distance function field to obtain a color value of the corresponding vertex, thereby obtaining a mesh model with color for the object to be reconstructed.

5. The method of claim 3, wherein, The method further comprises the following steps: assigning a color value to a two-dimensional vertex corresponding to each three-dimensional vertex of the mesh model of the object to be reconstructed in each mesh model patch based on each texture coordinate, thereby obtaining a corresponding mesh model patch with color on the vertices; performing linear interpolation on each mesh model patch with color on the vertices based on each texture coordinate, thereby obtaining a corresponding mesh model patch with color on the entire domain; determining a texture map of the object to be reconstructed based on each mesh model patch with color on the entire domain.

6. The method of claim 5, wherein, The method further comprises the following steps: performing an expansion operation and a corrosion operation on each mesh model patch with color on the entire domain in sequence to obtain a corresponding first mesh model patch; performing color value clustering on each first mesh model patch to obtain a corresponding second mesh model patch; constructing a texture map of the object to be reconstructed based on each second mesh model patch.

7. A three-dimensional object reconstruction apparatus, characterized by comprising: The method comprises the following steps: A determining module is configured to determine a three-dimensional point cloud, camera parameters and a mask image respectively according to multi-view images of an object to be reconstructed; A reconstructing module is configured to perform neural radiance field three-dimensional reconstruction based on the three-dimensional point cloud, the camera parameters and the mask image to obtain a signed distance function of the object to be reconstructed; A segmenting module is configured to perform three-dimensional model segmentation based on the signed distance function to generate texture coordinates of a mesh model corresponding to the object to be reconstructed; An interpolating module is configured to perform linear interpolation based on the texture coordinates to obtain a texture map of the object to be reconstructed; A constructing module is configured to construct a three-dimensional model of the object to be reconstructed based on the mesh model and the texture map.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the three-dimensional object reconstruction method of any one of claims 1-6.

9. A storage medium, which is a non-transitory computer-readable storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the three-dimensional object reconstruction method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the three-dimensional object reconstruction method of any one of claims 1-6.

Images