Three-dimensional modeling method and apparatus, electronic device and storage medium

Abstract

Embodiments of the present disclosure provide a three-dimensional modeling method and apparatus, an electronic device and a storage medium. The method comprises: acquiring a three-dimensional truncated signed distance field of an object, the three-dimensional truncated signed distance field comprising a plurality of voxels, a first voxel among the plurality of voxels having a truncated signed distance value, and the truncated signed distance value representing the distance between the voxel and the surface of a three-dimensional model of the object; for a second voxel, of which a truncated signed distance value is missing, among the plurality of voxels, on the basis of the three-dimensional truncated signed distance field, coordinates of the second voxel in the three-dimensional truncated signed distance field, and a distance determination model used for determining a truncated signed distance value, determining the truncated signed distance value of the second voxel; and on the basis of the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel, establishing the three-dimensional model of the object.

Description

Three-dimensional modeling method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 24, 2023, with application number 202310912430.X and invention name “Three-dimensional modeling method, device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to a three-dimensional modeling method, device, electronic device, and storage medium. Background Art

[0003] Volumetric video is a technology that can capture information in three-dimensional space, such as depth and color, and generate a three-dimensional model sequence. The generated three-dimensional model sequence can be connected to form a new video format that can be viewed from any perspective.

[0004] In the scenario of real-time transmission of volumetric video, it is necessary to generate and transmit a 3D model of each frame in the volumetric video in real time. However, the existing 3D model generation solutions used in this scenario suffer from low accuracy.

[0005] Summary of the Invention

[0006] Embodiments of the present disclosure provide a three-dimensional modeling method, apparatus, electronic device, and storage medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a three-dimensional modeling method, comprising:

[0008] Acquire a truncated signed distance field volume of the object; the truncated signed distance field volume is used to represent the physical space where the object is located; the truncated signed distance field volume includes a plurality of voxels; the voxels are used to represent a spatial region in the physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0009] determining, for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, a truncated signed distance value for the second voxel based on the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0010] A three-dimensional model of the object is established based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0011] In a second aspect, an embodiment of the present disclosure provides a three-dimensional modeling device, comprising:

[0012] A stereo acquisition unit is configured to acquire a truncated signed distance field stereo of the object; the truncated signed distance field stereo is configured to represent a physical space in which the object is located; the truncated signed distance field stereo comprises a plurality of voxels; the voxels are configured to represent a spatial region in the physical space in which the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0013] a distance determination unit configured to determine, for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, a truncated signed distance value of the second voxel based on the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0014] The model building unit is configured to build a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In one or more embodiments of the present disclosure, a truncated signed distance field stereo of the object is first obtained, where the truncated signed distance field stereo is used to represent the physical space where the object is located. The truncated signed distance field stereo includes a plurality of voxels, where the voxels are used to represent a spatial region in the physical space where the object is located. A first voxel among the plurality of voxels has a truncated signed distance value, where the truncated signed distance value represents the distance between the voxel and the surface of a three-dimensional model of the object. Then, for a second voxel among the plurality of voxels for which the truncated signed distance value is missing, a model is determined based on the truncated signed distance field stereo of the object, the coordinates of the second voxel in the truncated signed distance field stereo, and the distance used to determine the truncated signed distance value, and the truncated signed distance value of the second voxel is determined. Finally, a three-dimensional model of the object is established based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate one or more embodiments of the present disclosure or technical solutions in the prior art, the following briefly introduces the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments described in the present disclosure. Those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0019] FIG1 is a schematic diagram of a flow chart of a three-dimensional modeling method provided by an embodiment of the present disclosure;

[0020] FIG2 is a schematic diagram of the structure of a distance determination model provided by an embodiment of the present disclosure;

[0021] FIG3 is a schematic structural diagram of a three-dimensional modeling device provided in one embodiment of the present disclosure;

[0022] FIG4 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0024] The embodiments of the present disclosure provide a three-dimensional modeling method that can be applied in a real-time three-dimensional model generation scenario to generate a high-precision three-dimensional model.

[0025] The real-time 3D model generation scenario can be a scenario in which volumetric video is generated and transmitted in real time. To facilitate understanding of this embodiment, volumetric video is first introduced. Volumetric video includes multiple video frames, each of which contains a 3D model. The 3D models in each video frame constitute a 3D model sequence. Playing a volumetric video can be understood as playing this 3D model sequence.

[0026] The 3D model in a volumetric video is a model of a specific object, such as a person, that moves over time. Multiple cameras are placed at different locations around the object. During the object's motion, the multiple cameras capture the object at each moment. Each camera captures an image of the object from its position at each moment, effectively recording the object's motion using multiple cameras from different positions. At each moment, a 3D model is constructed for the object based on the multiple images captured by the multiple cameras. This 3D model can be understood as a model of the object at that moment. The 3D models of the object at each moment constitute a 3D model sequence, which can be used to generate a volumetric video.

[0027] Scenarios for real-time volumetric video generation and transmission can include live streaming of volumetric video. For example, a user watches live streaming of volumetric video through a terminal device. A server generates a 3D model for each frame in real time and sends it to the terminal device. The terminal device receives the 3D model sent by the server, renders each frame of the volumetric video based on the 3D model, and displays the video image of each frame. Each frame of the volumetric video includes a 3D model. Users can watch live streaming of volumetric video through VR (Virtual Reality) devices, or through mobile phones, tablets, in-vehicle computers, wearable devices, and the like.

[0028] In summary, the real-time generation and transmission of volumetric video requires the real-time generation and transmission of a 3D model. The 3D modeling method provided by the disclosed embodiments enables the real-time generation of high-precision 3D models, meeting the requirements for both the real-time generation and transmission of volumetric video.

[0029] FIG1 is a flow chart of a three-dimensional modeling method provided by an embodiment of the present disclosure. The method can be applied to and executed by a server. As shown in FIG1 , the flow includes:

[0030] Step S102: Acquire a truncated signed distance field (SDF) volume of the object; the truncated signed distance field (SDF) volume is used to represent the physical space where the object is located; the truncated signed distance field (SDF) volume includes a plurality of voxels; a voxel is used to represent a spatial region in the physical space where the object is located; a first voxel among the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0031] Step S104 , for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, determining the truncated signed distance value of the second voxel according to the truncated signed distance field volume of the object, the coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0032] Step S106 : Building a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0033] In this embodiment, a truncated signed distance field stereo of the object is first obtained. The truncated signed distance field stereo is used to represent the physical space where the object is located. The truncated signed distance field stereo includes multiple voxels. The voxel is used to represent a spatial area in the physical space where the object is located. A first voxel among the multiple voxels has a truncated signed distance value. The truncated signed distance value represents the distance between the voxel and the surface of the three-dimensional model of the object. Then, for a second voxel among the multiple voxels for which the truncated signed distance value is missing, a truncated signed distance value of the second voxel is determined based on the truncated signed distance field stereo of the object, the coordinates of the second voxel in the truncated signed distance field stereo, and a distance determination model used to determine the truncated signed distance value. Finally, a three-dimensional model of the object is established based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel. It can be seen that through this embodiment, the truncated signed distance value of the second voxel can be quickly and accurately determined through the distance determination model, and a three-dimensional model of the object can be established based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel. Therefore, this embodiment can be applied in real-time three-dimensional model generation scenarios, and can generate high-precision three-dimensional models efficiently and quickly.

[0034] The method in Figure 1 is described in detail below.

[0035] In this embodiment, the object may be an object to be modeled, including but not limited to people, animals, virtual objects, and the like. When modeling the object to be modeled (hereinafter referred to as the object), the object is placed in a physical space, such as a studio of a specific size. RGB (red-green-blue) images of the object are captured from different perspectives within the physical space, and grayscale images are generated based on the captured RGB images. After obtaining the grayscale images of the object at each perspective, a truncated signed distance field stereo image of the object can be generated.

[0036] The truncated signed distance field stereo is a 3D stereo grid with truncated signed distance values. The stereo grid includes multiple voxels, each of which has a truncated signed distance value. The truncated signed distance value represents the distance from the voxel to the model surface.

[0037] Based on this, in this embodiment, the truncated signed distance field (TSDF) volume of an object is a 3D mesh with truncated signed distance values. This mesh represents the physical space in which the object resides. The mesh includes multiple voxels, each of which corresponds to a spatial region in the physical space in which the object resides and represents that spatial region. Alternatively, the mesh and voxels represent the physical space in which the object resides and each spatial region in that physical space within a computer system. In the truncated signed distance field (TSDF) volume of the object, each voxel has a truncated signed distance value representing the distance from the voxel to the surface of the object's three-dimensional model.

[0038] The truncated signed distance value is calculated by the truncated signed distance function (TSDF). TSDF is a method for calculating hidden potential surfaces in 3D reconstruction. TSDF divides the space into regular voxels and stores a truncated signed distance value for each voxel. This truncated signed distance value represents the distance from the voxel to the model surface. The truncated signed distance value of the voxel inside the model surface is negative, and the truncated signed distance value of the voxel outside the model surface is positive. In one embodiment, the maximum and minimum values ​​of the truncated signed distance value are preset. When the calculated distance from the voxel to the model surface is greater than the maximum value, the truncated signed distance value is taken as the maximum value. When the calculated distance from the voxel to the model surface is less than the minimum value, the truncated signed distance value is taken as the minimum value.

[0039] In one embodiment, obtaining a truncated signed distance field stereo of an object includes:

[0040] Projecting a three-dimensional grid representing the physical space where the object is located onto each view of the object; the three-dimensional grid is divided into a plurality of voxels;

[0041] Determine the truncated signed distance value of the voxel according to the depth data of the voxel at each viewing angle obtained by projection and the depth map of the object at each viewing angle;

[0042] The voxel that obtains the truncated signed distance value is determined as the first voxel, and the three-dimensional grid that obtains the truncated signed distance value after projection is determined as the truncated signed distance field three-dimensional grid.

[0043] First, a three-dimensional mesh representing the physical space in which the object resides is obtained and divided into individual voxels. Next, the three-dimensional mesh is projected onto each viewpoint of the object, which is the viewpoint used to capture and model the object. After projecting the three-dimensional mesh onto each viewpoint, depth data for the voxels in the three-dimensional mesh at each viewpoint is obtained. Next, a truncated signed distance value for the voxel is determined based on the projected depth data for the voxel at each viewpoint and the depth map of the object at each viewpoint. For example, for the same voxel, the truncated signed distance value for the voxel is calculated based on the projected depth value for the voxel at each viewpoint and the depth value recorded in the depth map for each viewpoint. It will be understood that for voxels that can be seen from any viewpoint, the depth value of the voxel at that viewpoint can be projected, and thus the truncated signed distance value for the voxel can be calculated. If a voxel is not visible from any viewpoint, its depth value at any viewpoint cannot be projected, and thus the truncated signed distance value for the voxel cannot be calculated.

[0044] Therefore, when calculating the truncated signed distance value of a voxel, the truncated signed distance value of the first voxel in all voxels can be calculated, but there are still some voxels whose truncated signed distance values ​​cannot be calculated, and these voxels are called second voxels.

[0045] Based on this, in this embodiment, a plurality of voxels that obtain truncated signed distance values ​​are determined as first voxels, and the projected three-dimensional grid that records the truncated signed distance values ​​is determined as a truncated signed distance field volume.

[0046] It can be seen that through this embodiment, the three-dimensional grid used to represent the physical space where the object is located is projected to each viewing angle of the object. Based on the depth data of the voxels at each viewing angle obtained by the projection and the depth map of the object at each viewing angle, the truncated signed distance value of the first voxel is determined and a truncated signed distance field stereo is generated. The method of generating the truncated signed distance field stereo is efficient and accurate.

[0047] As previously described, the multiple voxels included in the truncated signed distance field volume of the object are divided into first voxels and second voxels. The first voxels are voxels for which truncated signed distance values ​​are recorded in the truncated signed distance field volume of the object, and the second voxels are voxels for which truncated signed distance values ​​are not determined during the generation of the truncated signed distance field volume. Based on this, in step S104, for the second voxel in the multiple voxels included in the truncated signed distance field volume of the object for which a truncated signed distance value is missing, the truncated signed distance value of the second voxel is determined based on the truncated signed distance field volume of the object, the coordinates of the second voxel in the truncated signed distance field volume, and the distance determination model used to determine truncated signed distance values.

[0048] In one embodiment, a truncated signed distance field volume includes multiple levels; each level includes multiple voxel units; a voxel unit includes at least one voxel; a voxel unit of a lower level is nested within a voxel unit of a higher level; and determining a truncated signed distance value of the second voxel based on the truncated signed distance field volume of the object, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value includes:

[0049] For a first voxel unit at each level that is occupied by the three-dimensional model of the object and has a missing truncated signed distance value, determine the truncated signed distance value of the first voxel unit according to the truncated signed distance field volume of the object, the coordinates of the first voxel unit in the truncated signed distance field volume, and the distance determination model;

[0050] The first voxel unit includes the second voxel; the coordinates of the first voxel unit in the truncated signed distance field stereo are the coordinates of the included second voxel; and the truncated signed distance value of the first voxel unit is the truncated signed distance value of the included second voxel.

[0051] In this embodiment, the truncated signed distance field is divided into multiple levels, each level includes multiple voxel units, each voxel unit includes at least one voxel, the lowest level voxel unit includes one voxel, and the lower level voxel units are nested within the higher level voxel units. For example, the first level voxel unit is the highest level voxel unit, and the first level voxel unit can be divided into 8 second level voxel units, and the second level voxel units are lower level voxel units, so that the lower level voxel units are nested within the higher level voxel units. How to achieve the division of voxel units at each level will be described later.

[0052] In this embodiment, after the truncated signed distance field volume is divided into multiple levels and the voxel units included in each level are determined, the occupancy value, coordinates, and truncated signed distance value of each voxel unit within each level are determined. As previously described, each voxel unit in each level includes at least one voxel. Therefore, the occupancy value of each voxel unit in each level is the occupancy value of a specific voxel included in the voxel unit. An occupancy value of 1 indicates that the voxel unit is occupied by the 3D model of the object, that is, the voxel unit is located inside or on the surface of the 3D model of the object. An occupancy value of 0 indicates that the voxel unit is not occupied by the 3D model of the object, that is, the voxel unit is located outside the 3D model of the object. The coordinates of the voxel unit in each level in the truncated signed distance field volume are the coordinates of a specific voxel included in the voxel unit in the truncated signed distance field volume. The truncated signed distance value of the voxel unit in each level is the truncated signed distance value of a specific voxel included in the voxel unit. The specific voxel mentioned above can be a voxel located in a corner of the voxel unit, such as the lower left corner or the upper right corner. Therefore, when a particular voxel within a voxel cell lacks a truncated signed distance value, the voxel cell also lacks a truncated signed distance value.

[0053] Based on the above, the following operations are performed in each level: a first voxel unit occupied by the three-dimensional model of the object and missing a truncated signed distance value is identified in each level, for example, a first voxel unit whose occupancy value is 0 and whose truncated signed distance value is missing is searched in each level, and a truncated signed distance value of the first voxel unit is determined based on a truncated signed distance field volume of the object, the coordinates of the first voxel unit in the truncated signed distance field volume, and a distance determination model, wherein since the truncated signed distance field of the first voxel unit is missing, the specific voxel in the first voxel unit must be the second voxel, and therefore the first voxel unit includes the second voxel, the coordinates of the first voxel unit in the truncated signed distance field volume are the coordinates of the included specific voxel, i.e., the second voxel, and the truncated signed distance value of the first voxel unit is the truncated signed distance value of the included specific voxel, i.e., the second voxel.

[0054] It can be seen that through this embodiment, when the truncated signed distance field stereo includes multiple levels and each level includes multiple voxel units, the first voxel unit of each level that is occupied by the three-dimensional model of the object and has a missing truncated signed distance value can be determined according to the truncated signed distance field stereo of the object, the coordinates of the first voxel unit in the truncated signed distance field stereo and the distance determination model, so that the truncated signed distance value of the first voxel unit can be determined efficiently and quickly through the distance determination model, and the determination result has high accuracy.

[0055] In one embodiment, the above method further includes:

[0056] determining, based on the truncated signed distance value of the voxel unit of the Nth level of the truncated signed distance field volume of the object and the truncated signed distance values ​​of the voxel units adjacent to the voxel unit of the Nth level, whether the voxel unit of the Nth level includes a voxel located on the surface of the three-dimensional model of the object; N is an integer greater than or equal to 1 and less than M; M is the total number of levels of the truncated signed distance field volume; M is an integer;

[0057] If included, the voxel unit of the Nth level is divided into multiple voxel units of the N+1th level according to the octree structure.

[0058] In this embodiment, after the truncated signed distance value of the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value at each level is completed through the previous process, for each voxel unit of the Nth level, based on the truncated signed distance value of the voxel unit and the truncated signed distance values ​​of multiple adjacent voxel units of the voxel unit, for example, the truncated signed distance values ​​of seven adjacent voxel units (the seven adjacent voxel units and the voxel unit can be combined into a large cube), it is determined whether the voxel unit contains voxels located on the surface of the three-dimensional model of the object. If contained, the voxel unit is divided into multiple voxel units of the N+1th level according to the octree structure, for example, the voxel unit is divided into eight voxel units of the N+1th level according to the octree structure.

[0059] It can be seen that through this embodiment, when the voxel unit of the Nth level includes voxels located on the surface of the three-dimensional model of the object, the voxel unit can be divided into multiple voxel units of the N+1th level according to the octree structure, thereby realizing the division of the voxel units of each level through the octree algorithm, and improving the speed of searching for voxels located on the surface of the three-dimensional model of the object in each voxel unit, thereby improving the modeling speed of the three-dimensional model.

[0060] In one embodiment, determining whether the voxel unit at the Nth level includes a voxel located on the surface of the three-dimensional model of the object based on the truncated signed distance value of the voxel unit at the Nth level of the truncated signed distance field volume of the object and the truncated signed distance values ​​of the voxel units adjacent to the voxel unit at the Nth level includes:

[0061] If the truncated signed distance value of the voxel unit at the Nth level has an opposite sign to the truncated signed distance value of any adjacent voxel unit, it is determined that the voxel unit at the Nth level includes voxels located on the surface of the three-dimensional model of the object.

[0062] In this embodiment, for each voxel unit of the Nth level, if the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance value of any adjacent voxel unit of the voxel unit, it is determined that the voxel unit contains voxels located on the surface of the three-dimensional model of the object.

[0063] For example, for a voxel unit at the Nth level, determine whether the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance values ​​of the seven adjacent voxel units of the voxel unit. If it is determined that the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance values ​​of any of the seven adjacent voxel units, it is determined that the voxel unit contains voxels located on the surface of the three-dimensional model of the object.

[0064] It can be seen that through this embodiment, by judging the positive and negative signs of the truncated signed distance value, it is possible to quickly determine whether the voxel unit of the Nth level contains voxels located on the surface of the three-dimensional model of the object. The determination process is fast and the determination result is highly accurate.

[0065] In a specific example, a truncated signed distance field volume is divided into a plurality of voxel units at a first level, each voxel unit includes at least one voxel, and the occupancy value of each voxel unit is set to the occupancy value of the voxel in the lower left corner of the voxel unit, the coordinates of each voxel unit in the truncated signed distance field volume are the coordinates of the voxel in the lower left corner of the voxel unit, and the truncated signed distance value of each voxel unit is the truncated signed distance value of the voxel in the lower left corner of the voxel unit. These three rules apply to voxel units at all levels. Next, within the first level, the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value is traversed, and the truncated signed distance value of the first voxel unit is determined based on the truncated signed distance field volume of the object, the coordinates of the first voxel unit in the truncated signed distance field volume, and the distance determination model, and the truncated signed distance value of the first voxel unit is determined. After each voxel unit in the first level has a truncated signed distance value, for each voxel unit, based on the truncated signed distance value of the voxel unit and the truncated signed distance values ​​of its seven adjacent voxel units (the seven adjacent voxel units and the voxel unit can be combined into a large cube), it is determined whether the voxel unit contains voxels located on the surface of the three-dimensional model of the object. If so, the voxel unit is divided into eight voxel units of the second level according to the octree structure, thereby obtaining each voxel unit of the second level.

[0066] In the second level, the above process is also repeated until the voxel unit of the Mth level is obtained, and each voxel unit of the Mth level includes only one voxel and cannot be subdivided. In the Mth level, for the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value, the truncated signed distance value of the first voxel unit is determined according to the truncated signed distance field stereo, the coordinates of the first voxel unit in the truncated signed distance field stereo and the distance determination model. After the truncated signed distance value of each voxel unit is completed, for each voxel unit, if the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance value of any adjacent voxel unit among the seven adjacent voxel units (the seven adjacent voxel units and the voxel unit can be combined into a large cube), it is determined that the voxel unit contains a voxel located on the surface of the three-dimensional model of the object, that is, the voxel unit is a voxel located on the surface of the three-dimensional model of the object. According to the voxels located on the surface of the three-dimensional model of the object determined in the voxel units of the Mth level, the three-dimensional model of the object can be reconstructed by a surface reconstruction method.

[0067] In one embodiment, the aforementioned determining a model based on the truncated signed distance field volume of the object, the coordinates of the first voxel unit in the truncated signed distance field volume, and the distance to determine the truncated signed distance value of the first voxel unit includes:

[0068] By using the distance determination model, feature extraction is performed on the truncated signed distance field volume of the object to obtain a feature volume;

[0069] The truncated signed distance value of the first voxel unit is determined by the distance determination model according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume.

[0070] Figure 2 is a schematic diagram of the distance determination model provided by one embodiment of the present disclosure. As shown in Figure 2, the model includes a feature extractor and a decoder. First, the feature extractor extracts features from the object's truncated signed distance field volume to obtain a feature volume. Next, the truncated signed distance value of the first voxel is determined based on the first voxel's coordinates in the truncated signed distance field volume and the feature volume.

[0071] It can be seen that through this embodiment, the truncated signed distance field stereo of the object can be feature extracted through the distance determination model to obtain a feature stereo, and the truncated signed distance value of the first voxel unit can be quickly and accurately determined based on the coordinates of the first voxel unit in the truncated signed distance field stereo and the feature stereo.

[0072] In one embodiment, the feature volume includes a feature value of a truncated signed distance value of each voxel in a truncated signed distance field volume of the object; and determining the truncated signed distance value of the first voxel unit based on the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume includes:

[0073] According to the coordinates of the first voxel unit in the truncated signed distance field volume, querying the characteristic value of the truncated signed distance value of the first voxel unit in the characteristic volume by linear interpolation;

[0074] The truncated signed distance value of the first voxel unit is determined according to the characteristic value of the truncated signed distance value of the first voxel unit.

[0075] In this embodiment, the feature volume includes the predicted feature value of the truncated signed distance value of each voxel in the truncated signed distance field volume of the object. That is, the process of extracting features from the truncated signed distance field volume of the object by the feature extractor to obtain the feature volume is a process of predicting the truncated signed distance value of each voxel in the truncated signed distance field volume of the object based on the truncated signed distance value of the first voxel in the truncated signed distance field volume of the object, and processing the predicted truncated signed distance values ​​into feature values.

[0076] Based on this, the eigenvalue of the truncated signed distance value of the first voxel unit is queried in the feature volume by linear interpolation based on the coordinates of the first voxel unit in the truncated signed distance field volume. For example, the coordinates of the first voxel unit in the truncated signed distance field volume are (100, 100, 100), and the corresponding coordinates converted to the feature volume are (51, 51, 51). However, there is no eigenvalue corresponding to the coordinates (51, 51, 51) in the feature volume. Therefore, multiple coordinates closest to the converted coordinates of the first voxel unit are determined in the feature volume, and the eigenvalues ​​of the truncated signed distance values ​​corresponding to these multiple coordinates are interpolated to obtain the eigenvalue of the truncated signed distance value of the first voxel unit.

[0077] Next, the characteristic value of the truncated signed distance value of the first voxel unit is decoded by the decoder shown in FIG2 to obtain the truncated signed distance value of the first voxel unit.

[0078] It can be seen that through this embodiment, according to the coordinates of the truncated signed distance field stereo of the first voxel unit, the characteristic value of the truncated signed distance value of the first voxel unit can be queried in the feature stereo by linear interpolation, and the truncated signed distance value of the first voxel unit can be determined according to the characteristic value of the truncated signed distance value of the first voxel unit, and the determination speed is fast and the determination result is accurate and reliable.

[0079] In one example, referring to Figure 2, the truncated signed distance field stereo of the object input into the distance determination model is a matrix of size 256*256*256. The matrix is ​​processed by the feature extractor to obtain a feature stereo of size 16*16*16. According to the coordinates of the first voxel unit in the truncated signed distance field stereo, the characteristic value of the truncated signed distance value of the first voxel unit obtained by linear interpolation in the feature stereo is 128-length data, and the 128-length data is decoded into the truncated signed distance value of the first voxel unit by the decoder.

[0080] In the above step S106 , a three-dimensional model of the object is established according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0081] In one embodiment, establishing a three-dimensional model of the object based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel includes:

[0082] determining a voxel located on a surface of the three-dimensional model of the object based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel;

[0083] A three-dimensional model of the object is created based on voxels located on the surface of the three-dimensional model of the object.

[0084] According to the foregoing introduction, predicting the truncated signed distance value in the first voxel unit is to predict the truncated signed distance value of a specific voxel in the first voxel unit, and the specific voxel in the first voxel unit is the second voxel. After obtaining the voxel units of the last level in the truncated signed distance field volume of the object using the above method, each voxel unit in the last level includes only one voxel and cannot be subdivided. In the last level, for the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value, the first voxel unit is the second voxel. The model is determined based on the truncated signed distance field volume, the coordinates of the first voxel unit in the truncated signed distance field volume, and the distance, and the truncated signed distance value of the first voxel unit is determined. After the truncated signed distance values ​​of each voxel unit in the last level are completed, for any voxel unit, if the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance values ​​of any of the seven adjacent voxel units (the seven adjacent voxel units and the voxel unit can be combined into a large cube), then the voxel unit is determined to contain a voxel located on the surface of the three-dimensional model of the object, that is, the voxel unit is a voxel located on the surface of the three-dimensional model of the object. Based on the voxels located on the surface of the three-dimensional model of the object determined in the last level of voxel units, a three-dimensional model of the object can be reconstructed using a surface reconstruction method. It can be understood that the voxels located on the surface of the three-dimensional model of the object determined in the last level of voxel units include a first voxel and a second voxel.

[0085] It can be seen that through this embodiment, the voxels located on the surface of the three-dimensional model of the object can be determined, and the three-dimensional model of the object can be established by surface reconstruction based on the voxels located on the surface of the three-dimensional model of the object, thereby generating a high-precision three-dimensional model in real time in a scenario where the three-dimensional model is generated and transmitted in real time.

[0086] Furthermore, as can be seen from the above process, this embodiment does not determine truncated signed distance values ​​for all second voxels. Instead, the truncated signed distance values ​​for the first voxel units occupied by the object model are determined layer by layer based on the hierarchical relationship. Determining the truncated signed distance value for the first voxel unit is equivalent to determining the truncated signed distance value for the second voxel. Furthermore, the hierarchical relationship is established based on whether the voxel unit includes voxels located on the surface of the object's three-dimensional model. Therefore, in this embodiment, only the truncated signed distance values ​​for the necessary second voxels need to be completed, significantly reducing the workload and improving the speed of generating high-precision 3D models in real time.

[0087] In one embodiment, the process of constructing voxel units at each level of the octree can be accelerated by CUDA (Compute Unified Device Architecture). The distance determination model can be accelerated by TensorRT (Nvidia inference engine acceleration tool) to accelerate the inference process.

[0088] The generation process of the above-mentioned distance determination model is introduced below.

[0089] In one embodiment, the above method further includes:

[0090] Acquire a sample truncated signed distance field volume of the sample object; the sample truncated signed distance field volume is used to represent the physical space where the sample object is located; the sample truncated signed distance field volume includes a plurality of sample voxels; the sample voxels are used to represent the spatial region in the physical space where the sample object is located; the sample voxels have sample truncated signed distance values; the sample truncated signed distance values ​​represent the distance between the sample voxels and the surface of the three-dimensional model of the sample object;

[0091] A neural network model is trained according to the sample truncated signed distance field stereo, the coordinates of the sample voxels in the sample truncated signed distance field stereo and the sample truncated signed distance values ​​of the sample voxels; the trained neural network model is a distance determination model.

[0092] In this embodiment, the sample object can be an object such as a person, an animal, or a virtual image. The truncated signed distance field stereo of the sample object is a sample truncated signed distance field stereo. The sample truncated signed distance field stereo is used to represent the physical space where the sample object is located. The sample truncated signed distance field stereo includes a plurality of voxels, which are called sample voxels, and each sample voxel is used to represent a spatial region in the physical space where the sample object is located. That is, the sample truncated signed distance field stereo and the sample voxel are the expressions of the physical space where the sample object is located and the spatial region in the physical space within the computer system. Each sample voxel has a truncated signed distance value, and the truncated signed distance value of the sample voxel is called the sample truncated signed distance value. The sample truncated signed distance value represents the distance between the sample voxel and the surface of the three-dimensional model of the sample object.

[0093] The specific meanings of the sample truncated signed distance field volume and sample truncated signed distance value can be found in the previous explanations of the truncated signed distance field volume and truncated signed distance value for an object, and will not be repeated here. The sample object is used to train the distance determination model. The sample truncated signed distance field volume of the sample object and the sample truncated signed distance value of each sample voxel can be calculated through computer simulation.

[0094] In this embodiment, a pre-built neural network model is trained based on the sample truncated signed distance field volume, the coordinates of the sample voxels in the sample truncated signed distance field volume, and the sample truncated signed distance values ​​of the sample voxels. The trained neural network model serves as a distance determination model. The pre-built neural network model includes a feature extractor and decoder as shown in FIG2 . The process of training the neural network model can be understood as training the feature extractor and decoder in FIG2 .

[0095] It can be seen that through this embodiment, the distance determination model can be trained based on the sample truncated signed distance field stereo of the sample object, the coordinates of the sample voxel in the sample truncated signed distance field stereo, and the sample truncated signed distance value of the sample voxel. The model training efficiency is high, and the distance determination model obtained by training can accurately determine the truncated signed distance of any voxel in the truncated signed distance field stereo of the object.

[0096] In one embodiment, training a neural network model based on a sample truncated signed distance field volume, coordinates of sample voxels in the sample truncated signed distance field volume, and sample truncated signed distance values ​​of the sample voxels includes:

[0097] Through the neural network model, the sample truncated signed distance field stereo is extracted to obtain the sample feature stereo;

[0098] Determining the predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field volume and the sample feature volume through a neural network model;

[0099] A neural network model is trained based on the predicted truncated signed distance values ​​of the sample voxels and the sample truncated signed distance values ​​of the sample voxels.

[0100] In this embodiment, a feature extractor to be trained in a neural network model performs feature extraction on a sample truncated signed distance field volume to obtain a sample feature volume. A predicted truncated signed distance value for the sample voxel is determined based on the coordinates of the sample voxel in the sample truncated signed distance field volume and the sample feature volume. The predicted truncated signed distance value is the predicted truncated signed distance value for the sample voxel. The neural network model is trained based on the predicted truncated signed distance value for the sample voxel and the sample truncated signed distance value for the sample voxel. The predicted truncated signed distance value is the predicted value, and the sample truncated signed distance value is the true value.

[0101] As can be seen, through this embodiment, the predicted truncated signed distance value of the sample voxel can be obtained through the neural network model, and the neural network model can be trained based on the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel. The predicted truncated signed distance value is the predicted value, and the sample truncated signed distance value is the true value. The neural network model is trained based on the predicted value and the true value, thereby improving the accuracy of the trained distance determination model.

[0102] In one embodiment, the sample feature volume includes a feature value of a predicted truncated signed distance value of each sample voxel; and determining the predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field volume and the sample feature volume includes:

[0103] According to the coordinates of the sample voxel in the sample truncated signed distance field volume, query the characteristic value of the predicted truncated signed distance value of the sample voxel in the sample feature volume by linear interpolation;

[0104] The predicted truncated signed distance value of the sample voxel is determined according to the characteristic value of the predicted truncated signed distance value of the sample voxel.

[0105] In this embodiment, after the sample truncated signed distance value of each sample voxel in the sample truncated signed distance field volume of the sample object is known, feature extraction is performed on the sample truncated signed distance field volume by a feature extractor to be trained.

[0106] Based on this, according to the coordinates of each sample voxel in the sample truncated signed distance field volume, the eigenvalue of the predicted truncated signed distance value of the sample voxel is queried in the sample feature volume by linear interpolation. For example, the coordinates of the sample voxel in the sample truncated signed distance field volume are (100, 100, 100), and the corresponding coordinates converted to the sample feature volume are (51, 51, 51). However, there is no eigenvalue corresponding to the coordinates (51, 51, 51) in the sample feature volume. Therefore, multiple coordinates closest to the converted coordinates of the sample voxel are determined in the sample feature volume, and the eigenvalues ​​of the truncated signed distance values ​​corresponding to these multiple coordinates are interpolated to obtain the eigenvalue of the predicted truncated signed distance value of the sample voxel. Then, the eigenvalue of the predicted truncated signed distance value of the sample voxel is decoded by the decoder shown in Figure 2 to obtain the predicted truncated signed distance value of the sample voxel.

[0107] It can be seen that through this embodiment, the characteristic value of the predicted truncated signed distance value of the sample voxel can be queried in the sample feature stereo according to the coordinates of the sample voxel in the sample truncated signed distance field stereo by linear interpolation, and the predicted truncated signed distance value of the sample voxel can be determined according to the characteristic value of the predicted truncated signed distance value of the sample voxel, and the determination result is accurate, reliable and efficient.

[0108] In one embodiment, the true value of the sample voxel, i.e., the sample truncated signed distance value, includes an observed value and a simulated value; and training a neural network model based on the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel includes:

[0109] Calculating a first loss function, a second loss function, and a third loss function according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel;

[0110] Training a neural network model based on the first loss function, the second loss function, and calculation results of the third loss function;

[0111] Among them, the first loss function is used to determine the first distance between the simulation value and the predicted truncated signed distance value and the second distance between the observation value and the predicted truncated signed distance value; the second loss function is used to constrain the association between the predicted truncated signed distance value and the occupancy value of the sample voxel; the occupancy value indicates whether the sample voxel is occupied by the three-dimensional model of the sample object; the third loss function is used to perform smoothing constraints on the predicted truncated signed distance value.

[0112] In this embodiment, the true value of the sample voxel, that is, the sample truncated signed distance value, includes an observed value and a simulated value. The simulated value is the most idealized sample truncated signed distance value of the sample voxel obtained by computer simulation, and the observed value is the value obtained by adding noise to the simulated value, and the noise is used to simulate the noise in the real world.

[0113] In this embodiment, there are three loss functions preset: the first, second, and third. The first loss function is used to determine the first distance between the simulated value of the sample voxel and the predicted truncated signed distance value of the sample voxel, and to determine the second distance between the observed value of the sample voxel and the predicted truncated signed distance value of the sample voxel. In one example, the first loss function is specifically as follows:

[0114] In this formula, L recon represents the reconstruction loss, p represents the sample voxel, represents the sample truncated signed distance field stereo, λ consis is the preset coefficient, is the predicted truncated signed distance value of the sample voxel, x *is the simulated value of the sample voxel, ρ is the function for calculating the huber weight, and x is the observed value of the sample voxel.

[0115] The second loss function is used to constrain the correlation between the predicted truncated signed distance value of the sample voxel and the occupancy value of the sample voxel, where the occupancy value indicates whether the sample voxel is occupied by the three-dimensional model of the sample object. The second loss function can make the predicted truncated signed distance value of the sample voxel indicate that the sample voxel is located inside or on the surface of the three-dimensional model of the sample object and is occupied, and make the predicted truncated signed distance value of the sample voxel indicate that the sample voxel is located outside the three-dimensional model of the sample object and is not occupied, and the occupancy value of the sample voxel is 0. In an example, the second loss function is specifically as follows:

[0116] Among them, L hull is the viewing cone constraint, o is the viewing cone value (i.e., occupancy value), o equal to 1 indicates that the sample voxel is inside or on the surface of the three-dimensional model of the sample object, o equal to 0 indicates that the sample voxel is outside the three-dimensional model of the sample object, p represents the sample voxel, represents a sample truncated signed distance field stereo, is the predicted truncated signed distance value for the sample voxel.

[0117] The third loss function is used to impose a smooth constraint on the predicted truncated signed distance value of the sample voxel. In one example, the third loss function is as follows:

[0118] Among them, L TV represents the smoothness constraint, p represents the sample voxel, represents a sample truncated signed distance field stereo, is the predicted truncated signed distance value of the sample voxel, is the TSDF value predicted by sample pixel p With the sample pixel (p x +δ x ,p y ,p z ) with a constant offset δ x The shorthand for the squared difference between and Same reason.

[0119] In this embodiment, the first loss function, the second loss function, and the third loss function are calculated based on the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel, and the neural network model is trained based on the calculation results of the first loss function, the second loss function, and the second loss function. In one example, the neural network model can be trained by combining the calculation results of the first loss function, the second loss function, and the second loss function using the following formula. In the formula, L represents the total loss of the model, λ TV and λ hull All are preset coefficients. L=L recon +λ TV L TV +λ ull L hull (4)

[0120] It can be seen that through this embodiment, multiple loss functions are set, and the distance determination model is constrained from different aspects through multiple loss functions, thereby improving the accuracy of the distance determination model and the accuracy of the three-dimensional model generated in real time.

[0121] In one embodiment, the feature extractor in the distance determination model includes a multi-stage cascaded 3D convolutional network. When training the distance determination model, the Adam optimizer is selected, the training weights are typically set to 1e-4, and the regularization term coefficient is set to 1e-5. Sample objects may include people and objects. The three-dimensional model of the sample object may include a mesh model of a specific scene composed of people and objects. The sample voxels may be partial voxels in a sample truncated signed distance field stereo. In the sample truncated signed distance field stereo, the sample voxels are obtained by non-uniformly sampling locations near the model surface of the sample object. The greater the curvature change of the model surface, the more sample voxels are collected. In one example, when sampling the sample voxels, the ratio of the number of sample voxels near the model surface to the number of sample voxels located outside the model surface is 3:1. Each time the distance determination model is trained, 5,000 to 10,000 sample voxels may be collected in the sample truncated signed distance field stereo.

[0122] In summary, the three-dimensional modeling method disclosed in this embodiment can rely on fewer sensors, such as three sensors, to photograph an object from three perspectives to obtain a depth map, and establish a truncated signed distance field stereo of the object based on the depth map. Due to the small number of sensors, the truncated signed distance field stereo will lack the truncated signed distance values ​​of some voxels. The truncated signed distance values ​​of the voxels in the truncated signed distance field stereo can be supplemented by the distance determination model, thereby reconstructing the three-dimensional model of the object. The reconstructed three-dimensional model is a watertight model with high accuracy, and the distance determination model and the octree algorithm make the model reconstruction speed faster, and the three-dimensional model can be generated and transmitted in real-time scenes.

[0123] An embodiment of the present disclosure further provides a three-dimensional modeling device for implementing the above-mentioned three-dimensional modeling method. FIG3 is a schematic diagram of the structure of the three-dimensional modeling device provided by an embodiment of the present disclosure. As shown in FIG3 , the device includes:

[0124] A stereo acquisition unit 31 is configured to acquire a truncated signed distance field (SDF) stereo image of an object; the SDF stereo image is configured to represent a physical space in which the object is located; the SDF stereo image comprises a plurality of voxels; the voxels are configured to represent a spatial region in the physical space in which the object is located; a first voxel among the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0125] a distance determination unit 32 configured to determine, for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, a truncated signed distance value for the second voxel based on the truncated signed distance field volume, the coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0126] The model building unit 33 is configured to build a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0127] Optionally, the stereo acquisition unit 31 is specifically configured to:

[0128] Projecting a three-dimensional grid representing the physical space where the object is located onto each viewing angle of the object; the three-dimensional grid is divided into the plurality of voxels;

[0129] determining a truncated signed distance value of the voxel according to the depth data of the voxel at each of the viewing angles and the depth map of the object at each of the viewing angles obtained by projection;

[0130] The voxel that obtains the truncated signed distance value is determined as the first voxel, and the three-dimensional mesh that obtains the truncated signed distance value after projection is determined as the truncated signed distance field three-dimensional mesh.

[0131] Optionally, the truncated signed distance field stereo includes multiple levels; each level includes multiple voxel units; the voxel unit includes at least one voxel; the voxel unit of the lower level is nested in the voxel unit of the higher level; the distance determination unit 32 is specifically configured to:

[0132] For a first voxel unit of each level occupied by the three-dimensional model of the object and having a missing truncated signed distance value, determining a truncated signed distance value of the first voxel unit according to the truncated signed distance field volume, a coordinate of the first voxel unit in the truncated signed distance field volume, and the distance determination model;

[0133] The first voxel unit includes the second voxel; the coordinates of the first voxel unit in the truncated signed distance field stereo are the coordinates of the included second voxel; and the truncated signed distance value of the first voxel unit is the truncated signed distance value of the included second voxel.

[0134] Optionally, a stratification unit is further included, configured to:

[0135] determining, based on the truncated signed distance values ​​of the voxel units at the Nth level of the truncated signed distance field volume and the truncated signed distance values ​​of the voxel units adjacent to the voxel units at the Nth level, whether the voxel units at the Nth level include voxels located on the surface of the three-dimensional model of the object; wherein N is an integer greater than or equal to 1 and less than M; and M is the total number of levels of the truncated signed distance field volume;

[0136] If included, the voxel unit of the Nth level is divided into multiple voxel units of the N+1th level according to the octree structure.

[0137] Optionally, the stratification unit is specifically used to:

[0138] If the positive and negative signs of the truncated signed distance values ​​of the voxel units of the Nth level are opposite to the positive and negative signs of the truncated signed distance values ​​of any of the adjacent voxel units, it is determined that the voxel units of the Nth level contain voxels located on the surface of the three-dimensional model of the object.

[0139] Optionally, the distance determining unit 32 is further specifically configured to:

[0140] Performing feature extraction on the truncated signed distance field stereo using the distance determination model to obtain a feature stereo;

[0141] The truncated signed distance value of the first voxel unit is determined by the distance determination model according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume.

[0142] Optionally, the feature volume includes a feature value of a truncated signed distance value of each voxel in the plurality of voxels; the distance determining unit 32 is further specifically configured to:

[0143] querying a characteristic value of the truncated signed distance value of the first voxel unit in the feature volume by linear interpolation according to the coordinates of the first voxel unit in the truncated signed distance field volume;

[0144] The truncated signed distance value of the first voxel unit is determined according to a characteristic value of the truncated signed distance value of the first voxel unit.

[0145] Optionally, the model building unit 33 is specifically configured to:

[0146] determining a voxel located on a surface of the three-dimensional model of the object based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel;

[0147] A three-dimensional model of the object is created based on voxels located on a surface of the three-dimensional model of the object.

[0148] Optionally, a model training unit is further included, configured to:

[0149] Acquire a sample truncated signed distance field volume of the sample object; the sample truncated signed distance field volume is used to represent the physical space where the sample object is located; the sample truncated signed distance field volume includes a plurality of sample voxels; the sample voxels are used to represent a spatial region in the physical space where the sample object is located; the sample voxels have sample truncated signed distance values; the sample truncated signed distance values ​​represent the distance between the sample voxels and the surface of the three-dimensional model of the sample object;

[0150] A neural network model is trained according to the sample truncated signed distance field stereo, the coordinates of the sample voxels in the sample truncated signed distance field stereo and the sample truncated signed distance values ​​of the sample voxels; the trained neural network model is the distance determination model.

[0151] Optionally, the model training unit is specifically used to:

[0152] Performing feature extraction on the sample truncated signed distance field stereo using the neural network model to obtain a sample feature stereo;

[0153] Determining, by the neural network model, a predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field volume and the sample feature volume;

[0154] A neural network model is trained based on the predicted truncated signed distance values ​​of the sample voxels and the sample truncated signed distance values ​​of the sample voxels.

[0155] Optionally, the sample feature volume includes a feature value of a predicted truncated signed distance value of each sample voxel; and the model training unit is further specifically configured to:

[0156] According to the coordinates of the sample voxel in the sample truncated signed distance field volume, querying the characteristic value of the predicted truncated signed distance value of the sample voxel in the sample feature volume by linear interpolation;

[0157] The predicted truncated signed distance value of the sample voxel is determined according to the characteristic value of the predicted truncated signed distance value of the sample voxel.

[0158] Optionally, the sample truncated signed distance value of the sample voxel includes an observed value and a simulated value; and the model training unit is further specifically configured to:

[0159] Calculating a first loss function, a second loss function, and a third loss function according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel;

[0160] Training a neural network model based on the calculation results of the first loss function, the second loss function, and the second loss function;

[0161] Among them, the first loss function is used to determine the first distance between the simulation value and the predicted truncated signed distance value and the second distance between the observation value and the predicted truncated signed distance value; the second loss function is used to constrain the association between the predicted truncated signed distance value and the occupancy value of the sample voxel; the occupancy value indicates whether the sample voxel is occupied by the three-dimensional model of the sample object; the third loss function is used to smoothly constrain the predicted truncated signed distance value.

[0162] The three-dimensional modeling device in the embodiment of the present disclosure can implement each process of the above-mentioned three-dimensional modeling method embodiment and achieve the same effects and functions, which will not be repeated here.

[0163] An embodiment of the present disclosure also provides an electronic device. FIG4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. As shown in FIG4 , the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 401 and a memory 402. One or more applications or data may be stored in the memory 402. Among them, the memory 402 may be a temporary storage or a persistent storage. The application stored in the memory 402 may include one or more modules (not shown in the figure), each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 401 may be configured to communicate with the memory 402 and execute a series of computer executable instructions in the memory 402 on the electronic device. The electronic device may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input or output interfaces 405, one or more keyboards 406, etc.

[0164] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer-executable instructions, wherein when the computer-executable instructions are executed, the processor implements the following process:

[0165] Acquire a truncated signed distance field volume of the object; the truncated signed distance field volume is used to represent the physical space where the object is located; the truncated signed distance field volume includes a plurality of voxels; the voxels are used to represent a spatial region in the physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0166] determining, for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, a truncated signed distance value for the second voxel based on the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0167] A three-dimensional model of the object is established based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0168] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned three-dimensional modeling method embodiment and achieve the same effects and functions, which will not be repeated here.

[0169] Another embodiment of the present disclosure further provides a computer-readable storage medium for storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:

[0170] Acquire a truncated signed distance field volume of the object; the truncated signed distance field volume is used to represent the physical space where the object is located; the truncated signed distance field volume includes a plurality of voxels; the voxels are used to represent a spatial region in the physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0171] determining, for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, a truncated signed distance value for the second voxel based on the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0172] A three-dimensional model of the object is established based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0173] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned three-dimensional modeling method embodiment and achieve the same effects and functions, which will not be repeated here.

[0174] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0175] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0176] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0177] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0178] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0179] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0181] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0182] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0183] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0184] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0185] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0186] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. A three-dimensional modeling method, comprising: Acquire a truncated signed distance field stereo of the object; the truncated signed distance field stereo is used to represent the physical space where the object is located; The truncated signed distance field volume comprises a plurality of voxels; the voxels are used to represent a spatial region in a physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object; For a second voxel in the plurality of voxels for which the truncated signed distance value is missing, determining a truncated signed distance value for the second voxel according to the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value; A three-dimensional model of the object is established according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

2. The method according to claim 1, wherein obtaining a truncated signed distance field stereo of the object comprises: Projecting a three-dimensional grid representing the physical space where the object is located to each viewing angle of the object; The three-dimensional grid is divided into the plurality of voxels; Determine a truncated signed distance value of the voxel according to the depth data of the voxel at each of the viewing angles and the depth map of the object at each of the viewing angles obtained by projection; The voxel determined to obtain the truncated signed distance value is determined as the first voxel, and the three-dimensional grid obtained after projection to obtain the truncated signed distance value is determined as the truncated signed distance field three-dimensional grid.

3. The method according to claim 1, wherein the truncated signed distance field volume comprises a plurality of levels; each of the levels comprises a plurality of voxel units; the voxel unit comprises at least one voxel; the voxel unit of a lower level is nested within the voxel unit of a higher level; and determining the truncated signed distance value of the second voxel according to the truncated signed distance field volume, the coordinates of the second voxel in the truncated signed distance field volume and a distance determination model for determining the truncated signed distance value comprises: For each of the levels of the first voxel unit occupied by the three-dimensional model of the object and having a missing truncated signed distance value, the first voxel unit is Determine the truncated signed distance value of the first voxel unit based on the coordinates in the truncated signed distance field stereo and the distance determination model; The first voxel unit includes the second voxel; the coordinates of the first voxel unit in the truncated signed distance field stereo are the coordinates of the included second voxel; and the truncated signed distance value of the first voxel unit is the truncated signed distance value of the included second voxel.

4. The method according to claim 3, further comprising: According to the truncated signed distance value of the voxel unit of the Nth level of the truncated signed distance field stereo and the truncated signed distance value of the voxel unit adjacent to the voxel unit of the Nth level, it is determined whether the voxel unit of the Nth level includes the voxel located on the surface of the three-dimensional model of the object; N is an integer greater than or equal to 1 and less than M; M is the total number of levels of the truncated signed distance field stereo; If included, the voxel unit of the Nth level is divided into multiple voxel units of the N+1th level according to the octree structure.

5. The method according to claim 4, wherein judging whether the voxel unit at the Nth level includes voxels located on the surface of the three-dimensional model of the object according to the truncated signed distance value of the voxel unit at the Nth level of the truncated signed distance field volume and the truncated signed distance values ​​of the voxel units adjacent to the voxel unit at the Nth level comprises: If the positive and negative signs of the truncated signed distance values ​​of the voxel units of the Nth level are opposite to the positive and negative signs of the truncated signed distance values ​​of any of the adjacent voxel units, it is determined that the voxel units of the Nth level include voxels located on the surface of the three-dimensional model of the object.

6. The method according to claim 3, wherein the determining the truncated signed distance value of the first voxel unit according to the truncated signed distance field volume, the coordinates of the first voxel unit in the truncated signed distance field volume and the distance determination model comprises: By using the distance determination model, feature extraction is performed on the truncated signed distance field stereo to obtain a feature stereo; The truncated signed distance value of the first voxel unit is determined by the distance determination model according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume.

7. The method according to claim 6, wherein the feature volume comprises a feature value of a truncated signed distance value of each voxel in the plurality of voxels; and determining the truncated signed distance value of the first voxel unit according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume comprises: According to the coordinates of the first voxel unit in the truncated signed distance field volume, querying the characteristic value of the truncated signed distance value of the first voxel unit in the characteristic volume by linear interpolation; The truncated signed distance value of the first voxel unit is determined according to the characteristic value of the truncated signed distance value of the first voxel unit.

8. The method according to claim 1, wherein the step of establishing the three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel comprises: determining a voxel located on a surface of the three-dimensional model of the object based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel; A three-dimensional model of the object is established based on voxels located on a surface of the three-dimensional model of the object.

9. The method according to claim 1, further comprising: Acquire a sample truncated signed distance field stereo of a sample object; the sample truncated signed distance field stereo is used to represent the physical space where the sample object is located; The sample truncated signed distance field stereo includes a plurality of sample voxels; the sample voxels are used to represent a spatial region in a physical space where a sample object is located; The sample voxel has a sample truncated signed distance value; the sample truncated signed distance value represents the distance between the sample voxel and the surface of the three-dimensional model of the sample object; A neural network model is trained according to the sample truncated signed distance field stereo, the coordinates of the sample voxels in the sample truncated signed distance field stereo and the sample truncated signed distance values ​​of the sample voxels; the trained neural network model is the distance determination model.

10. The method according to claim 9, wherein the training of a neural network model according to the sample truncated signed distance field volume, the coordinates of the sample voxels in the sample truncated signed distance field volume and the sample truncated signed distance values ​​of the sample voxels comprises: By using the neural network model, feature extraction is performed on the sample truncated signed distance field stereo to obtain a sample feature stereo; Determining, by the neural network model, a predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field stereo and the sample feature stereo; According to the predicted truncated signed distance value of the sample voxel and the sample truncation signed distance value of the sample voxel The distance value is used to train the neural network model.

11. The method according to claim 10, wherein the sample feature volume comprises a feature value of a predicted truncated signed distance value of each sample voxel; and determining the predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field volume and the sample feature volume comprises: According to the coordinates of the sample voxel in the sample truncated signed distance field volume, querying the characteristic value of the predicted truncated signed distance value of the sample voxel in the sample feature volume by linear interpolation; The predicted truncated signed distance value of the sample voxel is determined according to the characteristic value of the predicted truncated signed distance value of the sample voxel.

12. The method according to claim 10, wherein the sample truncated signed distance value of the sample voxel comprises an observed value and a simulated value; and training a neural network model according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel comprises: Calculating a first loss function, a second loss function, and a third loss function according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel; Training a neural network model based on the first loss function, the second loss function, and the calculation results of the second loss function; Among them, the first loss function is used to determine the first distance between the simulation value and the predicted truncated signed distance value and the second distance between the observed value and the predicted truncated signed distance value; the second loss function is used to constrain the association between the predicted truncated signed distance value and the occupancy value of the sample voxel; the occupancy value indicates whether the sample voxel is occupied by the three-dimensional model of the sample object; the third loss function is used to perform smoothing constraints on the predicted truncated signed distance value.

13. A three-dimensional modeling device, comprising: A stereo acquisition unit, used for acquiring a truncated signed distance field stereo of the object; The truncated signed distance field stereo is used to represent the physical space where the object is located; The truncated signed distance field volume comprises a plurality of voxels; the voxels are used to represent a spatial region in a physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object; A distance determining unit is configured to determine a second one of the plurality of voxels where the truncated signed distance value is missing. voxel, determining a truncated signed distance value of the second voxel according to the truncated signed distance field volume, the coordinates of the second voxel in the truncated signed distance field volume and a distance determination model for determining the truncated signed distance value; The model building unit is used to build a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

14. An electronic device comprising: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the method described in any one of claims 1 to 12 are implemented.

Images