Three-dimensional modeling method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional modeling method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a truncation symbol distance field stereo of an object; the truncated symbol distance field stereoscopically comprises a plurality of voxels; a first voxel of the plurality of voxels has a truncated symbol distance value; the truncated symbol distance value represents a distance between the voxel and a surface of the three-dimensional model of the object; for a second voxel with a missing truncated symbol distance value in the plurality of voxels, determining the truncated symbol distance value of the second voxel according to the truncated symbol distance field stereo, the coordinates of the second voxel in the truncated symbol distance field stereo and a distance determination model for determining the truncated symbol distance value; and establishing a three-dimensional model of the object according to the truncation symbol distance value of the first voxel and the determined truncation symbol distance value of the second voxel.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a three-dimensional modeling method, device, electronic device and storage medium. Background Art

[0002] Volumetric video is a technology that can capture information in three-dimensional space, such as depth information and color information, and generate a three-dimensional model sequence. The generated three-dimensional model sequence can be connected to form a new video format that can be viewed from any perspective.

[0003] In the scenario of real-time transmission of volumetric video, it is necessary to generate and transmit a 3D model of each frame in the volumetric video in real time. However, the 3D model generation solution in the prior art has the defect of low accuracy of the generated 3D model when applied to the real-time generation of 3D models. Summary of the invention

[0004] The embodiments of the present disclosure provide a three-dimensional modeling method, device, electronic device and storage medium, which can be applied in a three-dimensional model real-time generation scenario to generate a high-precision three-dimensional model.

[0005] In a first aspect, an embodiment of the present disclosure provides a three-dimensional modeling method, comprising:

[0006] Acquire a truncated signed distance field stereo of the object; the truncated signed distance field stereo is used to represent the physical space where the object is located; the truncated signed distance field stereo includes a plurality of voxels; the voxels are used to represent a spatial region in the physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0007] For a second voxel in the plurality of voxels for which the truncated signed distance value is missing, determining a truncated signed distance value for the second voxel according to the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0008] A three-dimensional model of the object is established according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0009] In a second aspect, an embodiment of the present disclosure provides a three-dimensional modeling device, including:

[0010] A stereo acquisition unit, configured to acquire a truncated signed distance field stereo of an object; the truncated signed distance field stereo is used to represent the physical space where the object is located; the truncated signed distance field stereo comprises a plurality of voxels; the voxels are used to represent a spatial region in the physical space where the object is located; a first voxel among the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0011] a distance determination unit, configured to determine, for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, a truncated signed distance value of the second voxel according to the truncated signed distance field volume, a coordinate of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0012] The model building unit is used to build a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the steps of the method described in the first aspect when executed by a processor.

[0015] In one or more embodiments of the present disclosure, a truncated signed distance field stereo of the object is first obtained, the truncated signed distance field stereo is used to represent the physical space where the object is located, the truncated signed distance field stereo includes a plurality of voxels, the voxel is used to represent the spatial region in the physical space where the object is located, a first voxel among the plurality of voxels has a truncated signed distance value, the truncated signed distance value represents the distance between the voxel and the surface of the three-dimensional model of the object, then for a second voxel among the plurality of voxels whose truncated signed distance value is missing, a model is determined according to the truncated signed distance field stereo of the object, the coordinates of the second voxel in the truncated signed distance field stereo, and the distance used to determine the truncated signed distance value, the truncated signed distance value of the second voxel is determined, and finally, according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel, a three-dimensional model of the object is established. It can be seen that through this embodiment, the truncated signed distance value of the second voxel can be quickly and accurately determined through the distance determination model, and the three-dimensional model of the object can be established based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel. Therefore, this embodiment can be applied in the real-time generation of three-dimensional models, and can generate high-precision three-dimensional models efficiently and quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0017] Figure 1 A schematic diagram of a flow chart of a three-dimensional modeling method provided in one embodiment of the present disclosure;

[0018] Figure 2 A schematic diagram of the structure of a distance determination model provided in one embodiment of the present disclosure;

[0019] Figure 3 A schematic diagram of the structure of a three-dimensional modeling device provided in one embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0022] The embodiments of the present disclosure provide a three-dimensional modeling method, which can be applied in a three-dimensional model real-time generation scenario to generate a high-precision three-dimensional model.

[0023] The 3D model real-time generation scene can be a scene of generating volumetric video in real time and transmitting the volumetric video. To facilitate understanding of this embodiment, the volumetric video is first introduced here. The volumetric video includes multiple video frames, each of which has a 3D model. The 3D models of each video frame form a 3D model sequence. The process of playing the volumetric video can be understood as the process of playing the 3D model sequence.

[0024] The three-dimensional model in the volumetric video is a model obtained by modeling a specific object, such as a person, and the specific object moves over time. First, multiple cameras are arranged at different positions around the specific object. During the movement of the specific object, the specific object is photographed by multiple cameras at each moment. Each camera obtains an image of the specific object from its position at each moment, which is equivalent to using multiple cameras to record the movement process of the specific object from different positions. At each moment, a three-dimensional model is established for the specific object based on multiple images taken by multiple cameras at that moment. The three-dimensional model can be understood as the model of the specific object at that moment. The three-dimensional model of the specific object at each moment constitutes a three-dimensional model sequence, and the volumetric video can be generated using the three-dimensional model sequence.

[0025] The scene of generating and transmitting volumetric video in real time may include a live broadcast scene of volumetric video. For example, a user watches a live broadcast of volumetric video through a terminal device, a server generates a three-dimensional model of each frame in real time and sends it to the terminal device, the terminal device receives the three-dimensional model sent by the server, renders each frame of the volumetric video based on the three-dimensional model of each frame and displays it, and each frame of the volumetric video includes a three-dimensional model. Among them, the user can watch the live broadcast of volumetric video through a VR (Virtual Reality) device, or watch the live broadcast of volumetric video through a mobile phone, a tablet computer, a car computer, a wearable device, etc.

[0026] In summary, in the process of generating and transmitting volumetric video in real time, it is necessary to generate and transmit a three-dimensional model in real time. The three-dimensional modeling method provided by the embodiment of the present disclosure can generate a high-precision three-dimensional model in real time, meet the real-time generation requirements of the three-dimensional model, and meet the real-time generation and real-time transmission requirements of the volumetric video.

[0027] Figure 1 A flow chart of a three-dimensional modeling method provided by an embodiment of the present disclosure is provided. The method can be applied to a server and executed by the server, such as Figure 1 As shown, the process includes:

[0028] Step S102, obtaining a truncated signed distance field stereo of the object; the truncated signed distance field stereo is used to represent the physical space where the object is located; the truncated signed distance field stereo includes a plurality of voxels; the voxel is used to represent a spatial region in the physical space where the object is located; the first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents the distance between the voxel and the surface of the three-dimensional model of the object;

[0029] Step S104, for a second voxel whose truncated signed distance value is missing among the plurality of voxels, determining a truncated signed distance value of the second voxel according to the truncated signed distance field stereo of the object, the coordinates of the second voxel in the truncated signed distance field stereo and a distance determination model for determining the truncated signed distance value;

[0030] Step S106 : establishing a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0031] In this embodiment, a truncated signed distance field stereo of the object is first obtained, the truncated signed distance field stereo is used to represent the physical space where the object is located, the truncated signed distance field stereo includes multiple voxels, the voxel is used to represent the spatial area in the physical space where the object is located, a first voxel among the multiple voxels has a truncated signed distance value, and the truncated signed distance value represents the distance between the voxel and the surface of the three-dimensional model of the object, then for a second voxel among the multiple voxels whose truncated signed distance value is missing, the truncated signed distance value of the second voxel is determined according to the truncated signed distance field stereo of the object, the coordinates of the second voxel in the truncated signed distance field stereo, and a distance determination model used to determine the truncated signed distance value, and finally, the three-dimensional model of the object is established according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel. It can be seen that through this embodiment, the truncated signed distance value of the second voxel can be quickly and accurately determined through the distance determination model, and the three-dimensional model of the object can be established based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel. Therefore, this embodiment can be applied in the real-time generation of three-dimensional models, and can generate high-precision three-dimensional models efficiently and quickly.

[0032] Below Figure 1 The method in is described in detail.

[0033] In this embodiment, the object may be an object to be modeled, and the object to be modeled includes but is not limited to a person, an animal, a virtual object, etc. When modeling an object to be modeled (hereinafter referred to as an object), the object is set in a physical space, and the physical space may be a space such as a studio of a specific size. RGB (red-green-blue) images of the object are taken from different perspectives in the physical space, and a grayscale image is generated based on the taken RGB image. After obtaining the grayscale image of the object at each perspective, a truncated signed distance field stereo of the object may be generated.

[0034] The truncated signed distance field stereo is a 3D stereo grid with truncated signed distance values. The stereo grid includes a plurality of voxels, each of which has a truncated signed distance value, and the truncated signed distance value represents the distance from the voxel to the model surface.

[0035] Based on this, in this embodiment, the truncated signed distance field stereo of the object is a 3D stereo grid with a truncated signed distance value, and the stereo grid is used to represent the physical space where the object is located. The stereo grid includes multiple voxels, and each voxel corresponds to a spatial area in the physical space where the object is located, and is used to represent the spatial area. It can also be understood that the stereo grid and the voxel are the physical space where the object is located and the expression of each spatial area in the physical space in the computer system. In the truncated signed distance field stereo of the object, each voxel has a truncated signed distance value, and the truncated signed distance value represents the distance from the voxel to the surface of the three-dimensional model of the object.

[0036] The truncated signed distance value is calculated by the truncated signed distance function (TSDF). TSDF is a method for calculating hidden potential surfaces in 3D reconstruction. TSDF divides the space into regular voxels and stores a truncated signed distance value for each voxel. This truncated signed distance value represents the distance from the voxel to the model surface. The truncated signed distance value of the voxel inside the model surface is a negative value, and the truncated signed distance value of the voxel outside the model surface is a positive value. In one embodiment, the maximum and minimum values ​​of the truncated signed distance value are preset. When the calculated distance from the voxel to the model surface is greater than the maximum value, the truncated signed distance value is taken as the maximum value. When the calculated distance from the voxel to the model surface is less than the minimum value, the truncated signed distance value is taken as the minimum value.

[0037] In one embodiment, obtaining a truncated signed distance field stereo of an object includes:

[0038] Projecting a three-dimensional grid representing the physical space where the object is located to each viewing angle of the object; the three-dimensional grid is divided into a plurality of voxels;

[0039] Determine a truncated signed distance value of the voxel according to the depth data of the voxel at each viewing angle obtained by projection and the depth map of the object at each viewing angle;

[0040] The voxel that obtains the truncated signed distance value is determined as the first voxel, and the three-dimensional grid that obtains the truncated signed distance value after projection is determined as the truncated signed distance field three-dimensional grid.

[0041] First, a three-dimensional grid for representing the physical space where the object is located is obtained, and the three-dimensional grid is divided into voxels one by one. Then, the three-dimensional grid is projected to each viewing angle of the object, which is the viewing angle used to capture images of the object and model it. After the three-dimensional grid is projected to each viewing angle, the depth data of the voxels in the three-dimensional grid at each viewing angle can be obtained. Then, according to the depth data of the voxels at each viewing angle obtained by the projection and the depth map of the object at each viewing angle, the truncated signed distance value of the voxel is determined. For example, for the same voxel, the truncated signed distance value of the voxel is calculated according to the depth value of the voxel at each viewing angle obtained by the projection and the depth value of the voxel recorded in the depth map at each viewing angle. It can be understood that for voxels that can be seen from any viewing angle, the depth value of the voxel at the viewing angle can be projected to obtain the depth value of the voxel, thereby calculating the truncated signed distance value of the voxel. If a voxel is not seen from any viewing angle, the depth value of the voxel at any viewing angle cannot be projected, so the truncated signed distance value of the voxel cannot be calculated.

[0042] Therefore, when calculating the truncated signed distance value of a voxel, the truncated signed distance value of the first voxel in all the voxels can be calculated, but there are still some voxels whose truncated signed distance values ​​cannot be calculated, and these voxels are called second voxels.

[0043] Based on this, in this embodiment, a plurality of voxels that obtain truncated signed distance values ​​are determined as first voxels, and the projected three-dimensional grid that records the truncated signed distance values ​​is determined as a truncated signed distance field three-dimensional grid.

[0044] It can be seen that through this embodiment, the three-dimensional grid used to represent the physical space where the object is located is projected to each viewing angle of the object, and the truncated signed distance value of the first voxel is determined and the truncated signed distance field stereo is generated based on the depth data of the voxel at each viewing angle obtained by the projection and the depth map of the object at each viewing angle. The method of generating the truncated signed distance field stereo is efficient and accurate.

[0045] As can be seen from the foregoing introduction, the multiple voxels included in the truncated signed distance field stereo of the object are divided into a first voxel and a second voxel, the first voxel is a voxel in which a truncated signed distance value is recorded in the truncated signed distance field stereo of the object, and the second voxel is a voxel in which a truncated signed distance value is not determined in the process of generating the truncated signed distance field stereo. Based on this, in the above step S104, for the second voxel in which the truncated signed distance value is missing among the multiple voxels included in the truncated signed distance field stereo of the object, the truncated signed distance value of the second voxel is determined according to the truncated signed distance field stereo of the object, the coordinates of the second voxel in the truncated signed distance field stereo, and the distance determination model used to determine the truncated signed distance value.

[0046] In one embodiment, a truncated signed distance field volume comprises a plurality of levels; each level comprises a plurality of voxel units; a voxel unit comprises at least one voxel; a voxel unit of a lower level is nested in a voxel unit of a higher level; and according to the truncated signed distance field volume of the object, a coordinate of a second voxel in the truncated signed distance field volume and a distance determination model for determining a truncated signed distance value, determining a truncated signed distance value of the second voxel comprises:

[0047] For a first voxel unit of each level occupied by the three-dimensional model of the object and having a missing truncated signed distance value, determine the truncated signed distance value of the first voxel unit according to the truncated signed distance field volume of the object, the coordinates of the first voxel unit in the truncated signed distance field volume, and the distance determination model;

[0048] The first voxel unit includes the second voxel; the coordinates of the first voxel unit in the truncated signed distance field stereo are the coordinates of the included second voxel; and the truncated signed distance value of the first voxel unit is the truncated signed distance value of the included second voxel.

[0049] In this embodiment, the truncated signed distance field is divided into a plurality of levels, each level includes a plurality of voxel units, each voxel unit includes at least one voxel, the lowest level voxel unit includes one voxel, and the low-level voxel unit is nested in the high-level voxel unit. For example, the first-level voxel unit is the highest-level voxel unit, and the first-level voxel unit can be divided into 8 second-level voxel units, and the second-level voxel unit is a lower-level voxel unit, so that the low-level voxel unit is nested in the high-level voxel unit. How to achieve the division of voxel units at each level will be described later.

[0050] In this embodiment, after the truncated signed distance field stereo is divided into multiple levels and the voxel units included in each level are determined, the occupancy value, coordinates and truncated signed distance value of each voxel unit in each level are determined. As can be seen from the previous introduction, the voxel unit of each level includes at least one voxel, so the occupancy value of the voxel unit of each level is the occupancy value of a specific voxel included in the voxel unit, and the occupancy value of 1 indicates that the voxel unit is occupied by the three-dimensional model of the object, that is, the voxel unit is located inside or on the surface of the three-dimensional model of the object, and the occupancy value of 0 indicates that the voxel unit is not occupied by the three-dimensional model of the object, that is, the voxel unit is located outside the three-dimensional model of the object. The coordinates of the voxel unit of each level in the truncated signed distance field stereo are the coordinates of a specific voxel included in the voxel unit in the truncated signed distance field stereo. The truncated signed distance value of the voxel unit of each level is the truncated signed distance value of a specific voxel included in the voxel unit. The above-mentioned specific voxel can be a voxel located at a top corner of the voxel unit, such as the lower left corner or the upper right corner. Therefore, when a particular voxel within a voxel cell lacks a truncated signed distance value, the voxel cell also lacks a truncated signed distance value.

[0051] Based on the above, the following operations are performed in each level: the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value is identified in each level, for example, the first voxel unit whose occupation value is 0 and whose truncated signed distance value is missing is searched in each level, and the truncated signed distance value of the first voxel unit is determined according to the truncated signed distance field stereo of the object, the coordinates of the first voxel unit in the truncated signed distance field stereo, and the distance determination model, and the truncated signed distance value of the first voxel unit is determined. Wherein, since the truncated signed distance field of the first voxel unit is missing, the specific voxel in the first voxel unit must be the second voxel, so the first voxel unit includes the second voxel, the coordinates of the first voxel unit in the truncated signed distance field stereo are the coordinates of the included specific voxel, that is, the second voxel, and the truncated signed distance value of the first voxel unit is the truncated signed distance value of the included specific voxel, that is, the second voxel.

[0052] It can be seen that through this embodiment, when the truncated signed distance field stereo includes multiple levels, each level includes multiple voxel units, for the first voxel unit of each level that is occupied by the three-dimensional model of the object and has a missing truncated signed distance value, the truncated signed distance value of the first voxel unit can be determined according to the truncated signed distance field stereo of the object, the coordinates of the first voxel unit in the truncated signed distance field stereo, and the distance determination model, so that the truncated signed distance value of the first voxel unit can be determined efficiently and quickly through the distance determination model, and the determination result has high accuracy.

[0053] In one embodiment, the above method flow further includes:

[0054] According to the truncated signed distance value of the voxel unit of the Nth level of the truncated signed distance field stereo of the object and the truncated signed distance value of the voxel unit adjacent to the voxel unit of the Nth level, it is determined whether the voxel unit of the Nth level includes the voxel located on the surface of the three-dimensional model of the object; N is an integer greater than or equal to 1 and less than M; M is the total number of levels of the truncated signed distance field stereo; M is an integer;

[0055] If included, the voxel unit of the Nth level is divided into multiple voxel units of the N+1th level according to the octree structure.

[0056] In this embodiment, after the truncated signed distance value of the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value at each level is completed through the previous process, for each voxel unit of the Nth level, based on the truncated signed distance value of the voxel unit and the truncated signed distance values ​​of multiple adjacent voxel units of the voxel unit, for example, the truncated signed distance values ​​of seven adjacent voxel units (the seven adjacent voxel units and the voxel unit can be combined into a large cube), it is determined whether the voxel unit contains voxels located on the surface of the three-dimensional model of the object; if contained, the voxel unit is divided into multiple voxel units of the N+1th level according to the octree structure, for example, the voxel unit is divided into eight voxel units of the N+1th level according to the octree structure.

[0057] It can be seen that through this embodiment, when the voxel unit of the Nth level includes voxels located on the surface of the three-dimensional model of the object, the voxel unit can be divided into multiple voxel units of the N+1th level according to the octree structure, so that the division of the voxel units of each level is realized through the octree algorithm, and the speed of searching for voxels located on the surface of the three-dimensional model of the object in each voxel unit is improved, thereby improving the modeling speed of the three-dimensional model.

[0058] In one embodiment, judging whether the voxel unit at the Nth level includes voxels located on the surface of the three-dimensional model of the object according to the truncated signed distance value of the voxel unit at the Nth level of the truncated signed distance field volume of the object and the truncated signed distance value of the voxel unit adjacent to the voxel unit at the Nth level includes:

[0059] If the truncated signed distance value of the voxel unit at the Nth level has an opposite sign to the truncated signed distance value of any adjacent voxel unit, it is determined that the voxel unit at the Nth level includes voxels located on the surface of the three-dimensional model of the object.

[0060] In this embodiment, for each voxel unit of the Nth level, if the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance value of any adjacent voxel unit of the voxel unit, it is determined that the voxel unit contains voxels located on the surface of the three-dimensional model of the object.

[0061] For example, for a voxel unit at the Nth level, determine whether the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance values ​​of the seven adjacent voxel units of the voxel unit. If it is determined that the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance values ​​of any of the seven adjacent voxel units, it is determined that the voxel unit contains voxels located on the surface of the three-dimensional model of the object.

[0062] It can be seen that through this embodiment, by judging the positive and negative signs of the truncated signed distance value, it is possible to quickly determine whether the voxel unit of the Nth level contains voxels located on the surface of the three-dimensional model of the object. The determination process is fast and the determination result is highly accurate.

[0063] In a specific example, the truncated signed distance field stereo is divided into a plurality of voxel units of the first level, each voxel unit includes at least one voxel, and the occupancy value of each voxel unit is set to the occupancy value of the voxel in the lower left corner included in the voxel unit, the coordinates of each voxel unit in the truncated signed distance field stereo are the coordinates of the voxel in the lower left corner included in the voxel unit, and the truncated signed distance value of each voxel unit is the truncated signed distance value of the voxel in the lower left corner included in the voxel unit. These three rules are applicable to voxel units of each level. Next, the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value is traversed in the first level, and the model is determined according to the truncated signed distance field stereo of the object, the coordinates of the first voxel unit in the truncated signed distance field stereo, and the distance, and the truncated signed distance value of the first voxel unit is determined. After each voxel unit in the first level has a truncated signed distance value, for each voxel unit, based on the truncated signed distance value of the voxel unit and the truncated signed distance values ​​of its seven adjacent voxel units (the seven adjacent voxel units and the voxel unit can be combined into a large cube), it is determined whether the voxel unit contains voxels located on the surface of the three-dimensional model of the object; if so, the voxel unit is divided into eight voxel units of the second level according to the octree structure, thereby obtaining each voxel unit of the second level.

[0064] In the second level, the above process is also repeated until the voxel unit of the Mth level is obtained, and each voxel unit of the Mth level includes only one voxel and cannot be subdivided. In the Mth level, for the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value, the truncated signed distance value of the first voxel unit is determined according to the truncated signed distance field stereo, the coordinates of the first voxel unit in the truncated signed distance field stereo and the distance determination model, and the truncated signed distance value of the first voxel unit is determined. After the truncated signed distance value of each voxel unit is completed, for each voxel unit, if the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance value of any one of the seven adjacent voxel units (the seven adjacent voxel units and the voxel unit can be combined into a large cube), it is determined that the voxel unit contains a voxel located on the surface of the three-dimensional model of the object, that is, the voxel unit is a voxel located on the surface of the three-dimensional model of the object. According to the voxels located on the surface of the three-dimensional model of the object determined in the voxel units of the Mth level, the three-dimensional model of the object can be reconstructed by a surface reconstruction method.

[0065] In one embodiment, the aforementioned determining the truncated signed distance value of the first voxel unit according to the truncated signed distance field volume of the object, the coordinates of the first voxel unit in the truncated signed distance field volume and the distance determination model includes:

[0066] Through the distance determination model, feature extraction is performed on the truncated signed distance field stereo of the object to obtain a feature stereo;

[0067] The truncated signed distance value of the first voxel unit is determined by the distance determination model according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume.

[0068] Figure 2 A schematic diagram of the structure of a distance determination model provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the model includes a feature extractor and a decoder. First, the feature extractor extracts features from the truncated signed distance field stereo of the object to obtain a feature stereo, and then determines the truncated signed distance value of the first voxel unit according to the coordinates of the first voxel unit in the truncated signed distance field stereo and the feature stereo.

[0069] It can be seen that through this embodiment, the truncated signed distance field stereo of the object can be feature extracted through the distance determination model to obtain the feature stereo, and the truncated signed distance value of the first voxel unit can be quickly and accurately determined according to the coordinates of the first voxel unit in the truncated signed distance field stereo and the feature stereo.

[0070] In one embodiment, the feature volume includes a feature value of a truncated signed distance value of each voxel in a truncated signed distance field volume of the object; and determining the truncated signed distance value of the first voxel unit according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume includes:

[0071] According to the coordinates of the first voxel unit in the truncated signed distance field volume, querying the characteristic value of the truncated signed distance value of the first voxel unit in the characteristic volume by linear interpolation;

[0072] The truncated signed distance value of the first voxel unit is determined according to the characteristic value of the truncated signed distance value of the first voxel unit.

[0073] In this embodiment, the feature stereo includes the predicted feature value of the truncated signed distance value of each voxel in the truncated signed distance field stereo of the object, that is, the process of extracting features from the truncated signed distance field stereo of the object by a feature extractor to obtain the feature stereo is a process of predicting the truncated signed distance value of each voxel in the truncated signed distance field stereo of the object based on the truncated signed distance value of the first voxel in the truncated signed distance field stereo of the object and processing the predicted truncated signed distance value into a feature value.

[0074] Based on this, according to the coordinates of the first voxel unit in the truncated signed distance field stereo, the eigenvalue of the truncated signed distance value of the first voxel unit is queried in the feature stereo by linear interpolation. For example, the coordinates of the first voxel unit in the truncated signed distance field stereo are (100, 100, 100), and the corresponding coordinates converted to the feature stereo are (51, 51, 51), but there is no eigenvalue corresponding to the coordinates (51, 51, 51) in the feature stereo. Therefore, multiple coordinates closest to the converted coordinates of the first voxel unit are determined in the feature stereo, and the eigenvalues ​​of the truncated signed distance value corresponding to the multiple coordinates are interpolated to obtain the eigenvalue of the truncated signed distance value of the first voxel unit.

[0075] Then, by Figure 2 The decoder shown decodes the characteristic value of the truncated signed distance value of the first voxel unit to obtain the truncated signed distance value of the first voxel unit.

[0076] It can be seen that through this embodiment, according to the coordinates of the truncated signed distance field stereo of the first voxel unit, the characteristic value of the truncated signed distance value of the first voxel unit can be queried in the feature stereo by linear interpolation, and the truncated signed distance value of the first voxel unit can be determined according to the characteristic value of the truncated signed distance value of the first voxel unit, and the determination speed is fast and the determination result is accurate and reliable.

[0077] In one example, reference Figure 2 The truncated signed distance field stereo of the object input into the distance determination model is a matrix of 256*256*256 size. The matrix is ​​processed by the feature extractor to obtain a feature stereo of 16*16*16 size. According to the coordinates of the first voxel unit in the truncated signed distance field stereo, the characteristic value of the truncated signed distance value of the first voxel unit obtained by linear interpolation in the feature stereo is 128-length data. The 128-length data is decoded into the truncated signed distance value of the first voxel unit by the decoder.

[0078] In the above step S106 , a three-dimensional model of the object is established according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0079] In one embodiment, building a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel includes:

[0080] determining a voxel located on a surface of the three-dimensional model of the object based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel;

[0081] A three-dimensional model of the object is created based on voxels located on the surface of the three-dimensional model of the object.

[0082] According to the foregoing introduction, predicting the truncated signed distance value in the first voxel unit is to predict the truncated signed distance value of a specific voxel in the first voxel unit, and the specific voxel in the first voxel unit is the second voxel. After the voxel units of the last level in the truncated signed distance field stereo of the object are obtained by the above method, each voxel unit of the last level includes only one voxel and cannot be subdivided. In the last level, for the first voxel unit occupied by the three-dimensional model of the object and missing the truncated signed distance value, the first voxel unit is the second voxel. According to the truncated signed distance field stereo, the coordinates and distance determination model of the first voxel unit in the truncated signed distance field stereo, the truncated signed distance value of the first voxel unit is determined. After the truncated signed distance values ​​of each voxel unit of the last level are completed, for any voxel unit, if the positive and negative signs of the truncated signed distance value of the voxel unit are opposite to the positive and negative signs of the truncated signed distance value of any adjacent voxel unit among the seven adjacent voxel units (the seven adjacent voxel units and the voxel unit can be combined into a large cube), it is determined that the voxel unit contains a voxel located on the surface of the three-dimensional model of the object, that is, the voxel unit is a voxel located on the surface of the three-dimensional model of the object. According to the voxels located on the surface of the three-dimensional model of the object determined in the voxel unit of the last level, the three-dimensional model of the object can be reconstructed by a surface reconstruction method. It can be understood that the voxels located on the surface of the three-dimensional model of the object determined in the voxel unit of the last level include the first voxel and the second voxel.

[0083] It can be seen that through this embodiment, the voxels located on the surface of the three-dimensional model of the object can be determined, and the three-dimensional model of the object can be established by surface reconstruction based on the voxels located on the surface of the three-dimensional model of the object, thereby generating a high-precision three-dimensional model in real time in the scenario of generating and transmitting the three-dimensional model in real time.

[0084] Furthermore, it can be seen from the above process that in this embodiment, the truncated signed distance values ​​are not determined for all the second voxels, but the truncated signed distance values ​​in the first voxel units occupied by the model of the object are determined hierarchically according to the hierarchical relationship, and determining the truncated signed distance value of the first voxel unit is equivalent to determining the truncated signed distance value of the second voxel. In addition, the hierarchical relationship is established based on whether the voxel unit includes voxels located on the surface of the three-dimensional model of the object. Therefore, in this embodiment, only the necessary truncated signed distance values ​​of the second voxels need to be completed, which greatly reduces the workload and improves the speed of generating high-precision three-dimensional models in real time.

[0085] In one embodiment, the process of constructing voxel units at each level of the octree can be accelerated by CUDA (Compute Unified Device Architecture). The distance determination model can accelerate the reasoning process by TensorRT (Nvidia reasoning engine acceleration tool).

[0086] The generation process of the above-mentioned distance determination model is introduced below.

[0087] In one embodiment, the above method flow further includes:

[0088] Acquire a sample truncated signed distance field stereo of the sample object; the sample truncated signed distance field stereo is used to represent the physical space where the sample object is located; the sample truncated signed distance field stereo includes a plurality of sample voxels; the sample voxels are used to represent the spatial region in the physical space where the sample object is located; the sample voxels have sample truncated signed distance values; the sample truncated signed distance values ​​represent the distance between the sample voxels and the surface of the three-dimensional model of the sample object;

[0089] The neural network model is trained according to the sample truncated signed distance field stereo, the coordinates of the sample voxels in the sample truncated signed distance field stereo and the sample truncated signed distance values ​​of the sample voxels; the trained neural network model is a distance determination model.

[0090] In this embodiment, the sample object may be an object such as a person, an animal, or a virtual image. The truncated signed distance field stereo of the sample object is a sample truncated signed distance field stereo. The sample truncated signed distance field stereo is used to represent the physical space where the sample object is located. The sample truncated signed distance field stereo includes a plurality of voxels, which are called sample voxels, and each sample voxel is used to represent a spatial region in the physical space where the sample object is located. That is, the sample truncated signed distance field stereo and the sample voxel are the expressions of the physical space where the sample object is located and the spatial region in the physical space in the computer system. Each sample voxel has a truncated signed distance value, and the truncated signed distance value of the sample voxel is called the sample truncated signed distance value. The sample truncated signed distance value represents the distance between the sample voxel and the surface of the three-dimensional model of the sample object.

[0091] For the specific meanings of the sample truncated signed distance field stereo and the sample truncated signed distance value, please refer to the explanation of the truncated signed distance field stereo and truncated signed distance value for the object in the previous text, which will not be repeated here. The sample object is an object used to train the distance determination model, and the sample truncated signed distance field stereo of the sample object and the sample truncated signed distance value of each sample voxel can be calculated by computer simulation.

[0092] In this embodiment, a pre-built neural network model is trained according to the sample truncated signed distance field stereo, the coordinates of the sample voxel in the sample truncated signed distance field stereo, and the sample truncated signed distance value of the sample voxel. The trained neural network model is a distance determination model. The pre-built neural network model includes the following: Figure 2 The process of training a neural network model can be understood as training Figure 2 The feature extractor and decoder in .

[0093] It can be seen that through this embodiment, the distance determination model can be trained according to the sample truncated signed distance field stereo of the sample object, the coordinates of the sample voxel in the sample truncated signed distance field stereo and the sample truncated signed distance value of the sample voxel. The model training efficiency is high, and the distance determination model obtained by training can accurately determine the truncated signed distance of any voxel in the truncated signed distance field stereo of the object.

[0094] In one embodiment, training a neural network model according to a sample truncated signed distance field volume, coordinates of sample voxels in the sample truncated signed distance field volume, and sample truncated signed distance values ​​of the sample voxels comprises:

[0095] Through the neural network model, the sample truncated signed distance field stereo is subjected to feature extraction to obtain the sample feature stereo;

[0096] Determine the predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field stereo and the sample feature stereo through the neural network model;

[0097] A neural network model is trained based on the predicted truncated signed distance values ​​of the sample voxels and the sample truncated signed distance values ​​of the sample voxels.

[0098] In this embodiment, the feature extractor to be trained in the neural network model is used to extract features from the sample truncated signed distance field stereo to obtain the sample feature stereo. The predicted truncated signed distance value of the sample voxel is determined according to the coordinates of the sample voxel in the sample truncated signed distance field stereo and the sample feature stereo, and the predicted truncated signed distance value is the predicted truncated signed distance value of the sample voxel. The neural network model is trained according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel, wherein the predicted truncated signed distance value is the predicted value, and the sample truncated signed distance value is the true value.

[0099] It can be seen that through this embodiment, the predicted truncated signed distance value of the sample voxel can be obtained through the neural network model, and the neural network model can be trained according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel. Among them, the predicted truncated signed distance value is the predicted value, and the sample truncated signed distance value is the true value, so that the neural network model is trained according to the predicted value and the true value, and the accuracy of the distance determination model obtained by training is improved.

[0100] In one embodiment, the sample feature volume includes a feature value of a predicted truncated signed distance value of each sample voxel; and determining the predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field volume and the sample feature volume comprises:

[0101] According to the coordinates of the sample voxel in the sample truncated signed distance field volume, query the characteristic value of the predicted truncated signed distance value of the sample voxel in the sample feature volume by linear interpolation;

[0102] The predicted truncated signed distance value of the sample voxel is determined according to the characteristic value of the predicted truncated signed distance value of the sample voxel.

[0103] In this embodiment, after the sample truncated signed distance value of each sample voxel in the sample truncated signed distance field volume of the sample object is known, feature extraction is performed on the sample truncated signed distance field volume by a feature extractor to be trained.

[0104] Based on this, according to the coordinates of each sample voxel in the sample truncated signed distance field stereo, the eigenvalue of the predicted truncated signed distance value of the sample voxel is queried in the sample feature stereo by linear interpolation. For example, the coordinates of the sample voxel in the sample truncated signed distance field stereo are (100,100,100), and the corresponding coordinates converted to the sample feature stereo are (51,51,51). However, there is no eigenvalue corresponding to the coordinate (51,51,51) in the sample feature stereo. Therefore, multiple coordinates closest to the converted coordinates of the sample voxel are determined in the sample feature stereo, and the eigenvalues ​​of the truncated signed distance values ​​corresponding to the multiple coordinates are interpolated to obtain the eigenvalue of the predicted truncated signed distance value of the sample voxel. Then, by Figure 2 The decoder shown decodes the feature value of the predicted truncated signed distance value of the sample voxel to obtain the predicted truncated signed distance value of the sample voxel.

[0105] It can be seen that through this embodiment, the characteristic value of the predicted truncated signed distance value of the sample voxel can be queried in the sample feature stereo according to the coordinates of the sample voxel in the sample truncated signed distance field stereo by linear interpolation, and the predicted truncated signed distance value of the sample voxel can be determined according to the characteristic value of the predicted truncated signed distance value of the sample voxel, and the determination result is accurate, reliable and efficient.

[0106] In one embodiment, the true value of the sample voxel, that is, the sample truncated signed distance value, includes an observed value and a simulated value; training a neural network model according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel includes:

[0107] Calculating a first loss function, a second loss function, and a third loss function according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel;

[0108] Training a neural network model based on the first loss function, the second loss function, and the calculation results of the third loss function;

[0109] Among them, the first loss function is used to determine the first distance between the simulation value and the predicted truncated signed distance value and the second distance between the observed value and the predicted truncated signed distance value; the second loss function is used to constrain the association between the predicted truncated signed distance value and the occupancy value of the sample voxel; the occupancy value indicates whether the sample voxel is occupied by the three-dimensional model of the sample object; the third loss function is used to perform smoothing constraints on the predicted truncated signed distance value.

[0110] In this embodiment, the true value of the sample voxel, that is, the sample truncated signed distance value, includes an observed value and a simulated value. The simulated value is the most idealized sample truncated signed distance value of the sample voxel obtained by computer simulation, and the observed value is the value obtained by adding noise to the simulated value, and the noise is used to simulate the noise in the real world.

[0111] In this embodiment, there are three loss functions preset, namely, the first, second and third. The first loss function is used to determine the first distance between the simulated value of the sample voxel and the predicted truncated signed distance value of the sample voxel and to determine the second distance between the observed value of the sample voxel and the predicted truncated signed distance value of the sample voxel. In an example, the first loss function is specifically as follows:

[0112]

[0113] In this formula, L recon represents the reconstruction loss, p represents the sample voxel, represents the sample truncated signed distance field stereo, λ consis is the preset coefficient, is the predicted truncated signed distance value of the sample voxel, x *is the simulated value of the sample voxel, ρ is the function for calculating the huber weight, and x is the observed value of the sample voxel.

[0114] The second loss function is used to constrain the correlation between the predicted truncated signed distance value of the sample voxel and the occupancy value of the sample voxel, where the occupancy value indicates whether the sample voxel is occupied by the three-dimensional model of the sample object. The second loss function can make the predicted truncated signed distance value of the sample voxel indicate that the sample voxel is located inside or on the surface of the three-dimensional model of the sample object and is occupied, and make the predicted truncated signed distance value of the sample voxel indicate that the sample voxel is located outside the three-dimensional model of the sample object and is not occupied, and the occupancy value of the sample voxel is 0. In an example, the second loss function is as follows:

[0115]

[0116] Among them, L hull is the visual cone constraint, o is the visual cone value (i.e., occupancy value), o equals 1, indicating that the sample voxel is inside or on the surface of the three-dimensional model of the sample object, and o equals 0, indicating that the sample voxel is outside the three-dimensional model of the sample object, and p represents the sample voxel. represents a sample truncated signed distance field stereo, is the predicted cutoff signed distance value for the sample voxel.

[0117] The third loss function is used to impose a smooth constraint on the predicted truncated signed distance value of the sample voxel. In one example, the third loss function is as follows:

[0118]

[0119] Among them, L TV represents a smoothness constraint, p represents a sample voxel, represents a sample truncated signed distance field stereo, is the predicted truncated signed distance value of the sample voxel, is the TSDF value predicted by sample pixel p With the sample pixel (p x +δ x ,p y ,p z ) with a constant offset δ x The shorthand for the squared difference between and Same reason.

[0120] In this embodiment, the first loss function, the second loss function and the third loss function are calculated according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel, and the neural network model is trained according to the calculation results of the first loss function, the second loss function and the second loss function. In one example, the neural network model can be trained by combining the calculation results of the first loss function, the second loss function and the second loss function through the following formula. In the formula, L represents the total loss of the model, λ TV and λ hull All are preset coefficients.

[0121] L=L recon +λ TV L TV +λ hull L hulnl (4)

[0122] It can be seen that through this embodiment, multiple loss functions are set, and the distance determination model is constrained from different aspects through the multiple loss functions, thereby improving the accuracy of the distance determination model and the accuracy of the three-dimensional model generated in real time.

[0123] In one embodiment, the feature extractor in the distance determination model includes a multi-stage cascaded 3D convolutional network. When training the distance determination model, the Adam optimizer is selected, the training weight is usually set to 1e-4, and the regularization term coefficient is set to 1e-5. The sample object may include a person and an object. The three-dimensional model of the sample object may include a grid model of a specific scene composed of a person and an object. The sample voxel may be a partial voxel in a sample truncated signed distance field stereo. In the sample truncated signed distance field stereo, the position of the model surface close to the sample object is non-uniformly sampled to obtain the sample voxel, and the greater the curvature change of the model surface, the more sample voxels are collected. In one example, when sampling to obtain the sample voxel, the ratio of the number of sample voxels close to the model surface to the number of sample voxels outside the model surface is 3:1. Each time the distance determination model is trained, 5000 to 10000 sample voxels may be collected in the sample truncated signed distance field stereo.

[0124] In summary, a three-dimensional modeling method disclosed in the present embodiment can rely on fewer sensors, such as three sensors to photograph an object from three perspectives to obtain a depth map, and establish a truncated signed distance field stereo of the object based on the depth map. Due to the fewer sensors, the truncated signed distance values ​​of some voxels will be missing in the truncated signed distance field stereo. The truncated signed distance values ​​of the voxels in the truncated signed distance field stereo can be supplemented by the distance determination model, thereby reconstructing the three-dimensional model of the object. The reconstructed three-dimensional model is a watertight model with high precision, and the distance determination model and the octree algorithm are used to make the model reconstruction speed faster, so that the three-dimensional model can be generated and transmitted in a real-time scene.

[0125] An embodiment of the present disclosure also provides a three-dimensional modeling device for implementing the above-mentioned three-dimensional modeling method. Figure 3 A schematic diagram of the structure of a three-dimensional modeling device provided in an embodiment of the present disclosure, such as Figure 3 As shown, the device comprises:

[0126] A stereo acquisition unit 31 is used to acquire a truncated signed distance field stereo of an object; the truncated signed distance field stereo is used to represent the physical space where the object is located; the truncated signed distance field stereo includes a plurality of voxels; the voxels are used to represent a spatial region in the physical space where the object is located; a first voxel among the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0127] a distance determination unit 32, configured to determine, for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, a truncated signed distance value of the second voxel according to the truncated signed distance field volume, the coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0128] The model building unit 33 is used to build a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0129] Optionally, the stereo acquisition unit 31 is specifically used for:

[0130] Projecting a three-dimensional grid for representing the physical space where the object is located to each viewing angle of the object; the three-dimensional grid is divided into the plurality of voxels;

[0131] Determine a truncated signed distance value of the voxel according to the depth data of the voxel at each of the viewing angles and the depth map of the object at each of the viewing angles obtained by projection;

[0132] The voxel determined to obtain the truncated signed distance value is determined as the first voxel, and the three-dimensional grid obtained after projection to obtain the truncated signed distance value is determined as the truncated signed distance field three-dimensional grid.

[0133] Optionally, the truncated signed distance field stereo includes multiple levels; each of the levels includes multiple voxel units; the voxel unit includes at least one voxel; the voxel unit of the lower level is nested in the voxel unit of the higher level; the distance determination unit 32 is specifically used for:

[0134] For a first voxel unit of each level occupied by the three-dimensional model of the object and having a missing truncated signed distance value, determine a truncated signed distance value of the first voxel unit according to the truncated signed distance field volume, the coordinates of the first voxel unit in the truncated signed distance field volume and the distance determination model;

[0135] The first voxel unit includes the second voxel; the coordinates of the first voxel unit in the truncated signed distance field stereo are the coordinates of the included second voxel; and the truncated signed distance value of the first voxel unit is the truncated signed distance value of the included second voxel.

[0136] Optionally, a stratification unit is further included, for:

[0137] According to the truncated signed distance value of the voxel unit of the Nth level of the truncated signed distance field stereo and the truncated signed distance value of the voxel unit adjacent to the voxel unit of the Nth level, it is determined whether the voxel unit of the Nth level includes the voxel located on the surface of the three-dimensional model of the object; N is an integer greater than or equal to 1 and less than M; M is the total number of levels of the truncated signed distance field stereo;

[0138] If included, the voxel unit of the Nth level is divided into multiple voxel units of the N+1th level according to the octree structure.

[0139] Optionally, the stratification unit is specifically used for:

[0140] If the positive and negative signs of the truncated signed distance values ​​of the voxel units of the Nth level are opposite to the positive and negative signs of the truncated signed distance values ​​of any of the adjacent voxel units, it is determined that the voxel units of the Nth level include voxels located on the surface of the three-dimensional model of the object.

[0141] Optionally, the distance determining unit 32 is further specifically configured to:

[0142] By using the distance determination model, feature extraction is performed on the truncated signed distance field stereo to obtain a feature stereo;

[0143] The truncated signed distance value of the first voxel unit is determined by the distance determination model according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume.

[0144] Optionally, the feature volume includes a feature value of a truncated signed distance value of each voxel in the plurality of voxels; the distance determination unit 32 is further specifically configured to:

[0145] According to the coordinates of the first voxel unit in the truncated signed distance field volume, querying the characteristic value of the truncated signed distance value of the first voxel unit in the characteristic volume by linear interpolation;

[0146] The truncated signed distance value of the first voxel unit is determined according to the characteristic value of the truncated signed distance value of the first voxel unit.

[0147] Optionally, the model building unit 33 is specifically used for:

[0148] Determining a voxel located on a surface of the three-dimensional model of the object based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel;

[0149] A three-dimensional model of the object is established based on voxels located on a surface of the three-dimensional model of the object.

[0150] Optionally, a model training unit is further included, for:

[0151] Acquire a sample truncated signed distance field stereo of a sample object; the sample truncated signed distance field stereo is used to represent the physical space where the sample object is located; the sample truncated signed distance field stereo includes a plurality of sample voxels; the sample voxels are used to represent the spatial region in the physical space where the sample object is located; the sample voxels have sample truncated signed distance values; the sample truncated signed distance values ​​represent the distance between the sample voxels and the surface of the three-dimensional model of the sample object;

[0152] A neural network model is trained according to the sample truncated signed distance field stereo, the coordinates of the sample voxels in the sample truncated signed distance field stereo and the sample truncated signed distance values ​​of the sample voxels; the trained neural network model is the distance determination model.

[0153] Optionally, the model training unit is specifically used for:

[0154] By using the neural network model, feature extraction is performed on the sample truncated signed distance field stereo to obtain a sample feature stereo;

[0155] Determining, by the neural network model, a predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field stereo and the sample feature stereo;

[0156] A neural network model is trained according to the predicted truncated signed distance values ​​of the sample voxels and the sample truncated signed distance values ​​of the sample voxels.

[0157] Optionally, the sample feature volume includes a feature value of a predicted truncated signed distance value of each sample voxel; and the model training unit is further specifically used for:

[0158] According to the coordinates of the sample voxel in the sample truncated signed distance field volume, querying the characteristic value of the predicted truncated signed distance value of the sample voxel in the sample feature volume by linear interpolation;

[0159] The predicted truncated signed distance value of the sample voxel is determined according to the characteristic value of the predicted truncated signed distance value of the sample voxel.

[0160] Optionally, the sample truncated signed distance value of the sample voxel includes an observed value and a simulated value; and the model training unit is further specifically used for:

[0161] Calculating a first loss function, a second loss function, and a third loss function according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel;

[0162] Training a neural network model based on the first loss function, the second loss function, and the calculation results of the second loss function;

[0163] Among them, the first loss function is used to determine the first distance between the simulation value and the predicted truncated signed distance value and the second distance between the observed value and the predicted truncated signed distance value; the second loss function is used to constrain the association between the predicted truncated signed distance value and the occupancy value of the sample voxel; the occupancy value indicates whether the sample voxel is occupied by the three-dimensional model of the sample object; the third loss function is used to perform smoothing constraints on the predicted truncated signed distance value.

[0164] The three-dimensional modeling device in the embodiment of the present disclosure can implement each process of the above-mentioned three-dimensional modeling method embodiment and achieve the same effects and functions, which will not be repeated here.

[0165] An embodiment of the present disclosure further provides an electronic device, Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Figure 4As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 401 and memory 402, and one or more applications or data may be stored in the memory 402. Among them, the memory 402 may be a temporary storage or a permanent storage. The application stored in the memory 402 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 401 may be configured to communicate with the memory 402 to execute a series of computer executable instructions in the memory 402 on the electronic device. The electronic device may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input or output interfaces 405, one or more keyboards 406, etc.

[0166] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0167] Acquire a truncated signed distance field stereo of the object; the truncated signed distance field stereo is used to represent the physical space where the object is located; the truncated signed distance field stereo includes a plurality of voxels; the voxels are used to represent a spatial region in the physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0168] For a second voxel in the plurality of voxels for which the truncated signed distance value is missing, determining a truncated signed distance value for the second voxel according to the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0169] A three-dimensional model of the object is established according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0170] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned three-dimensional modeling method embodiment and achieve the same effects and functions, which will not be repeated here.

[0171] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0172] Acquire a truncated signed distance field stereo of the object; the truncated signed distance field stereo is used to represent the physical space where the object is located; the truncated signed distance field stereo includes a plurality of voxels; the voxels are used to represent a spatial region in the physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object;

[0173] For a second voxel in the plurality of voxels for which the truncated signed distance value is missing, determining a truncated signed distance value for the second voxel according to the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value;

[0174] A three-dimensional model of the object is established according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

[0175] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned three-dimensional modeling method embodiment and achieve the same effects and functions, which will not be repeated here.

[0176] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0177] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0178] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0179] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0180] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0181] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0185] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0186] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0187] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0188] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A three-dimensional modeling method, characterized in that: include: Acquire a truncated signed distance field stereo of the object; the truncated signed distance field stereo is used to represent the physical space where the object is located; The truncated signed distance field volume comprises a plurality of voxels; the voxels are used to represent a spatial region in a physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object; For a second voxel in the plurality of voxels for which the truncated signed distance value is missing, determining a truncated signed distance value for the second voxel according to the truncated signed distance field volume, coordinates of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value; A three-dimensional model of the object is established according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

2. The method according to claim 1, characterized in that The obtaining of a truncated signed distance field stereo of the object comprises: Projecting a three-dimensional grid for representing the physical space where the object is located to each viewing angle of the object; the three-dimensional grid is divided into the plurality of voxels; Determine a truncated signed distance value of the voxel according to the depth data of the voxel at each of the viewing angles and the depth map of the object at each of the viewing angles obtained by projection; The voxel determined to obtain the truncated signed distance value is determined as the first voxel, and the three-dimensional grid obtained after projection to obtain the truncated signed distance value is determined as the truncated signed distance field three-dimensional grid.

3. The method according to claim 1, characterized in that The truncated signed distance field volume comprises a plurality of levels; each of the levels comprises a plurality of voxel units; the voxel unit comprises at least one voxel; the voxel unit of a lower level is nested in the voxel unit of a higher level; and the truncated signed distance value of the second voxel is determined according to the truncated signed distance field volume, the coordinates of the second voxel in the truncated signed distance field volume and the distance determination model for determining the truncated signed distance value, comprising: For a first voxel unit of each level occupied by the three-dimensional model of the object and having a missing truncated signed distance value, determine a truncated signed distance value of the first voxel unit according to the truncated signed distance field volume, the coordinates of the first voxel unit in the truncated signed distance field volume and the distance determination model; The first voxel unit includes the second voxel; the coordinates of the first voxel unit in the truncated signed distance field stereo are the coordinates of the included second voxel; and the truncated signed distance value of the first voxel unit is the truncated signed distance value of the included second voxel.

4. The method according to claim 3, characterized in that The method further comprises: According to the truncated signed distance value of the voxel unit of the Nth level of the truncated signed distance field stereo and the truncated signed distance value of the voxel unit adjacent to the voxel unit of the Nth level, it is determined whether the voxel unit of the Nth level includes the voxel located on the surface of the three-dimensional model of the object; N is an integer greater than or equal to 1 and less than M; M is the total number of levels of the truncated signed distance field stereo; If included, the voxel unit of the Nth level is divided into multiple voxel units of the N+1th level according to the octree structure.

5. The method according to claim 4, characterized in that The step of judging, based on the truncated signed distance value of the voxel unit of the Nth level of the truncated signed distance field stereo and the truncated signed distance value of the voxel unit adjacent to the voxel unit of the Nth level, whether the voxel unit of the Nth level includes voxels located on the surface of the three-dimensional model of the object, comprises: If the positive and negative signs of the truncated signed distance values ​​of the voxel units of the Nth level are opposite to the positive and negative signs of the truncated signed distance values ​​of any of the adjacent voxel units, it is determined that the voxel units of the Nth level include voxels located on the surface of the three-dimensional model of the object.

6. The method according to claim 3, characterized in that: The determining the truncated signed distance value of the first voxel unit according to the truncated signed distance field volume, the coordinates of the first voxel unit in the truncated signed distance field volume and the distance determination model comprises: By using the distance determination model, feature extraction is performed on the truncated signed distance field stereo to obtain a feature stereo; The truncated signed distance value of the first voxel unit is determined by the distance determination model according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume.

7. The method according to claim 6, characterized in that The feature volume includes a feature value of a truncated signed distance value of each voxel in the plurality of voxels; and determining the truncated signed distance value of the first voxel unit according to the coordinates of the first voxel unit in the truncated signed distance field volume and the feature volume, comprises: According to the coordinates of the first voxel unit in the truncated signed distance field volume, querying the characteristic value of the truncated signed distance value of the first voxel unit in the characteristic volume by linear interpolation; The truncated signed distance value of the first voxel unit is determined according to the characteristic value of the truncated signed distance value of the first voxel unit.

8. The method according to claim 1, characterized in that The step of establishing the three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel comprises: determining a voxel located on a surface of the three-dimensional model of the object based on the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel; A three-dimensional model of the object is established based on voxels located on a surface of the three-dimensional model of the object.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire a sample truncated signed distance field stereo of a sample object; the sample truncated signed distance field stereo is used to represent the physical space where the sample object is located; the sample truncated signed distance field stereo includes a plurality of sample voxels; the sample voxels are used to represent the spatial region in the physical space where the sample object is located; the sample voxels have sample truncated signed distance values; the sample truncated signed distance values ​​represent the distance between the sample voxels and the surface of the three-dimensional model of the sample object; A neural network model is trained according to the sample truncated signed distance field stereo, the coordinates of the sample voxels in the sample truncated signed distance field stereo and the sample truncated signed distance values ​​of the sample voxels; the trained neural network model is the distance determination model.

10. The method according to claim 9, characterized in that The training of a neural network model according to the sample truncated signed distance field stereo, the coordinates of the sample voxels in the sample truncated signed distance field stereo and the sample truncated signed distance values ​​of the sample voxels comprises: By using the neural network model, feature extraction is performed on the sample truncated signed distance field stereo to obtain a sample feature stereo; Determining, by the neural network model, a predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field stereo and the sample feature stereo; A neural network model is trained according to the predicted truncated signed distance values ​​of the sample voxels and the sample truncated signed distance values ​​of the sample voxels.

11. The method according to claim 10, characterized in that The sample feature volume includes a feature value of a predicted truncated signed distance value of each sample voxel; and determining the predicted truncated signed distance value of the sample voxel according to the coordinates of the sample voxel in the sample truncated signed distance field volume and the sample feature volume, comprises: According to the coordinates of the sample voxel in the sample truncated signed distance field volume, querying the characteristic value of the predicted truncated signed distance value of the sample voxel in the sample feature volume by linear interpolation; The predicted truncated signed distance value of the sample voxel is determined according to the characteristic value of the predicted truncated signed distance value of the sample voxel.

12. The method according to claim 10, characterized in that The sample truncated signed distance value of the sample voxel includes an observed value and a simulated value; and training a neural network model according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel includes: Calculating a first loss function, a second loss function, and a third loss function according to the predicted truncated signed distance value of the sample voxel and the sample truncated signed distance value of the sample voxel; Training a neural network model based on the first loss function, the second loss function, and the calculation results of the second loss function; Among them, the first loss function is used to determine the first distance between the simulation value and the predicted truncated signed distance value and the second distance between the observed value and the predicted truncated signed distance value; the second loss function is used to constrain the association between the predicted truncated signed distance value and the occupancy value of the sample voxel; the occupancy value indicates whether the sample voxel is occupied by the three-dimensional model of the sample object; the third loss function is used to perform smoothing constraints on the predicted truncated signed distance value.

13. A three-dimensional modeling device, characterized in that: include: A stereo acquisition unit, used to acquire a truncated signed distance field stereo of the object; the truncated signed distance field stereo is used to represent the physical space where the object is located; The truncated signed distance field volume comprises a plurality of voxels; the voxels are used to represent a spatial region in a physical space where the object is located; a first voxel of the plurality of voxels has a truncated signed distance value; the truncated signed distance value represents a distance between the voxel and a surface of a three-dimensional model of the object; a distance determination unit, configured to determine, for a second voxel in the plurality of voxels for which the truncated signed distance value is missing, a truncated signed distance value of the second voxel according to the truncated signed distance field volume, a coordinate of the second voxel in the truncated signed distance field volume, and a distance determination model for determining the truncated signed distance value; The model building unit is used to build a three-dimensional model of the object according to the truncated signed distance value of the first voxel and the determined truncated signed distance value of the second voxel.

14. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 12.

Images