Three-dimensional reconstruction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a three-dimensional reconstruction method and device, electronic equipment and a storage medium, and the method comprises the steps: constructing a symbol distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image; determining a three-dimensional model of the target object according to the constructed symbol distance field model; wherein the loss function used in the sign distance field model construction process comprises a first loss function used for determining curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small. The probability that holes appear on the surface of the reconstructed three-dimensional model can be effectively reduced.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a three-dimensional reconstruction method, device, electronic device, and storage medium. Background Art

[0002] 3D reconstruction is a widely used computer vision task, and 3D reconstruction based on the Signed Distance Field (SDF) model is one of the main reconstruction methods. However, when using the existing SDF model to build, the output 3D model surface often has holes, which will seriously affect subsequent mapping and display operations. Therefore, a 3D reconstruction method is urgently needed to solve the hole problem on the surface of the reconstructed 3D model. Summary of the invention

[0003] The embodiments of the present disclosure provide a three-dimensional reconstruction method, device, electronic device and storage medium, which can effectively reduce the probability of holes appearing on the surface of a reconstructed three-dimensional model.

[0004] In a first aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction method, comprising:

[0005] Constructing a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image;

[0006] Determining a three-dimensional model of the target object according to the constructed signed distance field model;

[0007] Among them, the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

[0008] In a second aspect, the present disclosure also provides a three-dimensional reconstruction device, including:

[0009] A construction module, used for constructing a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image;

[0010] A reconstruction module, used to determine the three-dimensional model of the target object according to the constructed signed distance field model;

[0011] Among them, the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

[0012] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0013] one or more processors;

[0014] a storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement a three-dimensional reconstruction method as described in any one of the embodiments of the present disclosure.

[0016] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to perform a three-dimensional reconstruction method as described in any one of the embodiments of the present disclosure.

[0017] The technical solution of the disclosed embodiment is to construct a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image; determine the three-dimensional model of the target object according to the constructed signed distance field model; wherein the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small. By first using a function with a larger curvature loss, the smoothness of the reconstructed three-dimensional model can be improved, and the probability of holes appearing on the surface of the reconstructed three-dimensional model can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0019] Figure 1 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of a video acquisition interface in a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of a virtual studio in a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of a framework for three-dimensional model reconstruction in a three-dimensional reconstruction method provided in an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of the structure of a three-dimensional reconstruction device provided by an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] Figure 1 A flowchart of a three-dimensional reconstruction method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case where three-dimensional reconstruction is performed using a signed distance field model. The method can be performed by a three-dimensional reconstruction device, which can be implemented in the form of software and / or hardware, and the device can be configured in an electronic device, such as a server.

[0034] like Figure 1 As shown, a three-dimensional reconstruction method provided in this embodiment may include:

[0035] S110, constructing a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image;

[0036] S120, determining a three-dimensional model of the target object according to the constructed signed distance field model;

[0037] Among them, the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

[0038] The Signed Distance Field (SDF) model is a way to represent the geometric information of a three-dimensional space. For a point in space, a signed distance value is defined, and the positive / negative sign in the signed distance value can represent whether the point is inside / outside the three-dimensional model, and the distance value can represent the distance between the point and the nearest edge point of the three-dimensional model. The surface of the three-dimensional model can be determined based on the part of the signed distance value of 0 output by the constructed signed distance field model, thereby determining the three-dimensional model in the space.

[0039] In this embodiment, the target object may be considered to be an object for which a three-dimensional model needs to be constructed, and the sample image containing the target object may present complete surface information of the target object.

[0040] In some optional implementations, before constructing the signed distance field model, the process may further include: acquiring a video frame containing the target object, and selecting a sample image from the video frame.

[0041] The 3D reconstruction device may provide a sample image acquisition interface, such as a photo acquisition interface or a video acquisition interface, to acquire target objects from different perspectives in all directions. Figure 2A schematic diagram of a video acquisition interface in a 3D reconstruction method provided by an embodiment of the present disclosure. Figure 2 , the user can be instructed to collect the target object at each viewing angle by means of a surrounding mask ( Figure 2 In this example, a soccer ball is shown in the figure to obtain complete surface information of the target object. Moreover, after the acquisition is completed at a certain viewing angle, the sub-enclosure area in the enclosing mask corresponding to the viewing angle can be distinguished from the sub-enclosure area of ​​the uncaptured viewing angle. For example, the sub-enclosure area corresponding to the acquired area can be transparent green, and the sub-enclosure area corresponding to the uncaptured area can be colorless. Figure 2 An example of an implementation method of a sub-enclosing area is shown in FIG. 1 , and other implementation methods may also be applied here, which are not exhaustive.

[0042] After the video acquisition is completed, the video frames can be obtained based on existing methods, and the video frames can be selected to obtain sample images. For example, the sample images can be selected based on at least one of the following methods: a video frame can be selected as a sample image every preset number of frames / time; for example, a frame can be selected as a sample image from each video frame with a common view relationship; for example, the sample video frame can be selected based on at least one indicator such as clarity, brightness, noise, etc. of the content of the video frame. When the change in the camera position corresponding to the video frame is less than the preset distance, it can be considered that the video frames have a common view relationship.

[0043] In these optional implementations, by acquiring video frames and selecting sample images, reconstruction of a three-dimensional model based on the video can be achieved.

[0044] In this embodiment, the camera pose of each sample image can be considered as the camera extrinsic parameters corresponding to each sample image (i.e., the rotation parameters and translation parameters of the camera). Among them, the existing pose solution method, such as the Structure from Motion (SfM) method, can be used to determine the camera pose corresponding to each sample image. Among them, the pixel coordinates in each sample image can be projected into the world coordinate system based on the camera parameters to obtain the sparse point cloud data corresponding to each sample image. Furthermore, the world coordinates of the pixel points in the sample image can be used to construct the signed distance field model.

[0045] In the process of constructing the traditional signed distance field model, there is no effective smoothing constraint on the three-dimensional model, which makes the three-dimensional model prone to holes. In the embodiment of the present disclosure, a first loss function for determining the curvature loss between the three-dimensional model and the target object can be used in the process of constructing the signed distance field model; and in the process of constructing from front to back, the weight of the first loss function is adjusted from large to small.

[0046] Wherein, adjusting the weight of the first loss function from large to small during the construction process may include, for example: setting a larger weight of the first loss function during the first preset number of iterations during the construction process, and gradually reducing the weight of the first loss function to zero during the middle preset number of iterations. Wherein, the first preset number of times and the middle preset number of times may be preset based on experiments or experience. Exemplarily, when the number of iterations of the signed distance field model and the color estimation model is 20,000 times, a first loss function with a larger weight may be set during the first 5,000 iterations, for example, the weight may be set to 1; and the weight of the first loss function may be linearly reduced to 0 during the middle 5,000-10,000 iterations.

[0047] By using a larger weight of the first loss function in the initial and intermediate stages of the signed distance field model construction, the curvature loss can play a constraining role in the reconstruction of the 3D model, improve the smoothness of the 3D model, and reduce the probability of holes on the surface of the 3D model, which is beneficial for subsequent mapping and display operations.

[0048] Exemplarily, the first loss function can be designed as follows:

[0049]

[0050] Among them, n can represent the normal information of the sampling point x, n ε Each point x in the neighborhood of the sampling point can be identified ε Normal information of ε =x+εη, εη can represent the neighborhood radius, and ε is an adjustable coefficient. Among them, ε can also be set to a larger value in the first preset number of iterations, and the weight of the first loss function gradually decreases in the middle preset number of iterations. Exemplarily, when the number of iterations of the signed distance field model and the color estimation model is 20,000 times, ε can be set to 0.001 in the first 5,000 iterations; ε can be linearly reduced to 0.0001 in the middle 5,000-10,000 iterations. By first performing curvature fitting in a large neighborhood and then performing curvature fitting in a small neighborhood, the accuracy of constructing the three-dimensional model can be improved.

[0051] In some optional implementations, the signed distance field model construction process may further include: updating parameters of the signed distance field model using a sliding window.

[0052] In the traditional signed distance field model construction process, the signed distance field model parameters (such as hash coding parameters) are often updated step by step or all signed distance field model parameters are directly updated. Updating the signed distance field model parameters step by step will result in a longer construction time, and directly updating all signed distance field model parameters will make it more likely for holes to appear on the surface of the 3D model.

[0053] In these optional implementations, during the construction of the signed distance field model, the signed distance field model may be updated once for every predetermined number of iterations. In each update process, the signed distance field model may be updated with multiple levels of parameters by using a sliding window, which can reduce the appearance of holes on the surface of the three-dimensional model while ensuring construction efficiency. Exemplarily, assuming that the number of network layers of the signed distance field model is 16, in each update process, the sliding window method may be used to update the network parameters of 4 layers of the signed distance field model at a time.

[0054] In some optional implementations, after determining the three-dimensional model of the target object, the method may further include: editing at least one of the following parameters of the three-dimensional model: the shape of the three-dimensional model, and the material and texture of the mesh.

[0055] In these optional implementations, the determined 3D model can be converted into a triangular patch model, and the 3D model can be edited again through existing 3D model editing tools, such as editing at least part of the geometric parameters in the 3D model to change the shape of the 3D model; or editing the material and texture of at least part of the mesh in the 3D model. This can meet the editing needs of users and present rich and diverse 3D model rendering effects.

[0056] In some optional implementations, the three-dimensional reconstruction method can also be applied to the field of virtual broadcasting; wherein the target object may include at least one of the following: a broadcast scene object, a broadcast character object, and a broadcast content object.

[0057] For example, Figure 3 A schematic diagram of a virtual studio in a 3D reconstruction method provided by an embodiment of the present disclosure. Figure 3 , the target objects to be constructed of the 3D model can be, for example, the scene objects of the studio (such as sofas, green plants and other scene objects), the character objects (such as virtual hosts, etc.) and the content objects (such as the entity objects corresponding to the content being reported by the host, etc.). For example, when the virtual studio is used for live streaming, the content objects can be the commodity objects being introduced (such as footballs), so that the 3D model of the commodity can be presented in all directions in the virtual studio, and the 3D model can be suspended in the air during the presentation process to achieve a more technological display.

[0058] In these optional implementations, by reconstructing the three-dimensional models of scene objects, character objects, and presentation content objects in the virtual studio, it is possible to render a presentation effect with excellent audio-visual experience and improve the audience's viewing experience. In addition, the three-dimensional reconstruction method can also be applied to other fields, such as the game field, virtual reality field, and augmented reality field, etc., which are not exhaustive here.

[0059] The technical solution of the disclosed embodiment is to construct a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image; determine the three-dimensional model of the target object according to the constructed signed distance field model; wherein the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the process of constructing from front to back, the weight of the first loss function is adjusted from large to small. By first using a function with a larger curvature loss during the construction process, the smoothness of the reconstructed three-dimensional model can be improved, and the probability of holes appearing on the surface of the reconstructed three-dimensional model can be reduced.

[0060] The various optional schemes in the three-dimensional reconstruction method provided in the embodiments of the present disclosure and the above embodiments can be combined. The three-dimensional reconstruction method provided in this embodiment describes the process of building a signed distance field model in detail. By jointly constructing a signed distance field model and a color estimation model, the two can be combined to determine a three-dimensional model with color information of the target object; by first using a function with a smaller color loss in the joint construction process, a smooth three-dimensional reconstruction result can be obtained in the early stage of construction, which is conducive to reducing the occurrence of holes on the surface of the three-dimensional model. In addition, by normalizing the color estimation model in the later stage of training, the learning information of the signed distance field model can be enhanced, thereby improving the details of the reconstructed three-dimensional model.

[0061] In a three-dimensional reconstruction method provided by this embodiment, a signed distance field model and a color estimation model are jointly constructed, and a three-dimensional model of a target object with color information is determined based on the signed distance field model and the color estimation model; wherein the loss function used in the process of constructing the signed distance field model also includes a second loss function for determining the color loss between the three-dimensional model and the target object; and in the process of constructing from front to back, the weight of the second loss function is adjusted from small to large.

[0062] For example, Figure 4 A schematic diagram of a framework for 3D model reconstruction in a 3D reconstruction method provided by an embodiment of the present disclosure. Figure 4As shown, the signed distance field model can determine the geometric information of the three-dimensional model, and the color estimation model can determine the color information of the three-dimensional model. According to the output of the signed distance field model, a colorless three-dimensional model (which can be called a white model) can be constructed, and the color estimation model can estimate the color of each position in the three-dimensional model, and render the three-dimensional model to obtain a three-dimensional model with color information.

[0063] In the process of jointly constructing the signed distance field model and the color estimation model, the color loss can be calculated based on the color value of each position in the three-dimensional model with color information and the color value of the corresponding position of the target object in the sample image, and the signed distance field model and the color estimation model can be optimized based on the color loss. Among them, the second loss function used to determine the color loss includes but is not limited to the mean square error (MSE) function of the three color channels (Red Green Blue, RGB), and other commonly used image loss functions can also be applied here. By constraining the signed distance field model and the color estimation model through the second loss function, the pixel value of the reconstructed three-dimensional model can be made as close as possible to the pixel value of the target object in the original image.

[0064] In the process of constructing the signed distance field model and the color estimation model from front to back, the weight of the second loss function is adjusted from small to large. Wherein, adjusting the weight of the second loss function from small to large during the construction process may include, for example: setting a smaller weight of the second loss function during the second preset number of iterations during the construction process, and setting a larger weight of the second loss function during the subsequent iterations. Wherein, the second preset number of times can be preset based on experiments or experience. Exemplarily, when the number of iterations of the signed distance field model and the color estimation model is 20,000 times, a second loss function with a smaller weight can be set in the first 2,000 times, for example, set to 1; in the subsequent iterations, it can be adjusted from small to large, for example, it can be set to 10.

[0065] By using a function with smaller color loss in the construction process, a smooth three-dimensional reconstruction result can be obtained in the early stage of construction, which is helpful to reduce the occurrence of holes on the surface of the three-dimensional model.

[0066] In some optional implementations, the process of constructing the signed distance field model and the color estimation model may include: updating the three-dimensional model with color information in response to the normalization processing of the color estimation model; and reversely adjusting the parameters of the signed distance field model according to the updated three-dimensional model with color information.

[0067] In the process of constructing the signed distance field model and the color estimation model, the color estimation model can be normalized in the pre-set number of iterations to enhance the learning information of the signed distance field model, so that the reconstructed three-dimensional model can be further improved in detail on the basis of being more accurate. In the pre-set number of iterations, the color estimation model can be normalized in the first 17,500 iterations to the 20,000 iterations.

[0068] The color estimation model may be normalized by using an existing model parameter normalization method. For example, the color estimation model may be normalized based on the following method:

[0069] For the i-th layer neural network y=σ(W i x+b i ) and the updateable parameter Lipschitz upper bound k i ,use Replace the original W i ,Right now:

[0070]

[0071] in, And the function m(.) can be realized by rescaling W i Each row of is used to normalize the weight matrix. Among them, the relevant loss function can be designed as Where l represents the number of neural network layers of the color estimation model.

[0072] By normalizing the color estimation model, the influence of some illumination changes on the 3D model can be eliminated, which is beneficial to improving the accuracy of the 3D model. After the color estimation model is normalized, the 3D model with color information can be updated, so that the loss can be calculated based on the color information of the updated 3D model and the color value of the corresponding position in the sample image, and then the signed distance field model can be optimized based on the loss. In this way, the learning information of the signed distance field model can be enhanced, thereby improving the details of the reconstructed 3D model.

[0073] The technical solution of the disclosed embodiment describes in detail the process of constructing the signed distance field model. By jointly constructing the signed distance field model and the color estimation model, the two can be combined to determine a three-dimensional model with color information of the target object; by first using a function with a smaller color loss in the joint construction process, a smooth three-dimensional reconstruction result can be obtained in the early stage of construction, which is conducive to reducing the occurrence of holes on the surface of the three-dimensional model. In addition, by normalizing the color estimation model in the later stage of training, the learning information of the signed distance field model can be enhanced, thereby improving the details of the reconstructed three-dimensional model. The three-dimensional reconstruction method provided in the disclosed embodiment and the three-dimensional reconstruction method provided in the above embodiment belong to the same disclosed concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0074] The various optional schemes in the three-dimensional reconstruction method provided in the embodiment of the present disclosure and the above-mentioned embodiment can be combined. The three-dimensional reconstruction method provided in the present embodiment optimizes the construction process of the signed distance field model. By using a phased construction method, a simplified signed distance field model can be used in the first phase to achieve rapid convergence and calculate the bounding box where the target object is located. In the second phase, the bounding box obtained in the first phase can be used to guide the sampling and construction of the original signed distance field model, thereby improving the construction speed of the signed distance field model and accelerating the network convergence.

[0075] Figure 5 A schematic diagram of a three-dimensional reconstruction method provided by an embodiment of the present disclosure. Figure 5 As shown, a three-dimensional reconstruction method provided in this embodiment may include:

[0076] S510: Simplify the signed distance field model to obtain a simplified model.

[0077] Among them, the depth and / or width of the signed distance field model can be simplified by using the existing model simplification method; wherein the depth of the signed distance field model can be understood as the number of neural network layers, and the width can be understood as the resolution size of each layer of the neural network. For example, the depth of the signed distance field model can be compressed to a first preset multiple and / or the width can be compressed to a second preset multiple, wherein the first / second preset multiple is less than 1, thereby obtaining a simplified model with a smaller parameter amount. Among them, the first preset multiple and the second preset multiple can be preset based on experience or experiments.

[0078] Exemplarily, when the signed distance field model includes 16 neural network layers, the depth of the signed distance field model can be compressed to one quarter of the original depth, that is, simplified to a simplified model including 4 neural network layers.

[0079] S520: Determine a plane in the sample image according to the sample image and the camera pose.

[0080] Among them, the camera pose can be considered as the camera external parameter, and the sparse point cloud data corresponding to the sample image can be determined according to the sample image and the camera parameters. Furthermore, the existing plane detection algorithm (such as the RANSAC algorithm) can be used to detect the plane in the point cloud, and the point cloud above the plane can be considered as the target object area, which is conducive to estimating the rough position of the target object, and determining a larger range of the target object based on the rough position. Bounding boxes of larger ranges in each sample image can be aligned (for example, the center of the large bounding box is moved to the origin of the world coordinate system), and the distance from the point cloud in the larger range of the bounding box to its edge is normalized, so that the camera normalization processing can be realized, that is, the pixels of different sample images can be converted to the same camera coordinates, which can facilitate the implementation of image processing and computer vision algorithms.

[0081] There is no strict time sequence relationship between steps S510 and S520.

[0082] S530: construct a simplified model based on the sample image, camera pose and plane.

[0083] In this embodiment, the construction of the simplified model can be considered as the construction of the first stage. In the process of constructing the simplified model, a smaller sampling amount and / or number of iterations can be set compared to the signed distance model. For example, the sampling amount of the signed distance field model can be compressed to a third preset multiple and / or the number of iterations can be compressed to a fourth preset multiple, wherein the third / fourth preset multiple is less than 1. The third preset multiple and the fourth preset multiple can be preset based on experience or experiments.

[0084] Exemplarily, when the maximum number of ray samplings of the signed distance field model is 8192 and the number of iterations is 20,000, the sampling amount of the signed distance field model can be compressed to one sixteenth of the original, and the number of iterations can be compressed to one tenth of the original, that is, the maximum number of ray samplings of the simplified model can be set to 512, and the number of iterations can be set to 2000.

[0085] By using smaller simplified model parameters, smaller sampling rate and number of iterations, it is possible to facilitate rapid convergence of the simplified model.

[0086] S540: Determine a bounding box of the target object according to the constructed simplified model.

[0087] In this embodiment, a rough three-dimensional model of the target object can be output based on the constructed simplified model, and then the topology can be performed outwards on this basis to obtain a smaller bounding box that can compactly surround the target object.

[0088] S550: construct a signed distance field model based on the sample image, camera pose and bounding box.

[0089] In this embodiment, the construction of the signed distance field model can be considered as the construction of the second stage. The bounding box determined by the simplified model construction in the first stage can be used to guide the sampling of the signed distance field model construction in the second stage. Therefore, sampling and construction can be performed only within a small range where the target object is located, which can improve the construction speed and accelerate the network convergence.

[0090] S560: Determine a three-dimensional model of the target object according to the constructed signed distance field model.

[0091] Among them, the loss function used in the process of building the signed distance field model includes a curvature loss function, and the weight of the loss function is adjusted from small to large.

[0092] In the process of constructing the traditional signed distance field model, the region of interest is not well calculated, resulting in a large optimization range of the signed distance field model, a long construction time and difficulty in effective convergence.

[0093] The technical solution of the disclosed embodiment optimizes the construction process of the signed distance field model. By using a phased construction method, a streamlined signed distance field model can be used in the first stage to achieve rapid convergence and calculate the bounding box where the target object is located. In the second stage, the bounding box obtained in the first stage can be used to guide the sampling and construction of the original signed distance field model, thereby improving the construction speed of the signed distance field model and accelerating network convergence. A three-dimensional reconstruction method provided in the disclosed embodiment and a three-dimensional reconstruction method provided in the above embodiment belong to the same disclosed concept. The technical details not described in detail in this embodiment can be found in the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0094] Figure 6 The 3D reconstruction device provided by the embodiment of the present disclosure is a schematic diagram of the structure of the 3D reconstruction device provided by the embodiment of the present disclosure. The 3D reconstruction device provided by the embodiment of the present disclosure is applicable to the case of performing 3D reconstruction through a signed distance field model.

[0095] like Figure 6 As shown, a three-dimensional reconstruction device provided by an embodiment of the present disclosure may include:

[0096] A construction module 610, configured to construct a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image;

[0097] A reconstruction module 620 is used to determine a three-dimensional model of the target object according to the constructed signed distance field model;

[0098] Among them, the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

[0099] In some optional implementations, the signed distance field model and the color estimation model are constructed together, and the reconstruction module can be used to determine a three-dimensional model of the target object with color information based on the signed distance field model and the color estimation model;

[0100] Among them, the loss function used in the process of constructing the signed distance field model also includes a second loss function for determining the color loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the second loss function is adjusted from small to large.

[0101] In some optional implementations, the construction module may be used to update the three-dimensional model having color information in response to normalization of the color estimation model during the construction of the signed distance field model and the color estimation model;

[0102] According to the updated three-dimensional model with color information, the parameters of the signed distance field model are reversely adjusted.

[0103] In some optional implementations, the building module is further used to:

[0104] During the construction of the signed distance field model, the parameters of the signed distance field model are updated using a sliding window approach.

[0105] In some optional implementations, the three-dimensional reconstruction device further includes:

[0106] A pre-construction module is used to simplify the signed distance field model before constructing the signed distance field model to obtain a simplified model;

[0107] According to the sample image and the camera pose, determine the plane in the sample image;

[0108] Build a simplified model based on sample images, camera poses and planes;

[0109] Determine the bounding box of the target object based on the constructed streamlined model;

[0110] Correspondingly, the construction module can be used to construct a signed distance field model based on sample images, camera poses and bounding boxes.

[0111] In some optional implementations, the three-dimensional reconstruction device may further include:

[0112] The editing module can be used to edit at least one of the following parameters of the three-dimensional model after outputting the three-dimensional model of the target object: the shape of the three-dimensional model, and the material and texture of the mesh.

[0113] In some optional implementations, the three-dimensional reconstruction device may further include:

[0114] The sample image selection module can be used to obtain a video frame containing a target object and select a sample image from the video frame before constructing a signed distance field model.

[0115] In some optional implementations, the method is applied to the field of virtual broadcasting; wherein the target object includes at least one of the following: a broadcasting scene object, a broadcasting character object, and a broadcasting content object.

[0116] A three-dimensional reconstruction device provided in an embodiment of the present disclosure can execute a three-dimensional reconstruction method provided in any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.

[0117] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0118] Reference below Figure 7 , which shows an electronic device (eg, Figure 7 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0119] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0120] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0121] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in a three-dimensional reconstruction method of an embodiment of the present disclosure are executed.

[0122] The electronic device provided in the embodiment of the present disclosure and the three-dimensional reconstruction method provided in the above embodiment belong to the same disclosed concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0123] An embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, a three-dimensional reconstruction method provided by the above embodiment is implemented.

[0124] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0125] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0126] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0127] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0128] A signed distance field model is constructed based on a sample image containing a target object and a camera pose corresponding to the sample image; a three-dimensional model of the target object is determined according to the constructed signed distance field model; wherein the loss function used in the construction process of the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

[0129] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0130] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the units and modules do not, in some cases, limit the units and modules themselves.

[0132] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), etc.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] According to one or more embodiments of the present disclosure, a three-dimensional reconstruction method is provided, the method comprising:

[0135] Constructing a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image;

[0136] Determining a three-dimensional model of the target object according to the constructed signed distance field model;

[0137] Among them, the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

[0138] According to one or more embodiments of the present disclosure, a 3D reconstruction method is provided, further comprising:

[0139] In some optional implementations, the signed distance field model and the color estimation model are constructed together, and a three-dimensional model of the target object having color information is determined based on the signed distance field model and the color estimation model;

[0140] Among them, the loss function used in the process of constructing the signed distance field model also includes a second loss function for determining the color loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the second loss function is adjusted from small to large.

[0141] According to one or more embodiments of the present disclosure, a 3D reconstruction method is provided, further comprising:

[0142] In some optional implementations, the process of constructing the signed distance field model and the color estimation model includes:

[0143] In response to the normalization of the color estimation model, updating the three-dimensional model having color information;

[0144] According to the updated three-dimensional model with color information, the parameters of the signed distance field model are reversely adjusted.

[0145] According to one or more embodiments of the present disclosure, a 3D reconstruction method is provided, further comprising:

[0146] In some optional implementations, the signed distance field model construction process further includes:

[0147] The parameters of the signed distance field model are updated using a sliding window approach.

[0148] According to one or more embodiments of the present disclosure, a 3D reconstruction method is provided, further comprising:

[0149] In some optional implementations, before constructing the signed distance field model, the method further includes:

[0150] Simplifying the signed distance field model to obtain a simplified model;

[0151] Determine a plane in the sample image according to the sample image and the camera pose;

[0152] Constructing the simplified model based on the sample image, the camera pose and the plane;

[0153] Determine the bounding box of the target object according to the constructed simplified model;

[0154] Accordingly, the signed distance field model is constructed based on the sample image, the camera pose and the bounding box.

[0155] According to one or more embodiments of the present disclosure, a 3D reconstruction method is provided, further comprising:

[0156] In some optional implementations, before constructing the signed distance field model, the method further includes:

[0157] A video frame containing the target object is acquired, and the sample image is selected from the video frame.

[0158] According to one or more embodiments of the present disclosure, a 3D reconstruction method is provided, further comprising:

[0159] In some optional implementations, the method is applied to the field of virtual broadcasting; wherein the target object includes at least one of the following: a broadcasting scene object, a broadcasting character object, and a broadcasting content object.

[0160] According to one or more embodiments of the present disclosure, a three-dimensional reconstruction device is provided, the device comprising:

[0161] A construction module, used for constructing a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image;

[0162] A reconstruction module, used to determine the three-dimensional model of the target object according to the constructed signed distance field model;

[0163] Among them, the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

[0164] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0165] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0166] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Constructing a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image; Determining a three-dimensional model of the target object according to the constructed signed distance field model; Among them, the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

2. The method according to claim 1, characterized in that: The signed distance field model and the color estimation model are constructed together, and a three-dimensional model of the target object having color information is determined according to the signed distance field model and the color estimation model; Among them, the loss function used in the process of constructing the signed distance field model also includes a second loss function for determining the color loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the second loss function is adjusted from small to large.

3. The method according to claim 2, characterized in that The process of constructing the signed distance field model and the color estimation model includes: In response to the normalization processing of the color estimation model, updating the three-dimensional model having color information; According to the updated three-dimensional model with color information, the parameters of the signed distance field model are reversely adjusted.

4. The method according to claim 1, characterized in that: The process of constructing the signed distance field model further includes: The parameters of the signed distance field model are updated using a sliding window approach.

5. The method according to claim 1, characterized in that Before constructing the signed distance field model, the method further includes: Simplifying the signed distance field model to obtain a simplified model; Determine a plane in the sample image according to the sample image and the camera pose; Constructing the simplified model based on the sample image, the camera pose and the plane; Determine the bounding box of the target object according to the constructed simplified model; Accordingly, the signed distance field model is constructed based on the sample image, the camera pose and the bounding box.

6. The method according to claim 1, characterized in that Before constructing the signed distance field model, the method further includes: A video frame containing the target object is acquired, and the sample image is selected from the video frame.

7. The method according to claim 1, characterized in that Applied to the field of virtual broadcasting; wherein the target object includes at least one of the following: a broadcasting scene object, a broadcasting character object and a broadcasting content object.

8. A three-dimensional reconstruction device, characterized in that: include: A construction module, used for constructing a signed distance field model based on a sample image containing a target object and a camera pose corresponding to the sample image; A reconstruction module, used to determine the three-dimensional model of the target object according to the constructed signed distance field model; Among them, the loss function used in the process of constructing the signed distance field model includes a first loss function for determining the curvature loss between the three-dimensional model and the target object; and in the construction process from front to back, the weight of the first loss function is adjusted from large to small.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement a three-dimensional reconstruction method as described in any one of claims 1-7.

10. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform a three-dimensional reconstruction method as claimed in any one of claims 1 to 7 when executed by a computer processor.

Images