Method, electronic device and computer program product for object modeling
By using camera spatial information to generate probability distribution of three-dimensional models in the NeRF 360 framework, the problem of slow training and inference speed of NeRF technology is solved, and fast three-dimensional rendering and high-quality reconstruction on smart devices is realized, suitable for the meta-universe and Virtual reality applications.
Patent Information
- Application Number
- CN202310645471.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-01
- Publication Date
- 2024-12-03
AI Technical Summary
The existing NeRF-based three-dimensional model processing technology requires strict supervision of data during the training process, the inference speed is slow, and the three-dimensional semantic representation is not effectively learned, making it difficult to run efficiently in real-time applications, and the user's degree of freedom to query and edit is limited. The influence of camera position and observation direction was not considered.
By obtaining the spatial information of the camera at different locations, a spatial distribution is generated to represent the probability distribution of the space occupied by the object. Based on this, an image is generated and the object model is adjusted, and the inference time is accelerated by using the NeRF 360 framework, and three-dimensional view rendering is realized in smart devices. Decoupled coordinates and orientation mappings are provided through equivalent view synthesis, providing an effective training scheme to reduce the workload of photo collection and processing.
It realizes fast three-dimensional scene rendering on smart devices, improves inference speed, reduces training time and data processing, and is suitable for meta-universe and virtual reality applications, providing higher visualization quality and faster training convergence.
Smart Images

Figure CN119068048A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the field of computers, and more particularly, to a method, device, and computer program product for object modeling. Background Art
[0002] Deep learning techniques have been widely used in various fields, including three-dimensional (3D) model processing. The basic idea of using deep learning for 3D new view synthesis is to use deep convolutional neural networks to learn implicit representations. Summary of the invention
[0003] Embodiments of the present disclosure relate to methods, electronic devices, and computer program products for object modeling.
[0004] According to a first aspect of the present disclosure, a method for object modeling is provided. The method includes obtaining first position spatial information of a camera at a first position and second position spatial information of a camera at a second position, wherein the first position is different from the second position. A spatial distribution is generated based on the first position spatial information and the second position spatial information, wherein the spatial distribution represents the probability distribution of the space occupied by the object captured from the position of the camera. A third image is generated based on a first image captured at the first position, the spatial distribution, and the first position spatial information. A fourth image is generated based on a second image captured at the second position, the spatial distribution, and the second position spatial information, and a model of the object is adjusted based on the first image, the second image, the third image, and the fourth image.
[0005] According to a second aspect of the present disclosure, an electronic device for object modeling is provided, comprising at least one processor, and a memory, coupled to the at least one processor and having instructions stored thereon, the instructions causing the electronic device to perform an action when executed by the at least one processor, the action comprising obtaining first position spatial information of a camera at a first position and second position spatial information of a camera at a second position, wherein the first position is different from the second position. A spatial distribution is generated based on the first position spatial information and the second position spatial information, wherein the spatial distribution represents a probability distribution of the space occupied by the object captured from the position of the camera. A third image is generated based on a first image captured at the first position, the spatial distribution, and the first position spatial information. A fourth image is generated based on a second image captured at the second position, the spatial distribution, and the second position spatial information, and a model of the object is adjusted based on the first image, the second image, the third image, and the fourth image.
[0006] According to a third aspect of the present disclosure, there is provided a computer program product, which is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions, which when executed cause a machine to perform the steps of the method implemented in the first aspect of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like elements throughout the exemplary embodiments of the present disclosure.
[0008] Figure 1 FIG. 1 illustrates an architecture diagram of a three-dimensional model processing system according to an embodiment of the present disclosure;
[0009] Figure 2 A flowchart of a method for object modeling according to an embodiment of the present disclosure is illustrated;
[0010] Figure 3 A schematic diagram of the process of training a three-dimensional model processing system is illustrated;
[0011] Figure 4 The figure shows the experimental effect of using the 3D model processing system to perform 3D reconstruction on a real person;
[0012] Figure 5 A diagram illustrating the concept of equivalent view synthesis according to an embodiment of the present disclosure;
[0013] Figure 6 A schematic block diagram of an example device that may be used to implement embodiments of the present disclosure is illustrated. DETAILED DESCRIPTION
[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0015] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0016] As a new generation of Internet trend, the Metaverse has been and is increasingly attracting public attention. The Metaverse is a virtual world built using digital technology that maps or transcends the real world and can interact with the real world. In order to build a realistic Metaverse world, accurate three-dimensional (3D) modeling of scenes / objects is required. One problem that needs to be solved is whether users can use Metaverse applications anywhere without worrying about delays in computing and data access.
[0017] One approach is to use deep learning techniques based on Neural Radiant Field (NeRF). Given paired camera positions and scenes, it uses differentiable volume rendering techniques to sample implicit 3D space for realistic scene reconstruction. However, there are many problems to be solved in NeRF-based implementations, among which the training process of NeRF requires strict supervision using paired data and the inference speed is slow, which makes it inefficient in real-time applications. NeRF-based implementations do not learn 3D semantic representations for scene understanding. For example, some implementations only solve auxiliary shape and color codes for conditional NeRF rendering, and they are only applicable to small and simple objects such as chairs and cups. Some implementations are hybrid models based on Clip language model and NeRF, which can be used for text-driven scene generation, generating scenes that match text descriptions, but cannot identify semantic differences at the immediate level. In some implementations, a semantic segmentation network is introduced for joint geometry and semantic prediction, and the correlation between segmentation and scene prediction can also benefit from the smoothness and self-similarity of self-supervised learning. In some implementations, information is also extracted from a self-supervised feature extractor (e.g., CLIP-LSeg or DINO) to process the 3D neural field in parallel. The disadvantage of these implementations is that the user query editing has limited freedom. The above implementations also do not consider the effects of camera pose, nor do they decompose the object shape from the viewing direction, which makes it difficult for the network to learn.
[0018] At least to solve the above and other potential problems, an embodiment of the present disclosure provides a method for object modeling. The method includes obtaining first position spatial information of a camera at a first position and second position spatial information of a camera at a second position, wherein the first position is different from the second position. Generate a spatial distribution based on the first position spatial information and the second position spatial information, wherein the spatial distribution represents the probability distribution of the space occupied by the object captured from the position of the camera. Generate a third image based on the first image captured at the first position, the spatial distribution, and the first position spatial information. Generate a fourth image based on the second image captured at the second position, the spatial distribution, and the second position spatial information, and adjust the model of the object based on the first image, the second image, the third image, and the fourth image. By using this method, a framework based on NeRF 360 that can speed up the inference time is provided, and can be embedded in portable devices such as smart phones for dedicated three-dimensional view rendering, and coordinate and direction mapping are decoupled by synthesizing an additional view-related three-dimensional deformation module for equivalent views. An effective and reliable training scheme is also proposed by this method, which can reduce the workload of photo collection and processing required.
[0019] The basic principles and several exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Figure 1 FIG. 1 illustrates an architecture diagram of a three-dimensional model processing system 100 according to an embodiment of the present disclosure. It should be understood that Figure 1 The numbers and arrangements of components, elements, and systems shown are merely examples, and the architecture diagram may include different numbers and arrangements of components, elements, and systems.
[0020] According to an implementation of the present disclosure, a three-dimensional model processing system (also referred to as a NeRF 360 system) 100 can implement arbitrary 360-degree three-dimensional scene or object modeling, and can include three main components: a basic NeRF network 103, two-dimensional (2D) to three-dimensional (3D) mappers 104 and 105, and a converter 106, wherein the basic NeRF network 103 can be used for implicit view synthesis of the captured object data 101 and 102. The two-dimensional to three-dimensional mappers 104 and 105 can be used for the projection of features in the data 101 and 102, and the two-dimensional to three-dimensional mappers 104 and 105 can be the same components or different components, and the present disclosure does not impose any restrictions on this. The converter 106 can be used for explicit three-dimensional feature deformation (wrap). According to some implementations of the present disclosure, NeRF (neural radiance field) 103 is a realistic neural rendering machine that can learn three-dimensional representations from multi-view images, and then it can render new views based on the input camera pose and spatial position. The high-definition visualization quality of its reconstruction surpasses other methods, an advantage that makes it suitable for metaverse architecture and virtual reality applications.
[0021] First, the basic NeRF network 103 in the 3D model processing system 100 can receive data T(x, d)101 and T(x, d)102 including the camera's pose information (d), the camera's position information (x), and image features. * (x, d) 102 is used as input to learn the corresponding object view rendering, where the camera pose can represent the rotation and translation of the camera coordinate system relative to the reference coordinate system (world coordinate system) and can include information such as the angle of the corresponding camera when taking the photo. Additionally or alternatively, in some embodiments, the poses of cameras used to take images of the same object at different positions can be symmetrical to each other.
[0022] In response to receiving the object data 101 and 102, (multiple) 2D to 3D mappers may project the 2D features of the image therein into a 3D space, thereby obtaining (multiple) explicit 3D representations, such as 3D representations 107 and 108. The converter may then perform explicit view conversion on the explicit 3D representations 107 and 108, such as rotating the corresponding camera angles, etc., to obtain converted 3D representations 109 and 110, and finally complete the synthesis of equivalent views. In some embodiments, the converted 3D representation 109 may be the result of the converter 106 processing the 3D representation 107. The converted 3D representation 110 may be the result of the converter 106 processing the 3D representation 108.
[0023] The three-dimensional model processing system 100 can be installed with any computing device that has processing computing resources or storage resources. For example, the computing device can have common capabilities such as receiving and sending data requests, real-time data analysis, local data storage, and real-time network connection. Computing devices can generally include various types of devices. Examples of computing devices may include, but are not limited to: desktop computers, notebook computers, smart phones, wearable devices, security equipment, smart manufacturing equipment, smart home devices, Internet of Things devices, smart cars, drones, etc. In an embodiment according to the present disclosure, any number and type of devices may be included, and the present disclosure does not impose any restrictions on this.
[0024] Combined with the above Figure 1 A schematic diagram of the architecture of a three-dimensional model processing system in which the method and / or process according to an embodiment of the present disclosure can be implemented is described below. Figure 2A flowchart of a method 200 for object modeling of a three-dimensional model processing system according to an embodiment of the present disclosure is described. As described above for the three-dimensional model processing system, the method 200 for object modeling according to an embodiment of the present disclosure can be executed at an edge device with computing capabilities or at a cloud server, and the present disclosure does not limit this. In order to more quickly realize the three-dimensional modeling and equivalent view synthesis of an object, a method 200 for an object according to an embodiment of the present disclosure is proposed.
[0025] At box 201, first position spatial information of a camera at a first position and second position spatial information of a camera at a second position are obtained, wherein the first position is different from the second position. According to an embodiment of the present disclosure, the three-dimensional model processing system may first obtain the position spatial information of a camera that photographs an object, and the position spatial information may include the position information of the camera (i.e., the position of the camera on the xyz space coordinate axis) and the pose information of the camera (i.e., the angle information presented by the camera at the position). Additionally or alternatively, the second position may be another position different from the first position, for example, the second position may be another position symmetrical to the first position.
[0026] At block 202, a spatial distribution is generated based on the first position spatial information and the second position spatial information, wherein the spatial distribution represents the probability distribution of the space occupied by the object captured from the position of the camera. The spatial distribution can be understood as the probability distribution along the ray captured by the camera. Through the spatial distribution, the distribution of objects on the ray can be approximately obtained.
[0027] At box 203, a third image is generated based on the first image captured at the first position, the spatial distribution, and the spatial information of the first position. According to an embodiment of the present disclosure, the three-dimensional model processing system can generate a first model of the object based on the first image and the spatial distribution. If the probability of the (multiple) points / positions sampled based on the spatial distribution is greater than a predetermined threshold, it can be considered that the (multiple) points / positions may have an object or a part of an object. Subsequently, the color, texture, depth and other information of the object or the part of the object at the point / position can be calculated finely. And the probability obtained before can be used to further calculate the color weight of the object or the part of the object. As an example, when the three-dimensional model processing system determines that there is a probability value higher than a predetermined threshold of the spatial distribution at a position in the first image, the three-dimensional model processing system can obtain one or more of the color information, texture information, and depth information at the position. Additionally or alternatively, the predetermined threshold can be a threshold specified by the user based on actual application needs.
[0028] The 3D model processing system can then perform a mesh transformation on the first model based on the camera's position in the first position space information to generate a second model, such as converting the first model by a certain angle to generate the second model. The 3D model processing system can then perform a 3D to 2D transformation on the second model, such as converting the second model into a 2D third image by projection via the converter therein. Additionally or alternatively, in the process of meshing the first model to generate the second model, the converter can also use bicubic interpolation on features in the first model, so that features in the second model can be predicted.
[0029] At box 204, a fourth image is generated based on the second image captured at the second position, the spatial distribution, and the second position spatial information. The three-dimensional model processing system can generate a third model based on the second image and the spatial distribution in a similar manner. The three-dimensional model processing system can also perform a mesh transformation on the third model based on the pose of the camera in the second position spatial information to generate a fourth model. Finally, the three-dimensional model processing system can perform a two-dimensional transformation on the fourth model to generate a fourth image.
[0030] In the generated image, the image content or features of the third image may correspond to the content or features of the second image, and the content or features of the fourth image may correspond to the content or features of the first image. In other words, the third image is the content of the second image predicted and generated by the 3D model processing system based on the first image, and the fourth image is the content of the first image predicted and generated by the 3D model processing system based on the second image.
[0031] At box 205, the model is adjusted based on the first image, the second image, the third image, and the fourth image. The three-dimensional model processing system may first calculate the first difference between the third image feature and the second image feature. The three-dimensional model processing system may also calculate the second difference between the fourth image feature and the first image feature. If the sum of the first difference and the second difference is greater than a predetermined threshold, it means that the three-dimensional model processing system still needs to be trained, and the three-dimensional model processing system may continue to be iteratively trained and adjusted by continuing to adjust the model of the object generated by the three-dimensional model processing system. If the sum of the first difference and the second difference is less than a predetermined threshold, it means that the three-dimensional model processing system has achieved the expected training effect, and therefore the iterative training and adjustment of the three-dimensional model processing system may be stopped by stopping adjusting the model of the object.
[0032] Additionally or alternatively, in the process of adjusting the model of the object, the model of the object may be adjusted based on spatial density information, color information, texture information, depth information of one or more of the first image, the second image, the third image, and the fourth image. In some embodiments, the predetermined threshold for whether to adjust the object may be specified by the user. In some embodiments, in order to increase the versatility of the generated model, the camera shake effect during the shooting process may also be adjusted, thereby enhancing the versatility and stability of the final generated model.
[0033] Figure 3 A schematic diagram of a process 300 for training a 3D model processing system is illustrated. Conventional NeRF-based systems typically process 3D information in an MLP (Multilayer perceptron) layer. Although this process is simple and efficient, it ignores 3D prior knowledge to achieve better reconstruction. In contrast, in an implementation according to the present disclosure, the 3D model processing system can be supervised trained using symmetric camera poses to inject symmetry properties into the modeling process of the object. The process 300 for training a 3D model processing system based on an embodiment of the present disclosure will be described below.
[0034] According to an embodiment of the present disclosure, given enough training images taken from a target scene, a three-dimensional model processing system can be trained. An image and a camera position x=(x, y, z) and a camera pose d=(θ, φ) corresponding to the image can be input into the three-dimensional model processing system, where θ and φ represent the x rotation angle and y rotation angle of the camera, respectively. The three-dimensional model processing system can output the corresponding spatial density / spatial distribution σ and image features such as RGB color c=(r, g, b) based on the acquired camera position x=(x, y, z) and camera pose d=(θ, φ). According to an embodiment of the present disclosure, the camera position X=(x, y, z) and the camera pose d=(θ, φ) can be collectively referred to as position space information.
[0035] As an example, the (multiple) images taken by the camera and the positional spatial information 301 associated with these images can be fed into the MLP layer 302 in the 3D model processing system for processing, thereby obtaining the corresponding spatial density / spatial distribution σ303, camera pose information T(d)310, T(d *) 311 and corresponding image features 304 and 305. Spatial distribution 303 can be defined as learning the probability of a space occupied by a three-dimensional object in the entire space by taking the camera's posture as the input to be learned. In some embodiments, image features 304 and 305 are image features obtained by a three-dimensional model processing system photographing the same object in multiple different postures. Additionally or alternatively, the first angle and the second angle in the multiple postures can be 180 degrees apart.
[0036] Additionally or alternatively, in some embodiments, the three-dimensional model processing system can also predict the pixel color to be generated. For example, the predicted color c(r) of a pixel is calculated by casting a ray into the volume starting from the camera origin O, in the direction d, r=o+td (t is the ray depth), and accumulating the color based on density along the ray. The three-dimensional model processing system can estimate the accumulated color by taking N point samples along the ray to perform volume rendering, referring to the following formula:
[0037]
[0038]
[0039]
[0040] in and are the rough estimated color and the fine estimated color, σ i is the volume density and r is the camera ray. spans the depth along (o, d) starting from the near camera plane and ending at the far camera plane. The depth t is selected from a uniform distribution using stratified sampling. Both the density σ and the color C are modeled using an MLP. The final rendering is trained in a self-supervised manner using the observed true per-pixel color. Instead of performing a single rendering pass, points are first sampled in a stratified manner. The density at these points is then used for importance sampling in the final pass. Therefore, the final model is learned by supervising the rendered pixel colors of both passes using the true colors.
[0041] According to an embodiment of the present disclosure, when training a three-dimensional model processing system, the distance from the camera position to the center of the target object can be standardized. In this way, the distance between the camera and the object is a constant and can be ignored in the calculation. In addition, the camera pose can be converted from an observer-centered perspective to an object-centered perspective. For example, given a camera position T(x, d), its symmetric camera position can be T*(x, d) = T(x, 360-d). In addition, according to an embodiment of the present disclosure, a Siren module can be used to divide the signal domain into blocks for feature extraction, thereby locally simplifying the signal. Sinusoidal activation is used for a wider spectrum, and the process can be expressed as:
[0042] h i =α i ⊙sin(w i h i-1 +b i ) (2)
[0043] where w i and b i are the learnable weights and biases of the i-th layer. α is the modulation variable. h i is the hidden feature and h0 is the input coordinate.
[0044] In some embodiments, the spatial distribution 303 can also be used together with one or more image features in the image features 304 and / or 305 to construct a model of the object. For example, the three-dimensional model processing system can divide the entire captured image space into several 1x1x1 spatial blocks, and then the three-dimensional model processing system can determine whether the spatial block has a probability value higher than a predetermined threshold of the spatial distribution 303. If the spatial block has a probability value higher than the predetermined threshold of the spatial distribution 303, it means that the spatial block may have the image features of the captured object. In other words, the spatial block is preliminarily rendered. Additionally or alternatively, the predetermined threshold of the spatial distribution 303 can be arbitrarily specified by the user.
[0045] After determining that the spatial block has a probability value higher than a predetermined threshold of the spatial distribution 303, the 3D model processing system can further obtain the corresponding color information, texture information, depth information, etc. in the spatial block. In some embodiments, the image features 304 and 305, the camera pose information 310, 311, and the corresponding color information, texture information, depth information, etc. can be fed to the 2D to 3D mappers 308 and 309 via the MLP layers 306 and 307 in the 3D model processing system for projecting the feature data, thereby obtaining one or more models corresponding to the image.
[0046] The 2D to 3D mappers 308 and 309 can in turn transform the received 2D features (F∈R N×D ) is transformed into a 3D shape / model (G∈R m×m×D ). The 2D to 3D mappers 308 and 309 may further apply a 3D convolutional network to process the 3D model to generate one or more cube vectors G∈R m×n×m 312-1, 312-2, 312-3 and 313-1, 313-2, 313-3, etc., where N=m×m is the number of pixels and D is the dimension of the two-dimensional feature.
[0047] According to an embodiment of the present disclosure, converters 314 and 315 take a three-dimensional feature map including cube vectors 312-1, 312-2, etc. as an input of dimension m×m×m. Additionally or alternatively, converters 314 and 315 may also input the camera poses 310 and 311 as conditions into grid generators 316 and 317 in converters 314 and 315, and grid generators 316 and 317 may convert the camera pose (θ, φ) into a six-degree camera matrix S, for example, after a cross multiplication operation. Converters 314 and 315 then convert the three-dimensional feature map grid into a target three-dimensional feature map / model 318 and 319 corresponding to the target camera pose. For sub-pixels, it uses bicubic interpolation to predict the features in the model to be generated.
[0048] According to an embodiment of the present disclosure, the target 3D feature maps / models 318 and 319 corresponding to the target camera pose converted by the converters 314 and 315 can be projected into two-dimensional feature maps F'320 and 321. The MLP layer 322 in the final three-dimensional model processing system can obtain corresponding color information such as RGB values, texture information, depth information, etc. from the two-dimensional feature maps F'320 and 321 for image reconstruction.
[0049] During the training process, the 3D model processing system may generate at least four images 323 to 326, wherein image 323 and image 324 are images obtained by the 3D model processing system by taking pictures at different positions using a camera without being processed by the 2D to 3D mapper 308 and / or 309 and the converter 314 and / or 315. Image 325 and image 326 are images generated by the 3D model processing system based on image 323 and image 324, after being processed by the 2D to 3D mapper 308 and / or 309 and the converter 314 and / or 315. Image 323 may correspond to the content and / or features in image 325, and image 324 may correspond to the content and / or features in image 326.
[0050] The 3D model processing system can eventually be trained as a Siamese network. In other words, for the paired compensated camera parameter / position space information T(x, d) and T(x, d*), the 3D model processing system can predict the corresponding four new views V x,d 、V x,d* 、V t x,d V t x,d* , where V t . is the transformed version of V., and the loss with the ground truth image can be calculated as:
[0051]
[0052]
[0053]
[0054] The mask M is applied to the MSE (mean square error) loss to constrain the training optimization to the region of interest, and M can be arbitrarily specified by the user.
[0055] In some embodiments, the three-dimensional model processing system may compare the difference or similarity between the content and / or features in image 323 and image 325 to obtain a first difference. The three-dimensional model processing system may also compare the difference or similarity between the content and / or features in image 324 and image 326 to obtain a second difference. Finally, by calculating whether the sum of the first difference and the second difference meets the predetermined threshold specified by the user, it is determined whether the training goal required by the user has been achieved. For example, if the sum of the calculated first difference and the second difference is greater than the predetermined threshold, it means that the three-dimensional model processing system needs to continue to be trained. If the sum of the calculated first difference and the second difference is less than the predetermined threshold, it means that the training of the three-dimensional model processing system has achieved the desired goal, and the training of the three-dimensional model processing system can be stopped. In some embodiments, the image content and / or features may include spatial density information, color information, texture information, and depth information.
[0056] During training, paired images can be used for better optimization. In other words, when capturing images of the target object, the camera can be evenly placed around the object and ensure that there are symmetrical views for training. Alternatively, the position of the camera can be fixed and the object can be rotated 360 degrees to capture multiple symmetrical views. In order to improve the versatility of the 3D model processing system, ray direction jitter can also be applied during object capture. For example, sub-pixel jitter can be applied to the ray direction. As opposed to capturing at the center of the pixel, jitter ψ can be used as x i =o+t i (d+ψ). ψ can be uniformly sampled so that the generated rays remain confined to the boundaries of their assigned pixels.
[0057] Figure 4 The diagram shows an experimental effect of using a 3D model processing system to perform 3D reconstruction of a real person 400. During the training process, 30 photos taken at symmetrical angles were collected. For the purpose of illustration, Figure 4 As shown, the three-dimensional reconstruction views of the character are respectively performed at different angles and in symmetrical directions, wherein view 410 is a view for three-dimensionally reconstructing the character from 0 degrees, view 411 is a view for three-dimensionally reconstructing the character from 90 degrees, view 412 is a view for three-dimensionally reconstructing the character from 180 degrees, and view 413 is a view for three-dimensionally reconstructing the character from 270 degrees.
[0058] It can be seen from the quantitative results shown in Table 1 below that, compared with other technologies, the three-dimensional model reconstructed based on the method of the present disclosure can have better visualization quality and faster training convergence.
[0059]
[0060]
[0061] It can be observed from the table that compared with the known technology, the three-dimensional model processing system implemented according to the present disclosure has better visual quality in terms of PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measure), and the improvement is about 0.6dB and 0.01 respectively. At the same time, the implementation according to the present disclosure can significantly reduce the training time originally required by 33% to 88%. As for the reasoning time, FPS (frames per second) can be used to measure the calculation time. Initially, the method of the present disclosure cannot achieve real-time rendering, and the rendering is about 0.051 frames per second. The method of the present disclosure is further improved by using spherical harmonics (SH) to encode the color space for offline three-dimensional extraction. The three-dimensional model processing system (SH) can reach 154.12 in FPS, which is much higher than other technical solutions.
[0062] Therefore, the present disclosure proposes a simple and efficient method for 360-degree three-dimensional scene reconstruction or rendering. Without any additional data storage and computing costs, the implementation of the present disclosure can be used on edge computing devices, such as smart phones and laptops to achieve arbitrary 360-degree three-dimensional scene modeling. It is a core component of metaverse applications, such as customized virtual tours. At the same time, the implementation can be applied to data center displays, virtual item displays, and so on. Coordinate and direction mapping are decoupled by synthesizing additional view-related 3D deformation modules for equivalent views. The present disclosure uses equivalent view synthesis learning to perform NeRF-based three-dimensional scene manipulation, while using explicit three-dimensional deformation to supervise the generation of symmetric scenes. The present disclosure can also be used in many metaverse-based applications.
[0063] Figure 5 FIG500 illustrates the concept of equivalent view synthesis according to an embodiment of the present disclosure. Figure 5As shown on the left side of FIG, given a chair 510, the camera can be moved around the chair 510 to capture all views, such as view 511, view 512, view 513, and view 514 captured at different viewing angles for the chair 510. Equivalently, Figure 5 As shown on the right side of FIG. 5 , the position of the camera is fixed, and the chair 514 can be rotated 360 degrees to capture views 511', 512', 513', and 514' captured when the chair 510 is rotated to different angles. The following formula exemplarily shows the concept of equivalent view synthesis according to an embodiment of the present disclosure:
[0064] H(T z (Z))=T z (H(Z)), and H -1 (T x (X)) = T x (H -1 (x)) (4)
[0065] Where H is the 3D to 2D mapping, T z It is a 3D conversion. x is a 2D transformation, and H -1 It is a 2D to 3D mapping.
[0066] In other words, when learning a new view of a 3D object, rotating the object and then capturing a new view is equivalent to rotating the camera around the object and then capturing a new view. In some embodiments, in order to consider the impact of the background, a binary mask can be used to select a region of interest for 3D reconstruction.
[0067] Figure 6 A schematic block diagram of an example device 600 that may be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device in the device 600 can be implemented by using the device 600. As shown in the figure, the device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or computer program instructions loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the storage device 600 can also be included. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0068] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage page 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0069] The various processes and processing described above, such as method 200, may be performed by processing unit 601. For example, in some embodiments, method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by CPU 601, one or more actions of method 200 described above may be performed.
[0070] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.
[0071] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.
[0072] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0073] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0074] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0075] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0076] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0077] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0078] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for object modeling, the method comprising: Acquire first position spatial information of a camera at a first position and second position spatial information of the camera at a second position, wherein the first position is different from the second position; generating a spatial distribution based on the first position spatial information and the second position spatial information, wherein the spatial distribution represents a probability distribution of a space occupied by the object captured from the position of the camera; generating a third image based on the first image captured at the first position, the spatial distribution, and the first position spatial information; generating a fourth image based on the second image captured at the second position, the spatial distribution, and the second position spatial information; as well as A model of the object is adjusted based on the first image, the second image, the third image, and the fourth image.
2. The method according to claim 1, wherein generating a third image based on the first image captured at the first position, the spatial distribution, and the first position spatial information comprises: generating a first model based on the first image and the spatial distribution; Performing mesh transformation on the first model based on the position and posture of the camera in the first position space information to generate a second model; as well as Performing a two-dimensional transformation on the second model, thereby generating the third image.
3. The method of claim 2, wherein generating a first model based on the first image and the spatial distribution comprises: determining whether there is a probability value above a predetermined threshold of the spatial distribution at a location in the first image; as well as In response to the existence of the probability value higher than the predetermined threshold of the spatial distribution at the position in the first image, one or more of the color information, texture information, and depth information at the position is acquired.
4. The method according to claim 2, wherein mesh transforming the first model based on the position of the camera in the first position space information to generate the second model further comprises: Bicubic interpolation is used on features in the first model to predict features in the second model.
5. The method according to claim 1, wherein generating a fourth image based on the second image captured at the second position, the spatial distribution, and the second position spatial information comprises: generating a third model based on the second image and the spatial distribution; Performing mesh transformation on the third model based on the position and posture of the camera in the second position space information to generate a fourth model; as well as Performing a two-dimensional transformation on the fourth model, thereby generating the fourth image.
6. The method of claim 1, wherein the third image feature corresponds to the second image feature and the fourth image feature corresponds to the first image feature.
7. The method of claim 6, wherein adjusting the model based on the first image, the second image, the third image, and the fourth image comprises: calculating a first difference between the third image feature and the second image feature; calculating a second difference between the fourth image feature and the first image feature; In response to a sum of the first difference and the second difference being greater than a predetermined threshold, continuing to adjust the model of the object; and In response to a sum of the first difference and the second difference being less than the predetermined threshold, adjusting the model of the object is stopped.
8. The method of claim 1, wherein adjusting the model based on the first image, the second image, the third image, and the fourth image further comprises: The model of the object is adjusted based on spatial density information, color information, texture information, and depth information in the first image, the second image, the third image, and the fourth image.
9. The method of claim 1, wherein acquiring the first image based on the first position spatial information and acquiring the second image based on the second position spatial information comprises: Using a camera to photograph the same object at the first position and the second position respectively to acquire the first image and the second image; and The camera is shaken during use.
10. An electronic device, comprising: processor; as well as A memory is coupled to the processor and stores instructions that, when executed by the processor, cause the device to perform the following actions: Acquire first position spatial information of a camera at a first position and second position spatial information of the camera at a second position, wherein the first position is different from the second position; generating a spatial distribution based on the first position spatial information and the second position spatial information, wherein the spatial distribution represents a probability distribution of a space occupied by an object captured from a position of the camera; generating a third image based on the first image captured at the first position, the spatial distribution, and the first position spatial information; generating a fourth image based on the second image captured at the second position, the spatial distribution, and the second position spatial information; as well as A model of the object is adjusted based on the first image, the second image, the third image, and the fourth image.
11. The electronic device according to claim 10, wherein generating a third image based on the first image captured at the first position, the spatial distribution, and the first position spatial information comprises: generating a first model based on the first image and the spatial distribution; Performing mesh transformation on the first model based on the position and posture of the camera in the first position space information to generate a second model; as well as Performing a two-dimensional transformation on the second model, thereby generating the third image.
12. The electronic device of claim 11, wherein generating a first model based on the first image and the spatial distribution comprises: determining whether there is a probability value above a predetermined threshold of the spatial distribution at a location in the first image; as well as In response to the existence of the probability value higher than the predetermined threshold of the spatial distribution at the position in the first image, one or more of the color information, texture information, and depth information at the position is acquired.
13. The electronic device according to claim 11, wherein performing mesh transformation on the first model based on the position and posture of the camera in the first position space information to generate the second model further comprises: Bicubic interpolation is used on features in the first model to predict features in the second model.
14. The electronic device according to claim 10, wherein generating a fourth image based on the second image captured at the second position, the spatial distribution, and the second position spatial information comprises: generating a third model based on the second image and the spatial distribution; Performing mesh transformation on the third model based on the position and posture of the camera in the second position space information to generate a fourth model; as well as Performing a two-dimensional transformation on the fourth model, thereby generating the fourth image.
15. The electronic device of claim 10, wherein the third image feature corresponds to the second image feature, and the fourth image feature corresponds to the first image feature.
16. The electronic device of claim 15, wherein adjusting the model based on the first image, the second image, the third image, and the fourth image comprises: calculating a first difference between the third image feature and the second image feature; calculating a second difference between the fourth image feature and the first image feature; In response to a sum of the first difference and the second difference being greater than a predetermined threshold, continuing to adjust the model of the object; and In response to a sum of the first difference and the second difference being less than the predetermined threshold, adjusting the model of the object is stopped.
17. The electronic device according to claim 10, wherein adjusting the model based on the first image, the second image, the third image, and the fourth image further comprises: The model of the object is adjusted based on spatial density information, color information, texture information, and depth information in the first image, the second image, the third image, and the fourth image.
18. The electronic device according to claim 10, wherein acquiring the first image based on the first position spatial information and acquiring the second image based on the second position spatial information comprises: Using a camera to photograph the same object at the first position and the second position respectively to acquire the first image and the second image; and The camera is shaken during use.
19. A computer program product tangibly stored on a non-transitory computer readable storage medium and comprising computer executable instructions that, when executed, cause a computer to: Acquire first position spatial information of a camera at a first position and second position spatial information of the camera at a second position, wherein the first position is different from the second position; generating a spatial distribution based on the first position spatial information and the second position spatial information, wherein the spatial distribution represents a probability distribution of a space occupied by an object captured from a position of the camera; generating a third image based on the first image captured at the first position, the spatial distribution, and the first position spatial information; generating a fourth image based on the second image captured at the second position, the spatial distribution, and the second position spatial information; as well as A model of the object is adjusted based on the first image, the second image, the third image, and the fourth image.
20. The computer program product of claim 19, wherein generating a third image based on the first image captured at the first location, the spatial distribution, and the first location spatial information comprises: generating a first model based on the first image and the spatial distribution; Performing mesh transformation on the first model based on the position and posture of the camera in the first position space information to generate a second model; as well as Performing a two-dimensional transformation on the second model, thereby generating the third image.
Citation Information
Patent Citations
Three-dimensional scene rendering method and device and storage medium
CN114119849A
Method and device for generating computer-generated hologram field based on neural radiation field
CN114529679A
Image generation method and device, electronic equipment and storage medium
CN115100339A
Three-dimensional reconstruction model training method and three-dimensional scene rendering method and device
CN115578515A
Neural radiation field reconstruction optimization method and device based on point cloud
CN115690324A
