Monocular three-dimensional reconstruction training method based on view angle range enhancement, editable method and method for performing three-dimensional reasoning in combination with poses
Monocular 3D reconstruction training is carried out through pre-acquisition angle and pose distribution, combined with multi-view fusion technology, the problems of unknown poses and limited samples of monocular 3D reconstruction are solved, and the reconstruction accuracy and user experience are improved.
Patent Information
- Application Number
- CN202411079610.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-08-07
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-13
AI Technical Summary
Monocular visual three-dimensional reconstruction algorithm is not ideal when the training samples are limited, and the reconstruction results are inaccurate due to unknown poses. The existing technology mostly uses restricted dead poses or evenly distributed poses, which limits user shooting freedom and reconstruction effects. .
By pre-acquisitioning multiple shooting angles of objects, the probability distribution or specify the main viewing angle is calculated, the rendering samples are used to generate multiple renderings, and the three-dimensional model training is performed using a deep neural network with an encoder-decoder structure, and the pose information is used to reconstruct. Multi-layer perceptron or Gaussian sputtering network optimization further improves accuracy through multi-view results fusion.
It quickly converges to a state with good results in a specific pose under a limited capacity network, improves the accuracy of monocular three-dimensional reconstruction and user freedom, and enhances the accuracy and editability of reconstruction results.
Smart Images
Figure CN119131234A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a monocular three-dimensional reconstruction training method based on visual range enhancement, an editable method, and a method for three-dimensional reasoning combined with posture. Background Art
[0002] Monocular vision 3D reconstruction is to reconstruct 3D scenes or 3D objects through images captured by a single camera. The monocular reconstruction algorithm requires a large number of sample training and takes a long time and a lot of computing resources. When the training samples are limited, the reconstruction effect of the current monocular reconstruction algorithm is not ideal.
[0003] Compared with 2D images, 3D objects have the concept of posture in space. Posture is a problem that has no unique solution in monocular 3D reconstruction, because the monocular only has partial observation information, and the position of the camera is unknown, and the posture of the camera will affect the reconstruction result. Currently, most of the monocular 3D reconstruction networks with good results are limited to posture, or use uniformly distributed postures for training. However, being limited to a single posture restricts user usage. It is difficult for users to take pictures in a specified posture, or it is difficult to find pictures on the Internet that fully meet the posture requirements. The uniformly distributed posture makes it difficult to achieve ideal reconstruction results for each posture because of the limitations of the network's expression and storage capabilities. Therefore, there is an urgent need for a monocular 3D reconstruction method that can solve diverse postures.
[0004] On the other hand, in the inference process, due to the limitation of shooting posture, a single picture only partially observes the object, and the reconstructed model is usually not accurate enough. However, there are a large number of pictures of the same object from different observation angles on the Internet. How to use these pictures to make the reconstruction result more accurate is also a technical problem that needs to be solved urgently. Summary of the invention
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] The present invention provides a monocular 3D reconstruction training method based on visual range enhancement, comprising:
[0007] The first step is to take multiple photos of different objects of the specified category through preliminary experiments, read the shooting angles, and calculate the probability distribution Rk based on the shooting angles. The probability distribution Rk refers to the law of the probability value Ri of any posture in space; or directly specify several main perspectives Ki; the posture is the posture relative to the typical posture (canonical pose) of the object in the object coordinate system, rather than the camera posture. For example, the front of a typical position of a chair is the direction of the chair upright with the back facing the person and away from the observer, and the seat cushion is perpendicular to the ground and closer to the observer than the back of the chair.
[0008] The second step is to render and sample the 3D models in the data set based on the rendering angle of probability distribution or a specified number of main viewing angles Ki, and obtain rendering images of multiple rendering angles of each 3D model in the data set;
[0009] In the third step, the 3D model and the rendering image from one of the rendering angles are used as sample pairs to train the monocular 3D model reconstruction model.
[0010] Compared with 2D images, 3D objects have the concept of posture in space. Posture is a problem that has no unique solution in monocular 3D reconstruction, because the monocular only has partial observation information, and the position of the camera is unknown, and the posture of the camera will affect the reconstruction result. Currently, most of the monocular 3D reconstruction networks with good effects are limited to posture, or use uniformly distributed postures for training, but the uniformly distributed posture makes the reconstruction effect of each posture may not be so good (compared to being limited to a single posture, and the single posture is too restricted, and it is difficult for users to shoot completely according to a single posture). The inventor has found in practice that the posture of most users taking pictures has a certain regularity. Training according to the pre-collected posture distribution can help the limited capacity network converge to a state that works well for certain specific postures, meet the monocular 3D reconstruction needs of most users, and also help the network quickly converge to a good state.
[0011] Furthermore, the main viewing angle Ki is (roll angle, pitch angle), specifically, five main viewing angles: (0,0), (15,15), (15,0), (-15,0) and (-15,15).
[0012] Furthermore, the specific process of the third step is:
[0013] For each sample pair, each point in the space is projected onto the image plane using the camera’s intrinsic parameters and rendering angle during rendering sampling. Then, its projection position is determined on each feature layer of the encoder, and each feature is concatenated to obtain a feature vector p containing pose information.
[0014] Encode the rendered image through the encoder to obtain the image feature vector;
[0015] The feature vector p containing the pose information is concatenated with the image feature vector and input into the decoder of the neural network, and then output to a decoder used to reconstruct the three-dimensional model. Finally, the three-dimensional model is reconstructed by the decoder. The neural network is used to output the vector in the latent space as a three-dimensional model.
[0016] Furthermore, the step of obtaining the feature vector P may also be:
[0017] The feature volume W is extracted from one or more sampled renderings. For the query point x on the camera ray, it is projected to the image coordinate π(x) on the image plane by using the known camera intrinsic parameters, and then bilinear interpolation is performed between the pixel-level features. The feature vector p is extracted from the feature volume W through projection and interpolation.
[0018] For a query point x on a camera ray of other viewpoints, whose view direction is d, a feature vector p is extracted from a feature volume W by projection and interpolation, and the feature vector p is passed together with the spatial coordinates to a network field storing three-dimensional information, and the output volume density value and RGB color information are volume rendered and compared with the pixel value of the known viewpoint, and then the comparison value is used as a loss function to optimize the neural network; wherein the network field is a multi-layer perceptron or Gaussian Splatting; the query point x and the direction d are represented in the camera coordinate system of the input viewpoint.
[0019] Furthermore, the three-dimensional model reconstruction model is a deep neural network with an encoder-decoder structure, a neural network based on a diffusion model (Diffusion Model) or a generative adversarial network (GAN), and a neural network based on optimizing a neural radiation field (NeRF).
[0020] The present invention also provides an editable method for three-dimensional objects based on a monocular three-dimensional reconstruction model with enhanced viewing range. After completing the training of the monocular three-dimensional model reconstruction model through the above-mentioned training method, at least two pictures to be inferred from viewpoints are input, and different three-dimensional model reconstruction models are output. Then, the results output by different three-dimensional model reconstruction models are fused to obtain an optimized three-dimensional model reconstruction model.
[0021] Furthermore, the image to be inferred comes from the search of a single image based on user input. The image retrieval algorithm is used to search for images of the same object in the single image uploaded by the user, and the images of the same object taken from multiple different angles are obtained.
[0022] Preferably, before fusion, it is also necessary to check whether the dimensions of the reconstructed models of the three-dimensional models to be fused are consistent. If they are inconsistent, it is necessary to sample the same model dimensions through a sampling algorithm before fusion.
[0023] Furthermore, the steps of fusing the output results of different 3D reconstruction models are as follows:
[0024] (1) Extract the vertex positions of the 3D object from different 3D model reconstruction models to form a shape matrix S, where each row of the shape matrix represents the vertex position of a sample and each column represents the coordinates (x, y, z) of a vertex;
[0025] (2) Performing PCA (principal component analysis) on the shape matrix S to obtain N principal components and eigenvalues;
[0026] (3) In the PCA space, calculate the average vector of the vertex positions of all sample shapes. The average vector is the fused shape.
[0027] Furthermore, the specific processing process of the average value vector is:
[0028] Each 3D data is represented as a linear combination of N principal components, and the average of these linear coefficients is calculated. The average coefficient is multiplied by the principal component to obtain the fused shape.
[0029] Furthermore, the color processing process of the three-dimensional object is:
[0030] In the PCA space, the average value vector of the chromaticity values or RGB component values of all sample shape vertices is calculated. The chromaticity average value vector is the fused color.
[0031] Furthermore, the results of the reconstruction of different 3D models are fused to obtain the average shape and vertex change pattern, and the key coordinates for surface editing are obtained after clustering, and the key coordinates are used to perform displacement editing on the vertices. Furthermore, the steps of fusing the results of the reconstruction of different 3D models are as follows:
[0032] The vertex displacement-based 3D generation algorithm obtains multiple models or vertex numbers for normalization and / or registration to obtain multiple 3D models with the same number of vertices and poses, and then calculates the average value of each vertex position or vertex chromaticity value to obtain the vertex position or vertex color of the fused shape.
[0033] The present invention also provides a method for three-dimensional reasoning in combination with posture, and the specific reasoning steps are: the first step is to shoot the selected object or load a picture;
[0034] The second step is to display a pose coordinate indication adjustment box in the canonical space on the screen, and the user scales the adjustment box and adjusts it in various directions, where the center of the adjustment box is the center of the object, and the XYZ directions of the box are the pose of the object in the canonical space, or a pose estimation method is used to estimate the pose of the object;
[0035] In the third step, the camera pose calculated based on the pose obtained in the second step is input into the monocular 3D model reconstruction model trained by the above training method, and then the 3D model is edited using any of the above editable methods.
[0036] Furthermore, the adjustment box is a virtual box surrounding the object, where the virtual box is a default shape or a bounding box obtained based on depth estimation and / or saliency segmentation results.
[0037] The present invention has the following beneficial effects:
[0038] (1) The present invention performs training according to the pre-collected posture distribution, which helps the limited capacity network converge to a state with good effects for certain specific postures, meets the monocular 3D reconstruction needs of most users, and also helps the network converge quickly to a good state.
[0039] (2) The present invention improves the accuracy of 3D reconstruction by adding additional perspectives by fusing the results of multiple perspectives. Users can search for pictures of the same object taken from different angles on the Internet for input, or use an image retrieval algorithm to automatically retrieve multi-angle pictures of the same object based on a single picture input by the user, which helps to improve the accuracy of the reconstruction results;
[0040] (3) The present invention adds posture priors and also inputs postures during reasoning, which can improve the accuracy of three-dimensional reconstruction. Most of the existing postures are based on the camera's perspective, while the present invention uses the posture from the object's perspective (canonical space) and observes it together with the input image, introducing stronger prior knowledge. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is a flowchart of the three-dimensional reconstruction training in Example 1.
[0042] Figure 2 The corresponding three-dimensional reconstruction result image is obtained by inputting the main viewing angle A(0,0) in Example 1.
[0043] Figure 3 The corresponding three-dimensional reconstruction result image is obtained by inputting the main viewing angle B (15, 15) in Example 1.
[0044] Figure 4 This is a schematic diagram of a fused three-dimensional model obtained by fusing Internet images in Example 2 with images of the main viewing angle A (0,0) and the main viewing angle B (15,15) input by the user.
[0045] Figure 5 This is a flow chart of fusing the output results of different three-dimensional model reconstruction models in Example 2.
[0046] Figure 6 This is a flowchart for reasoning about the posture of a three-dimensional object in Example 3. DETAILED DESCRIPTION
[0047] The specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. It should be pointed out that the embodiments are only specific explanations of the invention and should not be regarded as limitations of the invention. The purpose of the embodiments is to enable those skilled in the art to better understand and reproduce the technical solutions of the present invention. The protection scope of the present invention shall still be based on the scope defined by the claims.
[0048] Example 1
[0049] like Figure 1 As shown, this embodiment provides a monocular 3D reconstruction training method based on visual range enhancement, including:
[0050] In the first step, through preliminary experiments, different objects of the specified category are photographed multiple times, the shooting angles are read, and the probability distribution Rk is calculated based on the shooting angles. The probability distribution Rk refers to the law of the probability value Ri of any posture in space; or directly specify several main perspectives Ki; the main perspective Ki is (roll angle, pitch angle), specifically (0,0), (15,15), (15,0), (-15,0) and (-15,15). User experiments show that these perspectives are common shooting perspectives; the posture is the posture of the object relative to the canonical pose of the object in the object coordinate system. For example, the front of a typical position of a chair is the direction of the chair upright with the back facing the person and away from the observer, and the seat cushion is perpendicular to the ground and closer to the observer than the back of the chair.
[0051] The second step is to render and sample the 3D models in the data set based on the rendering angle of probability distribution or a specified number of main viewing angles Ki, and obtain rendering images of multiple rendering angles of each 3D model in the data set;
[0052] In the third step, the 3D model and the rendering image of one of the rendering angles are used as sample pairs to train the monocular 3D model reconstruction model; specifically, for each sample pair, each point in the space is projected onto the image plane through the camera's intrinsic parameters and the rendering angle during rendering sampling, and then its projection position is determined on each feature layer of the encoder, and each feature is concatenated to obtain a feature vector p containing pose information; in addition, the feature vector P can also be obtained by extracting a feature volume W for one or more sampled rendering images, and for a query point x on the camera light, projecting it to the image coordinate π(x) on the image plane by using the known camera intrinsic parameters, and then performing bilinear interpolation between pixel-level features, and extracting the feature vector p from the feature volume W through projection and interpolation. For a query point x on a camera ray of other viewpoints, whose view direction is d, a feature vector p is extracted from a feature volume W by projection and interpolation, and the feature vector p is passed together with the spatial coordinates to a network field storing three-dimensional information, and the output volume density value and RGB color information are volume rendered and compared with the pixel value of the known viewpoint, and then the comparison value is used as a loss function to optimize the neural network; wherein the network field is a multi-layer perceptron or Gaussian Splatting; the query point x and the direction d are represented in the camera coordinate system of the input viewpoint.
[0053] Encode the rendered image through the encoder to obtain the image feature vector;
[0054] The feature vector p containing the pose information is concatenated with the image feature vector and input into the decoder of the neural network, and then output to a decoder used to reconstruct the 3D model. The decoder finally reconstructs the 3D model. The neural network is used to output the vector in the latent space as a 3D model. The 3D model reconstruction model is a deep neural network with an encoder-decoder structure, a neural network based on a diffusion model or a generative adversarial network (GAN), and a neural network based on optimizing the neural radiant field (NeRF). Figure 2 and 3 As shown, the corresponding three-dimensional reconstruction results are obtained by inputting the main viewing angle A (0, 0) and the main viewing angle B (15, 15).
[0055] Compared with 2D images, 3D objects have the concept of posture in space. Posture is a problem that has no unique solution in monocular 3D reconstruction, because the monocular only has partial observation information, and the position of the camera is unknown, and the posture of the camera will affect the reconstruction result. Currently, most of the monocular 3D reconstruction networks with good effects are limited to posture, or use uniformly distributed postures for training, but the uniformly distributed posture makes the reconstruction effect of each posture may not be so good (compared to being limited to a single posture, and the single posture is too restricted, and it is difficult for users to shoot completely according to a single posture). The inventor has found in practice that the posture of most users taking pictures has a certain regularity. Training according to the pre-collected posture distribution can help the limited capacity network converge to a state that works well for certain specific postures, meet the monocular 3D reconstruction needs of most users, and also help the network quickly converge to a good state.
[0056] Example 2
[0057] This embodiment provides an editable method for three-dimensional objects. After completing the training of the monocular three-dimensional model reconstruction model through the training method of embodiment 1, the pictures to be inferred from at least two viewpoints are input, different three-dimensional model reconstruction models are output, and then the results output by the different three-dimensional model reconstruction models are fused to obtain an optimized three-dimensional model reconstruction model; the accuracy of three-dimensional reconstruction is improved by adding additional viewpoints by fusion of the results of multiple viewpoints, wherein the picture to be inferred comes from the search of a single picture input by the user, and the picture retrieval algorithm is used to search for the same object picture in the single picture uploaded by the user, and the same object is obtained from multiple different angles. Figure 4 As shown, the fused three-dimensional model is obtained by combining Internet images with the images of the main viewing angle A (0,0) and the main viewing angle B (15,15) input by the user.
[0058] Before fusion, it is also necessary to check whether the dimensions of the reconstructed models of the 3D models to be fused are consistent. If they are inconsistent, it is necessary to sample them to the same model dimensions through a sampling algorithm before fusion; Figure 5 As shown, the specific steps for fusion are:
[0059] S1, extract the vertex positions of the three-dimensional object from different three-dimensional model reconstruction models to form a shape matrix S, where each row of the shape matrix represents the vertex position of a sample and each column represents the coordinates (x, y, z) of a vertex;
[0060] S2, perform PCA (principal component analysis) on the shape matrix S to obtain N principal components and eigenvalues;
[0061] S3, in the PCA space, calculate the average vector of the vertex positions of all sample shapes. The average vector is the fused shape. The specific processing process of the average vector is as follows:
[0062] Each 3D data is represented as a linear combination of N principal components, and the average of these linear coefficients is calculated. The average coefficient is multiplied by the principal component to obtain the fused shape.
[0063] In some preferred schemes, the color of the three-dimensional object can also be processed. The specific process is: in the PCA space, the average vector of the chromaticity values or RGB component values of all sample shape vertices is calculated. The chromaticity average vector is the fused color.
[0064] In addition, when the number of vertices of multiple three-dimensional models is the same, the step of fusing the results output by different three-dimensional model reconstruction models can also be: obtaining multiple models based on a three-dimensional generation algorithm based on vertex displacement, or normalizing and / or aligning the number of vertices to obtain multiple three-dimensional models with the same number of vertices and consistent postures, and then calculating the average value of each vertex position or vertex chromaticity value to obtain the vertex position or vertex color of the fused shape.
[0065] In some preferred schemes, by fusing the output results of different 3D reconstruction models, the average shape and vertex change pattern are obtained, and the key coordinates for surface editing are obtained after clustering. The key coordinate points are used to perform displacement editing on the vertices, so that the object can be better edited.
[0066] Example 3
[0067] like Figure 6 As shown, this embodiment provides a method for three-dimensional reasoning combined with posture, and the specific reasoning steps are:
[0068] In the first step, take a picture of the selected object or load a picture;
[0069] The second step is to display a pose coordinate indication adjustment box in the canonical space on the screen, and the user scales the adjustment box and adjusts it in various directions, where the center of the adjustment box is the center of the object, and the XYZ directions of the box are the pose of the object in the canonical space, or a pose estimation method is used to estimate the pose of the object;
[0070] In the third step, the camera pose calculated based on the pose obtained in the second step is input into the monocular 3D model reconstruction model trained by the above training method, and then the 3D model is edited using any of the above editable methods. The adjustment box is a virtual box surrounding the object, and the virtual box is a default shape or a bounding box obtained based on depth estimation and / or saliency segmentation results.
[0071] In addition, pose reasoning can also be done directly through a pre-trained pose estimation neural network.
[0072] The present invention increases the posture prior and also inputs the posture during reasoning, which can improve the accuracy of three-dimensional reconstruction. Most of the existing postures are based on the camera perspective, while the present invention adopts the posture based on the object perspective (canonical space), combined with the input picture for observation, thereby introducing stronger prior knowledge.
[0073] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0074] It should be noted that the technical features not described in detail in the present invention can be implemented by any existing technology.
Claims
1. A monocular 3D reconstruction training method based on visual range enhancement, characterized in that: include: The first step is to take multiple photos of different objects of the specified category through preliminary experiments, read the shooting angle, and calculate the probability distribution Rk based on the shooting angle; Or directly specify several main perspectives Ki; The second step is to render and sample the 3D models in the data set based on the rendering angle of probability distribution or a specified number of main viewing angles Ki, and obtain rendering images of multiple rendering angles of each 3D model in the data set; In the third step, the 3D model and the rendering image from one of the rendering angles are used as sample pairs to train the monocular 3D model reconstruction model.
2. The monocular 3D reconstruction training method based on visual range enhancement according to claim 1, characterized in that: The posture is a posture relative to a typical posture of the object in the object coordinate system.
3. The monocular 3D reconstruction training method based on visual range enhancement according to claim 1, characterized in that: The specific process of training in the third step is: For each sample pair, each point in the space is projected onto the image plane using the camera’s intrinsic parameters and rendering angle during rendering sampling, and then its projection position is determined on each feature layer of the encoder, and each feature is connected to obtain a feature vector p containing pose information; Encode the rendered image through the encoder to obtain the image feature vector; The feature vector p containing the pose information and the image feature vector are connected together and input into the decoder of the neural network, and then output to a decoder used to reconstruct the three-dimensional model. Finally, the three-dimensional model is reconstructed by the decoder, where the neural network is used to output the vector in the latent space as a three-dimensional model.
4. The monocular 3D reconstruction training method based on visual range enhancement according to claim 1, characterized in that: The steps of obtaining the feature vector P may also be: The feature volume W is extracted from one or more sampled renderings. For the query point x on the camera ray, it is projected to the image coordinate π(x) on the image plane by using the known camera intrinsic parameters, and then bilinear interpolation is performed between the pixel-level features. The feature vector p is extracted from the feature volume W through projection and interpolation.
5. The monocular 3D reconstruction training method based on visual range enhancement according to claim 1, characterized in that: The three-dimensional model reconstruction model is a deep neural network with an encoder-decoder structure, a neural network based on a diffusion model or adversarial generation, or a neural network based on optimization of a neural radiation field.
6. The monocular 3D reconstruction training method based on visual range enhancement according to claim 1, characterized in that: The main viewing angle Ki is (roll angle, pitch angle), specifically, there are five main viewing angles: (0,0), (15,15), (15,0), (-15,0) and (-15,15).
7. A method for editing a three-dimensional object based on a monocular three-dimensional reconstruction model with enhanced viewing range, characterized in that: After completing the training of the monocular 3D model reconstruction model by any training method in claims 1-6, input the images to be inferred from at least two viewpoints, output different 3D model reconstruction models, and then fuse the output results of different 3D model reconstruction models to obtain an optimized 3D model reconstruction model.
8. The editable method of a three-dimensional object based on a monocular three-dimensional reconstruction model with enhanced viewing range according to claim 7, characterized in that: The image to be inferred comes from the search of a single image based on user input. The image retrieval algorithm is used to search for images of the same object in the single image uploaded by the user, and the images of the same object taken from multiple different angles are obtained.
9. The editable method of a three-dimensional object based on a monocular three-dimensional reconstruction model with enhanced viewing range according to claim 7, characterized in that: The steps to fuse the outputs of different 3D reconstruction models are as follows: (1) Extract the vertex positions of the 3D object from different 3D model reconstruction models to form a shape matrix S, where each row of the shape matrix represents the vertex position of a sample and each column represents the coordinates (x, y, z) of a vertex; (2) Perform PCA processing on the shape matrix S to obtain N principal components and eigenvalues; (3) In the PCA space, calculate the average vector of the vertex positions of all sample shapes. The average vector is the fused shape.
10. The editable method of a three-dimensional object based on a monocular three-dimensional reconstruction model with enhanced viewing range according to claim 9, characterized in that: The specific processing process of the mean value vector is: Each 3D data is represented as a linear combination of N principal components, and the average of these linear coefficients is calculated. The average coefficient is multiplied by the principal component to obtain the fused shape.
11. The editable method of a three-dimensional object based on a monocular three-dimensional reconstruction model with enhanced viewing range according to claim 9, characterized in that: The color processing process for three-dimensional objects is: In the PCA space, the average value vector of the chromaticity values or RGB component values of all sample shape vertices is calculated. The average value vector is the fused color.
12. The editable method of a three-dimensional object based on a monocular three-dimensional reconstruction model with enhanced viewing range according to claim 7, characterized in that: By fusing the output results of different 3D reconstruction models, the average shape and vertex change pattern are obtained. After clustering, the key coordinates for surface editing are obtained, and the key coordinate points are used to perform displacement editing on the vertices.
13. The editable method of a three-dimensional object based on a monocular three-dimensional reconstruction model with enhanced viewing range according to claim 7, characterized in that: The steps to fuse the outputs of different 3D reconstruction models are as follows: The vertex displacement-based 3D generation algorithm obtains multiple models or vertex numbers for normalization and / or registration to obtain multiple 3D models with the same number of vertices and poses, and then calculates the average value of each vertex position or vertex chromaticity value to obtain the vertex position or vertex color of the fused shape.
14. A method for three-dimensional reasoning combined with posture, characterized in that: The specific reasoning steps are: In the first step, take a picture of the selected object or load a picture; The second step is to display the pose coordinate indication adjustment box in the standard space on the screen, and the user scales the adjustment box and adjusts it in various directions, where the center of the adjustment box is the center of the object, and the XYZ direction of the adjustment box is the pose of the object in the standard space, or use a pose estimation method to estimate the pose of the object; In the third step, the camera posture calculated based on the posture obtained in the second step is input into the monocular three-dimensional model reconstruction model trained by any training method described in any one of claims 1-6, and then the three-dimensional model is edited using any editable method described in claims 7-13.
15. A method for performing three-dimensional reasoning in combination with position and posture according to claim 14, characterized in that: The adjustment box is a virtual box around the object. The virtual box is a default shape or a bounding box obtained based on depth estimation and / or saliency segmentation results.