Method for obtaining three-dimensional model by means of photographing, method for obtaining picture by means of three-dimensional model, part machining method, and 3D printer

Abstract

Provided in the present invention are an interaction and generation method for obtaining a three-dimensional model by means of photographing, a method for obtaining a picture by means of a three-dimensional model, and a part machining and manufacturing method. The interaction and generation method comprises: a user using a terminal to capture scene information, wherein the scene information can be an outdoor or indoor scene, the terminal can be a smart phone, an AR / VR device or other smart terminals having a photographic function, and the captured scene information is RGB and / or RGBD (including depth) information; and sending an RGB image and / or RGBD image I to a monocular three-dimensional reconstruction algorithm, so as to obtain a corresponding three-dimensional model, which can be displayed on an interaction interface and superimposed with a real scene, or can be displayed on a single page.

Description

Method for obtaining three-dimensional model by shooting, method for obtaining picture from three-dimensional model, parts processing method and 3D printer Technical Field

[0001] The present invention relates to the technical field of computer software. Background Art

[0002] The rapid development of portable displays and AR / VR has created a huge demand for 3D content. Computer-aided design (CAD) methods require users to master complex CAD software commands and be able to parse shapes into sequential commands, which limits its application among professional users. These limitations have driven the demand for alternative methods that can open the door to 3D modeling for the general public. Three-dimensional modeling through photos is a method that has been developed over a long period of time and can lower the threshold of 3D modeling for users. Traditional methods based on computational photography (photogrammetry) and advanced neural network-based methods have been able to restore the captured scenes very well. However, for the specific task of three-dimensional reconstruction of objects, existing three-dimensional reconstruction methods often find it difficult to obtain accurate foreground and background segmentation, that is, to reconstruct the objects of interest to the user well so that they can be input into subsequent applications such as 3D printers and CNC engraving machines. The present invention proposes a new interactive method for obtaining three-dimensional models by shooting to solve the above problems. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an interactive and generating method for obtaining a three-dimensional model by shooting, a method for obtaining a picture from a three-dimensional model, and a processing and manufacturing method for parts in response to the problems existing in the background technology.

[0004] Specifically, first, the present invention provides a method for interacting and generating a specified model in a scene obtained by shooting, the method comprising the following steps:

[0005] In the first step, the user uses a terminal to capture scene information. The scene information can be outdoor or indoor scenes. The terminal can be a smartphone, AR / VR device, or other smart terminal with a camera function. The captured scene information is RGB and / or RGBD (including depth) information.

[0006] The second step is to display the scene information on the terminal and prompt the user to select the object of interest. The selection method can be to mark a bounding box of the object of interest, or,

[0007] Display the scene information on the terminal and prompt the user to select an object of interest by specifying one or more points on the object area of ​​interest;

[0008] In the third step, the scene graph and the box or point hints obtained in the second step are fed into a segmentation neural network to obtain a mask belonging to the object. The mask is then used to segment the RGB image and / or RGBD image containing the object. Alternatively, the object in the marked box is directly captured as the RGB image and / or RGBD image containing the object.

[0009] The fourth step is an optional step, in which the mask and / or the segmented object RGB image and / or RGBD image are displayed to the user. If the user identifies an inappropriate segmentation, the third step is repeated or additional point prompts are added;

[0010] In the fifth step, the RGB image and / or RGBD image I of only a single object is fed into the monocular 3D reconstruction algorithm to obtain the corresponding 3D model, which can be superimposed on the real scene display on the interactive interface or displayed on a single page.

[0011] As an advantage, the method may further include the following subsequent steps:

[0012] The sixth step is an optional step in which the user views the model obtained in the fifth step. The user moves the camera to a new perspective, and the model is rotated to the new perspective. The model can then be displayed to the user from the new perspective.

[0013] The seventh step is an optional step, in which the three-dimensional model obtained in the fifth step can be optimized based on the RGB image at the new perspective; the optimization includes optimizing the structure of the three-dimensional model and / or optimizing the texture of the three-dimensional model and / or optimizing the shape of the model;

[0014] Steps 6 and 7 can be repeated to obtain a sufficiently detailed model;

[0015] The above steps can be repeated for multiple objects in the scene;

[0016] Preferably, the angle difference of the view rotation can be obtained based on the posture information read by the IMU of the mobile phone, or can be obtained using an estimation based on optical flow or a Structure-from-Motion algorithm.

[0017] As an advantage, the method may further include the following subsequent steps: adding a step after the fifth step, the sixth step, or the seventh step:

[0018] Step 8: Convert the reconstructed 3D model into a compact surface-line combination, which the user can edit.

[0019] Specifically, the compact surface-line combination can be a B-Rep representation, a CSG representation, or a CAD construct sequence. Specifically, the conversion of the B-Rep representation can be performed using:

[0020] The method converts the three-dimensional model into a B-Rep representation, detects geometric primitives of different orders of the model, namely, vertices, edges and surface patches, and comprehensively models the correspondence between these primitives as a chain complex.

[0021] As an advantage, the following subsequent steps may also be included: the eighth step specifically includes:

[0022] 8.1 is used to generate geometric primitives and their relationships, and estimate their probabilities. The relationship extraction can use a convolutional neural network, such as a sparse convolution-based encoder, and then use a Transformer decoder to output corner points, curves, and patches and their mutual topological relationships;

[0023] As a preferred technical solution, mutual communication between element groups is further introduced in the transformer decoder, which effectively transforms other element groups into the context for generating the current group;

[0024] 8.2 Based on the probabilistic structure predicted in 8.1, a deterministic B-Rep chain complex is recovered by solving a global optimization problem to maximize the likelihood under the constraints of structural validity and perform geometric refinement. Given the solved structure, the geometry of the elements is optimized by fitting their corresponding type geometric representations to the input points and constraining their shapes so that the topology is strictly satisfied. The closed curves (patches) solved (predicted) are constructed accordingly.

[0025] 8.2.1 Use spline patches for fitting, fitting each patch as a spline (except planes and spheres) to the input points, curves, and corners;

[0026] 8.2.2 Fit each curve to the patch and corner points;

[0027] 8.2.3 Fit each corner point to a curve and patch;

[0028] 8.2.4 Convert spline patches into type primitives;

[0029] For each type of primitive patch, an iterative fitting process is again used to refine their geometry;

[0030] 8.2.5 Fit each type of patch to the input points, curves, and corners;

[0031] 8.2.6 Fit each curve to the patch and corner points;

[0032] 8.2.7 Fit each corner point to a curve and patch.

[0033] As an advantage, the following subsequent steps may also be included: the eighth step specifically includes:

[0034] The method of converting the 3D model into CSG representation is adopted:

[0035] 8.1 Receive a 3D shape as input and encode it using a convolutional neural network or a multi-layer perceptron (MLP)

[0036] 8.2 Use MLP to reconstruct the shape by predicting a set of primitives from the encoded feature vector. These primitives are combined into intermediate convex and concave shapes through the CSG tree. The primitives are represented by the tetrahedron equation.

[0037] ,

[0038] This equation determines whether a query point qj = (x, y, z) is inside or outside the primitive; the tetrahedron equation has the ability to generate several useful primitives, including a sphere, a plane, or a cylinder;

[0039] 8.3 CSG Tree Construction: The predicted primitives are then assembled into a CSG tree using a fixed-order combination strategy; the CSG tree represents a 3D shape as a combination of the union and intersection of the predicted primitives;

[0040] 8.4 Shape Reconstruction: Evaluate the CSG tree to reconstruct the 3D shape; this involves evaluating the tetrahedron equations of the primitives and combining them according to the CSG tree structure to obtain the final reconstructed shape; the CSG tree represents the 3D shape as a combination of the union and intersection of the predicted primitives; the assembly process starts with the first primitive and proceeds recursively until all primitives are combined into the final CSG tree;

[0041] 8.5 Primitive Dropout: To further improve the compactness of the learned primitives, apply primitive dropout technology based on importance measurement to remove redundant primitives; this step is optional.

[0042] As an advantage, the following subsequent steps may also be included: the eighth step specifically includes:

[0043] A method of converting a three-dimensional model into CAD instructions is adopted, wherein the CAD instructions may be a sketch and extrude representation method;

[0044] 8.1 Encode the 3D model into a feature vector. The encoding can be a convolutional neural network or a multi-layer perceptron (MLP).

[0045] 8.2 Use a Transformer decoder to output the sketch and extrude instruction sets one by one.

[0046] 8.3 Use a trained Latent-Diffusion model or autoencoder or VQ-VAE or Transformer, with the input being a 3D model and the output being the 3D model, to constrain the transformer decoder generation in 8.2; this step is optional.

[0047] As an advantage, the method may further include the following subsequent steps:

[0048] Add one more step after step 5 or 6 or 7 or 8:

[0049] The ninth step is to disassemble the generated 3D model. Users can edit each component after disassembly. The editing includes replacing components, changing dimensions, and displacing points, lines, and surfaces inside components.

[0050] Specifically, the disassembly step may be:

[0051] 9.1 Use a component instance segmentation network to segment the 3D model into component instances, assigning each facet to the instance it belongs to;

[0052] or,

[0053] 9.1 CNN-based segmentation network divides the shape into a set of coarse parts. The coarse parts are used as input to the recurrent network;

[0054] 9.2 Recursive Part Decomposition: The recursive network will take the part from 9.1 as input and recursively decompose it into smaller sub-parts. This involves applying a set of RNN-based decomposition modules to the input part in a bottom-up manner. Each decomposition module takes as input a point cloud representing the part and outputs a set of sub-parts. The sub-parts are then recursively decomposed until the desired level of granularity is reached.

[0055] 9.3 Hierarchical Output: Output a hierarchical segmentation of the shape, where each component is assigned a label representing its semantic category and a set of subcomponents; the hierarchical segmentation is represented as a tree structure, where the root node represents the entire shape and each child node represents a component of the shape;

[0056] Training is performed using the steps 9.1-9.3 of the supervised learning method, where the ground-truth segmentation is used as the target output; the loss function used during training is a combination of classification loss and geometric consistency loss; the classification loss penalizes errors in the predicted part labels, while the geometric consistency loss penalizes errors in the predicted part shapes.

[0057] The present invention also provides another method for interacting and generating a specified model in a scene obtained by photographing, the method comprising the following steps:

[0058] In the first step, the user uses a terminal to capture scene information. The scene information can be outdoor or indoor scenes. The terminal can be a smartphone, AR / VR device, or other smart terminal with a camera function.

[0059] The second step is to display the scene information on the terminal and prompt the user to select the object of interest. The selection method can be to mark a bounding box of the object of interest, or,

[0060] Display the scene information on the terminal and prompt the user to select an object of interest by specifying one or more points on the object area of ​​interest;

[0061] In the third step, the scene graph and the box or point hints obtained in the second step are fed into a segmentation neural network to obtain a mask belonging to the object. The mask is then used to segment the RGB image containing the object. Alternatively, the area within the marked box is directly captured as the RGB image containing the object.

[0062] The fourth step is an optional step, in which the mask and / or the segmented object RGB image are displayed to the user. If the user identifies an inappropriate segmentation, the third step is repeated or additional point prompts are added;

[0063] Step 5: Use the segmented RGB image I to estimate the object's pose in space, obtaining the object's pose relative to the camera (6D Pose). This pose is then compared with a pre-set pose range. The pre-set pose range can be a fixed interval R1-R2 or a probability distribution in the entire space, i.e., each pose has a "fit probability";

[0064] Step 6: If the current shooting posture in step 5 is inappropriate (e.g., the posture is outside the posture range, or the "suitability probability" of the posture range is lower than a threshold), prompt the user and guide the user on the interactive interface to rotate the camera so that the object is presented in a suitable posture; the inappropriate judgment is that the posture of the object does not fall within the set posture range R1-R2, or the probability value of belonging to the posture is lower than a certain threshold; preferably, the threshold is 0.86;

[0065] In the seventh step, the RGB image of a single object in a suitable position is fed into a monocular 3D reconstruction algorithm to obtain a corresponding 3D model, which can be overlaid and displayed on an interactive interface. As a preferred technical solution, the guidance on the interactive interface can be to indicate the direction in which the camera needs to be rotated (such as an arrow overlaid with the real scene).

[0066] As an advantage, the method may further include the following subsequent steps:

[0067] In the eighth step, the model is rotated to the view corresponding to the first step. The model is also rotated to the viewing angle when the model was taken in the first step, and the model is optimized based on the RGB image in the third step.

[0068] The ninth step is optional. The obtained model is uniformly sampled from N images within a range of viewing angles. These N images are fed into a discriminator. Based on the discriminator's output, the discriminator selects K viewing angle combinations that require additional user photography to achieve better modeling results. In other words, the user needs to shoot in these K viewing angle combinations to achieve better modeling results.

[0069] The discriminator can be a pre-trained CLIP encoder, or a pre-trained Diffusion-Inversion method that calculates the loss of natural images generated by diffusion, or an encoder that calculates VGG Loss;

[0070] Step 10 is an optional step in which the user views the model. When the user moves the camera to a new viewing angle, the model rotates to the new viewing angle and is displayed to the user from the new viewing angle. The new viewing angle can be a combination of the viewing angles in step 9. In this case, the interface prompts the user to shoot at an appropriate position. The new viewing angle can also be any viewing angle that the user can freely rotate.

[0071] The eleventh step is an optional step, in which the model can be optimized based on the RGB image at the new perspective; the optimization includes optimizing the structure of the three-dimensional model and / or optimizing the texture of the three-dimensional model and / or optimizing the shape of the model;

[0072] Steps 10 and 11 can be repeated to obtain a sufficiently detailed model and obtain more clues about texture information.

[0073] The present invention also provides another method for interacting and generating a specified model in a scene obtained by photographing, the method comprising the following steps:

[0074] Step 1: The user uses a terminal to capture scene information, which can be an outdoor or indoor scene. The terminal can be a smartphone, AR / VR device, or other smart terminal with a camera function. When shooting, the user is prompted to only capture a single object. As a preferred technical solution, a fixed-size frame is displayed on the shooting interface and superimposed on the real-time shooting interface, prompting the user to place the object to be photographed in the area within the frame; alternatively, the framing space is a fixed-size square frame, so an additional fixed-size frame is not displayed separately.

[0075] Optionally, the user can be guided to take a photo with a solid color background. A color-based segmentation model can then be used to obtain a mask of the area occupied by the object, and the mask can be used to obtain a picture of only one object as input I. Alternatively, a saliency segmentation model or other segmentation model can be used to obtain a mask of the area occupied by the object, and the mask can be used to obtain a picture of only one object as input I. Alternatively, the picture taken in this step can be directly used as input I.

[0076] The second step is to use the segmented RGB image I to estimate the pose of the object in space, obtaining the pose of the object relative to the camera (6D Pose). This pose is then compared with a pre-set pose range. The pre-set pose range can be a fixed interval R1-R2 or a probability distribution in the entire space, that is, each pose has a "fit probability";

[0077] Step 3: If the current shooting posture in step 5 is inappropriate (e.g., the posture is outside the posture range, or the "suitability probability" of the posture range is lower than a threshold), prompt the user and guide the user on the interactive interface to rotate the camera so that the object is presented in a suitable posture; the judgment of inappropriateness is that the posture of the object does not fall within the set posture range R1-R2, or the probability value of belonging to the posture is lower than a certain threshold; preferably, the threshold is 0.86;

[0078] In the fourth step, the RGB image of a single object in a suitable position is fed into a monocular 3D reconstruction algorithm to obtain a corresponding 3D model, which can be overlaid and displayed on an interactive interface. As a preferred technical solution, the guidance on the interactive interface can be to indicate the direction in which the camera needs to be rotated (such as an arrow overlaid with the real scene).

[0079] The present invention also provides another method for interacting and generating a specified model in a scene obtained by photographing, the method comprising the following steps:

[0080] Step 1: The user uses a terminal to capture scene information, which can be an outdoor or indoor scene. The terminal can be a smartphone, AR / VR device, or other smart terminal with a camera function. When shooting, the user is prompted to only capture a single object. As a preferred technical solution, a real-time shooting interface is displayed on the shooting interface, prompting the user to place the object in the area within the frame. Alternatively, the framing space is a fixed-size square frame, so no additional fixed-size frame is displayed separately.

[0081] Optionally, the user can be guided to take a photo with a solid color background. A color-based segmentation model can then be used to obtain a mask of the area occupied by the object, and the mask can be used to obtain a picture of only one object as input I. Alternatively, a saliency segmentation model or other segmentation semantic model can be used to obtain a mask of the area occupied by the object, and the mask can be used to obtain a picture of only one object as input I. Alternatively, the picture taken in this step can be directly used as input I.

[0082] The second step is to use the segmented RGB image I to estimate the pose of the object in space, obtaining the pose of the object relative to the camera (6D Pose). This pose is then compared with a pre-set pose range. The pre-set pose range can be a fixed interval R1-R2 or a probability distribution in the entire space, that is, each pose has a "fit probability";

[0083] Step 3: If the current shooting posture in step 5 is inappropriate (e.g., the posture is outside the posture range, or the "suitability probability" of the posture range is lower than a threshold), prompt the user and guide the user on the interactive interface to rotate the camera so that the object is presented in a suitable posture; the judgment of inappropriateness is that the posture of the object does not fall within the set posture range R1-R2, or the probability value of belonging to the posture is lower than a certain threshold; preferably, the threshold is 0.86;

[0084] In the fourth step, the RGB image of a single object in a suitable position is fed into a monocular 3D reconstruction algorithm to obtain a corresponding 3D model, which can be overlaid and displayed on an interactive interface. As a preferred technical solution, the guidance on the interactive interface can be to indicate the direction in which the camera needs to be rotated (such as an arrow overlaid with the real scene).

[0085] Preferably, add one step before the first step, or before the fifth step, or before the seventh step:

[0086] The user inputs or selects a hint about the photographed object, which can be a natural text description, such as "a table with long legs", or a category selection box or category text description, or a voice description; the category information will be input into the monocular 3D reconstruction algorithm and / or the pose estimation (6D Pose estimation) algorithm to enable the algorithm to perform better; optionally, separate monocular 3D reconstruction algorithms are trained for different categories, that is, the object input hint is used to select a suitable 3D reconstruction network model or model weights.

[0087] Preferably, add one step before the first step, or before the fifth step, or before the seventh step:

[0088] The user inputs a hint about the texture of the photographed object, which can be a natural text description, such as "a table with a retro-style texture", or a voice description, or a style reference picture; the texture description is input into a monocular 3D reconstruction algorithm or a separate texture generation algorithm to ensure that the generated 3D model has complete texture.

[0089] On this basis, the present invention provides a three-dimensional reconstruction method for multiple objects in a scene, which adopts the above-mentioned interaction and generation method to perform on any multiple objects in the shooting scene to obtain multiple model combinations in the scene.

[0090] On this basis, the present invention provides a method for rapid processing and manufacturing of parts, which includes the following steps:

[0091] The first step is to obtain a three-dimensional digital file of one or more models or multiple parts using the above-mentioned three-dimensional reconstruction method;

[0092] The second step is to input the 3D model into a parser and convert it into machine-readable instructions; for example, the parser is a slicing software that converts the 3D model into a machine-readable Gcode file;

[0093] In the third step, a processing device performs processing according to machine-readable instructions; the processing device can be a 3D printer, a laser engraving machine, a milling machine, or a CNC engraving machine.

[0094] On this basis, the present invention provides a 3D printer,

[0095] It includes at least one processing unit, one acquisition unit, and one execution unit;

[0096] The acquisition unit includes an image acquisition device and a matching lighting component to collect RGB or RGBD information of the object.

[0097] The processing unit runs the above-mentioned interaction and generation method or three-dimensional reconstruction method;

[0098] The execution unit includes a print head, a motion component, and a printing base. The motion component enables the print head to move to any position within the XYZ range of the printing base and the machine frame.

[0099] As a preferred technical solution, scale lines are printed on the printing base plate, and the scale lines represent scale information such as the angle, length, and width of the rotating object. DETAILED DESCRIPTION

[0100] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0101] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention. Example 1

[0102] A method for interacting and generating a specified model in a scene obtained by shooting, comprising the following steps:

[0103] In the first step, the user uses a terminal to capture scene information. The scene information can be outdoor or indoor scenes. The terminal can be a smartphone, AR / VR device or other smart terminal with a camera function. The scene information captured is RGB and / or RGBD (including depth) information.

[0104] The second step is to display the scene information on the terminal and prompt the user to select the object of interest. The selection method can be to mark a bounding box of the object of interest, or,

[0105] Display the scene information on the terminal and prompt the user to select the object of interest by specifying one or more points on the object area of ​​interest.

[0106] In the third step, the scene graph and the boxes or points obtained in the second step are fed into a segmentation neural network to obtain a mask belonging to the object. The mask is then used to segment the RGB image and / or RGBD image containing the object. Alternatively, the object within the marked box is directly captured as the RGB image and / or RGBD image containing the object.

[0107] The fourth step is an optional step, in which the mask and / or the segmented object RGB image and / or RGBD image are displayed to the user. When the user identifies an inappropriate segmentation, the third step is repeated or additional point prompts are added.

[0108] In the fifth step, the RGB image and / or RGBD image I of only a single object is fed into the monocular 3D reconstruction algorithm to obtain the corresponding 3D model, which can be superimposed on the real scene display on the interactive interface or displayed on a single page.

[0109] The monocular 3D reconstruction algorithm may be:

[0110] 5.1 Use a pre-trained generalized neural network, such as a neural network based on the PIFU algorithm or the PiFU-HD algorithm, or a neural network based on single-object optimization, such as optimizing a neural radiance field (NeRF), or a retrieval algorithm to search and compare the model with models in the database to select the most suitable one.

[0111] Alternatively, 5.1 adopts a two-stage network based on generalization and post-optimization, such as first using a pre-trained monocular reconstruction network based on encoder-decoder, or using a monocular reconstruction network with a diffusion structure to obtain a rough 3D model, or using SDS or VSD Loss function and reconstruction loss to optimize a NeRF, and then

[0112] 5.2 Use DMTNet to optimize the 3D model.

[0113] or,

[0114] 5.1 Use model retrieval to obtain the number of models O;

[0115] 5.2 Select the top-K (closest K) models in O (K is less than or equal to M);

[0116] 5.3 Use DMTNet to optimize the 3D model structure of the model in 5.2;

[0117] 5.4 Use a loss function including SDS or VSD loss and reconstruction loss to optimize the NeRF and texture of the 3D model to obtain a NeRF that matches the input image. The loss function includes a 3D prior based on Zero-1-to-3 and a 2D prior based on the Stable Diffusion algorithm.

[0118] 5.3 and 5.4 can be used separately or in reverse order;

[0119] 5.5 Based on the calculated performance of the optimized models, select the M most appropriate models based on numerical evaluation metrics and display them to the user. Alternatively, display all K models, but sort them in the order of the evaluation metrics. The evaluation metrics may be a weighted sum of one or more of CLIP-similarity, PSNR, LPIPS, reconstruction loss, or VGG loss. Example 2

[0120] Based on the above embodiment, on the basis of steps 1 to 5, add:

[0121] The sixth step is an optional step in which the user views the model obtained in the fifth step. The user moves the camera to a new perspective, and the model is rotated to the new perspective, and the model can be displayed to the user from the new perspective.

[0122] Step 7 is an optional step. The 3D model obtained in step 5 can be optimized based on the RGB image at the new viewing angle. The optimization includes optimizing the 3D model structure, optimizing the 3D model texture, and / or optimizing the model shape.

[0123] The specific implementation plan of optimization under the new perspective can be: Image-based optimization:

[0124] 7.1 Convert the RGB image into a projection of the 3D model. The projection can be a binary projection, i.e., an image where each pixel has a value of 0 or 1, where the value is 1 when the point can be projected onto the surface and 0 otherwise, or a rendering with fixed lighting and camera distance, or a normal map, or a depth map, or a direct rendering of the mesh surface. It can be implemented by an image-image translation neural network pre-trained on a large number of RGB images and 3D models, or by a renderer, such as the SoftRas renderer.

[0125] 7.2 Convert the 3D model from step 6 into a projection from this perspective.

[0126] 7.3 Compare the projection in 7.1 with the projection in 7.2. The optimizer changes the 3D model to make the error between the two projections small enough.

[0127] Alternatively, the iterative editing method of the model in step 7 can be: optimization based on the SDF value:

[0128] 7.4 Take the surface points of the outer contour area of ​​the 3D model and use a differentiable renderer.

[0129] 7.5 Take the surface points of the object corresponding to the RGB image. The surface points can be obtained by emitting rays from the four boundaries of the image and retaining only the first black pixel hit by the ray, or directly using the outer contour of the segmentation mask as the surface point.

[0130] 7.6 Take the images in 7.4 and 7.5 and optimize the model using the bidirectional 2D Chamfer loss. The optimization can be the model generated in the second step or the intermediate variables generated during the generation of monocular 3D reconstruction, such as the model's latent code.

[0131] or;

[0132] NeRF field-based optimization:

[0133] 7.8 uses an iterative algorithm where, in each iteration, a view is randomly sampled and the 3D object is rendered from that view. The new density mass added to the original NeRF representation of the model should occupy the region specified by the given newly acquired RGB image clue. We then maximize the object mask value of the newly acquired region by minimizing the following loss: the rendered image is used to compute the SDS loss to semantically align the text clues.

[0134] 7.9 calculates the distance between a point and the RGB image contour to preserve the NeRF of the part reconstructed using other views before optimization. Finally, the object mask rendering from the RGB image perspective is used to define a loss function. The optimizer optimizes the loss function.

[0135] Optimization based on depth estimation:

[0136] 7.10 Performing monocular depth estimation on the image from the new perspective to obtain depth information T1. The monocular depth estimation can use a pre-trained monocular depth estimation network;

[0137] or;

[0138] 7.10 Capture multiple images within a short time t, and assist in depth estimation based on the slight displacement and posture changes of the user's handheld camera during the continuous capture of multiple images.

[0139] or;

[0140] 7.10 Collect multiple images with different focus points within a short time t, and use depth-from-focus (DFF) and / or depth-from-defocus (DFD) and / or optical model-based methods to obtain depth information from the images with different focus points.

[0141] 7.11 Warp the depth information T2 from other perspectives to the given new perspective to obtain a depth information T1'. Based on the difference between the known depth information T1 and T1' from the new perspective, optimize T2 and thus optimize the 3D model.

[0142] Steps 6 and 7 can be repeated to obtain a sufficiently detailed model.

[0143] The above steps can be repeated for multiple objects in the scene.

[0144] Specifically, the angle difference of the view rotation can be obtained based on the posture information read by the IMU of the mobile phone, or can be obtained using an estimation based on optical flow or a Structure-from-Motion algorithm. Example 3

[0145] Based on the above embodiment or combination of embodiments, add one step after the fifth, sixth or seventh step:

[0146] Step 8: Convert the reconstructed 3D model into a compact surface-line combination, which the user can edit.

[0147] Specifically, the compact surface-line combination can be a B-Rep representation, a CSG representation, or a CAD construct sequence. Specifically, the conversion of the B-Rep representation can be performed using:

[0148] The method converts the three-dimensional model into a B-Rep representation, detects geometric primitives of different orders of the model, namely, vertices, edges and surface patches, and comprehensively models the correspondence between these primitives as a chain complex.

[0149] 8.1 is used to generate geometric primitives and their mutual relationships and estimate their probabilities. The extraction of the relationship can use a convolutional neural network, such as a sparse convolution-based encoder, and then use a Transformer decoder to output corner points, curves, and patch sets and their mutual topological relationships.

[0150] As a preferred technical solution, mutual communication between element groups is further introduced in the transformer decoder, which effectively transforms other element groups into the context for generating the current group.

[0151] 8.2 Based on the probabilistic structure predicted in 8.1, a deterministic B-Rep chain complex is recovered by solving a global optimization problem to maximize the likelihood under the constraints of structural validity and perform geometric refinement. Given the solved structure, the geometry of the elements is optimized by fitting their corresponding type geometric representations to the input points and constraining their shapes so that the topology is strictly satisfied. The closed curves (patches) solved (predicted) will be constructed accordingly.

[0152] 8.2.1 Use spline patches for fitting, fitting each patch as a spline (except planes and spheres) to the input points, curves, and corners;

[0153] 8.2.2 Fit each curve to the patch and corner points;

[0154] 8.2.3 Fit each corner point to a curve and patch;

[0155] 8.2.4 Convert spline patches into type primitives;

[0156] For each type of primitive patch, an iterative fitting process is again used to refine their geometry.

[0157] 8.2.5 Fit each type of patch to the input points, curves, and corners;

[0158] 8.2.6 Fit each curve to the patch and corner points;

[0159] 8.2.7 Fit each corner point to a curve and patch. Example 4

[0160] As an alternative to Example 3, a method of converting a 3D model into a CSG representation is adopted:

[0161] 8.1 Receive a 3D shape as input and encode it using a convolutional neural network or a multi-layer perceptron (MLP)

[0162] 8.2 Use MLP to reconstruct the shape by predicting a set of primitives from the encoded feature vector. These primitives are combined into intermediate convex and concave shapes through the CSG tree. The primitives are represented by the tetrahedron equation.

[0163] This equation determines whether a query point qj = (x, y, z) is inside or outside the primitive. The tetrahedron equation has the ability to generate several useful primitives, including a sphere, a plane, or a cylinder.

[0164] 8.3 CSG Tree Construction: The predicted primitives are then assembled into a CSG tree using a fixed order combination strategy. The CSG tree represents a 3D shape as a combination of the union and intersection of the predicted primitives.

[0165] 8.4 Shape Reconstruction: Evaluate the CSG tree to reconstruct the 3D shape. This involves evaluating the tetrahedron equations of the primitives and combining them according to the CSG tree structure to obtain the final reconstructed shape. The CSG tree represents the 3D shape as a combination of the union and intersection of the predicted primitives. The assembly process starts with the first primitive and proceeds recursively until all primitives are combined into the final CSG tree.

[0166] 8.5 Primitive Dropout: To further improve the compactness of the learned primitives, apply primitive dropout techniques based on importance metrics to remove redundant primitives. This step is optional. Example 5

[0167] As an alternative to embodiment 3 or 4, a method of converting a three-dimensional model into a CAD instruction is adopted, and the CAD instruction may be a sketch and extrude representation method.

[0168] 8.1 Encode the three-dimensional model into a feature vector. The encoding can be a convolutional neural network or a multi-layer perceptron MLP.

[0169] 8.2 Use a Transformer decoder to output the sketch and extrude instruction sets one by one.

[0170] 8.3 Use a trained Latent-Diffusion model, autoencoder, VQ-VAE, or Transformer, with a 3D model as input and the same 3D model as output, to constrain the Transformer decoder generation in 8.2. This step is optional. Example 6

[0171] Based on the above embodiment or combination of embodiments, the above embodiment adds a step after the fifth step, the sixth step, the seventh step, or the eighth step of the above embodiment:

[0172] The ninth step is to disassemble the generated 3D model. Users can edit each component after disassembly. The editing includes replacing components, changing dimensions, and displacing points, lines, and surfaces inside components.

[0173] Specifically, the disassembly step may be:

[0174] 9.1 Use a component instance segmentation network to segment the 3D model into component instances, assigning each facet to the instance it belongs to. Or;

[0175] 9.1 CNN-based segmentation network divides the shape into a set of coarse parts. The coarse parts are used as input to the recurrent network.

[0176] 9.2 Recursive Part Decomposition: The recursive network takes the part from 9.1 as input and recursively decomposes it into smaller sub-parts. This involves applying a set of RNN-based decomposition modules to the input part in a bottom-up manner. Each decomposition module takes as input a point cloud representing the part and outputs a set of sub-parts. The sub-parts are then recursively decomposed until the desired level of granularity is reached.

[0177] 9.3 Hierarchical Output: Output a hierarchical segmentation of the shape, where each part is assigned a label representing its semantic category and a set of subparts. The hierarchical segmentation is represented as a tree structure, where the root node represents the entire shape and each child node represents a part of the shape.

[0178] Training is performed using the supervised learning method steps 9.1-9.3, where the ground-truth segmentation is used as the target output. The loss function used during training is a combination of a classification loss and a geometric consistency loss. The classification loss penalizes errors in the predicted part labels, while the geometric consistency loss penalizes errors in the predicted part shapes. Example 7

[0179] A method for interacting and generating a specified model in a scene obtained by shooting, using perspective enhancement information, includes the following steps:

[0180] In the first step, the user uses a terminal to capture scene information, which can be an outdoor or indoor scene. The terminal can be a smartphone, AR / VR device or other smart terminal with a camera function.

[0181] The second step is to display the scene information on the terminal and prompt the user to select the object of interest. The selection method can be to mark a bounding box of the object of interest, or,

[0182] The scene information is displayed on the terminal, prompting the user to select an object of interest by specifying one or more points on the object area of ​​interest.

[0183] In the third step, the scene graph and the boxes or points obtained in the second step are fed into a segmentation neural network to obtain a mask belonging to the object. The mask is then used to segment the RGB image containing the object. Alternatively, the area within the marked box can be directly captured as the RGB image containing the object.

[0184] The fourth step is an optional step, in which the mask and / or the segmented object RGB image are displayed to the user. When the user identifies an inappropriate segmentation, the third step is repeated or additional point prompts are added.

[0185] In the fifth step, the segmented RGB image I is used to estimate the object's pose in space, obtaining the object's pose relative to the camera (6D Pose). This pose is then compared with a pre-defined pose range. The pre-defined pose range can be a fixed interval R1-R2 or a probability distribution over the entire space, where each pose has a "fit probability."

[0186] Step 6: If the current shooting pose in step 5 is inappropriate (e.g., the pose is outside the pose range, or the "suitability probability" within the pose range is below a threshold), the user is prompted and guided on the interactive interface to rotate the camera so that the object is presented in a suitable pose. The inappropriateness judgment is determined by the object's pose not falling within the set pose range R1-R2, or the probability of belonging to that pose falling below a certain threshold. Preferably, the threshold is 0.86.

[0187] In the seventh step, the RGB image of the single object in the appropriate pose is fed into the monocular 3D reconstruction algorithm to generate the corresponding 3D model, which can be overlaid on the interactive interface. As a preferred technical solution, the guidance on the interactive interface can indicate the direction in which the camera needs to be rotated (such as an arrow overlaid with the real scene).

[0188] The following are optional steps to further optimize the model:

[0189] In the eighth step, the model is rotated to the view corresponding to the first step. The model is also rotated to the viewing angle when shooting in the first step, and the model is optimized based on the RGB image in the third step.

[0190] The ninth step is an optional step. The obtained model is uniformly sampled from N pictures within a viewing angle range. These N pictures are fed into a discriminator. Based on the output of the discriminator, a view combination K is selected to determine whether the user needs to take additional photos to obtain a better model effect. That is, the user needs to take photos in the K view combinations to have a chance of obtaining a better modeling effect.

[0191] The discriminator can be a pre-trained CLIP encoder, or a pre-trained Diffusion-Inversion method that calculates the loss of natural images generated by diffusion, or an encoder that calculates VGG Loss.

[0192] Step 10 is optional, allowing the user to view the model. By moving the camera to a new perspective, the model rotates to that new perspective and is displayed to the user from this new perspective. This new perspective can be a combination of perspectives from step 9, in which case the interface prompts the user to shoot from an appropriate position. Alternatively, the new perspective can be any perspective that the user can freely rotate.

[0193] The eleventh step is an optional step, in which the model is optimized based on the RGB image at the new viewing angle. The optimization includes optimizing the structure of the three-dimensional model and / or optimizing the texture of the three-dimensional model and / or optimizing the shape of the model.

[0194] Steps 10 and 11 can be repeated to obtain a sufficiently detailed model and obtain more clues about texture information. Example 8

[0195] A method for interacting and generating a specified model in a scene obtained by shooting uses a fixed selection area to avoid manual selection by the user, thereby reducing the amount of computation and user interaction cost.

[0196] Replace the first to fourth steps of Example 7 with:

[0197] Step 1: The user uses a terminal to capture scene information. The scene information can be an outdoor or indoor scene. The terminal can be a smartphone, AR / VR device, or other smart terminal with a camera function. When shooting, the user is prompted to only capture a single object. As a preferred technical solution, a fixed-size frame is displayed on the shooting interface superimposed on the real-time shooting interface, prompting the user to place the object to be captured in the area within the frame. Alternatively, the framing space is a fixed-size square frame, so an additional fixed-size frame is not displayed separately.

[0198] Optionally, the user can be guided to take a photo with a solid color background. A color-based segmentation model can be used to obtain a mask of the area occupied by the object, and the mask can be used to obtain a picture with only one object as input I. Alternatively, a saliency segmentation model or other segmentation model can be used to obtain a mask of the area occupied by the object, and the mask can be used to obtain a picture with only one object as input I; or, the picture taken in this step can be directly used as input I. Example 9

[0199] A method for interacting and generating a specified model in a scene obtained by shooting uses a saliency detection method to avoid manual selection by users, thereby reducing computational complexity and user interaction costs.

[0200] Replace the first to fourth steps of Example 7 with:

[0201] Step 1: The user uses a terminal to capture scene information. The scene information can be outdoor or indoor scenes. The terminal can be a smartphone, AR / VR device, or other smart terminal with a camera function. During the shooting process, the user is prompted to only capture a single object. As a preferred technical solution, a real-time shooting interface is displayed on the shooting interface, prompting the user to place the object in the area within the frame. Alternatively, the framing space is a fixed-size square frame, so an additional fixed-size box is not displayed separately.

[0202] Optionally, the user can be guided to take a photo with a solid color background. A color-based segmentation model can be used to obtain a mask of the area occupied by the object, and the mask can be used to obtain a picture with only one object as input I. Alternatively, a saliency segmentation model or other segmentation semantic model can be used to obtain a mask of the area occupied by the object, and the mask can be used to obtain a picture with only one object as input I; or, the picture taken in this step can be directly used as input I. Example 10

[0203] In this embodiment, hint-based enhancement is used, specifically,

[0204] Before the first step, the fifth step, or the seventh step in the above embodiment, a further step is added: the user inputs or selects a hint about the photographed object. The hint can be a natural text description, such as "a table with long legs," a category selection box, a text description of the category, or a voice description. This category information is also input into the monocular 3D reconstruction algorithm and / or the 6D pose estimation algorithm to improve algorithm performance. Optionally, separate monocular 3D reconstruction algorithms can be trained for different categories, i.e., the object input hint is used to select an appropriate 3D reconstruction network model or model weights. Example 11

[0205] In this embodiment, the text hint texture is enhanced. Specifically,

[0206] Before the first step, the fifth step, or the seventh step in the above embodiment, a step is added: the user inputs a hint about the texture of the photographed object. The hint can be a natural text description, such as "a table with a retro-style texture," a voice description, or a style reference image. This texture description is input into a monocular 3D reconstruction algorithm or a separate texture generation algorithm to ensure that the generated 3D model has complete textures. Example 12

[0207] This embodiment provides a method for 3D reconstruction of multiple objects in a scene, which applies all the steps of the above embodiment to any multiple objects in a captured scene to obtain multiple model combinations in the scene. Example 13

[0208] This embodiment provides a method for rapid machining and manufacturing of parts, comprising the following steps:

[0209] The first step is to obtain a three-dimensional digital file of one or more models or multiple parts using the method of Example 12.

[0210] The second step is to input the 3D model into a parser and convert it into machine-readable instructions. For example, the parser is a slicing software that converts the 3D model into a machine-readable Gcode file.

[0211] In the third step, a processing device performs processing according to machine-readable instructions. The processing device can be a 3D printer, a laser engraving machine, a milling machine, or a CNC engraving machine. Example 14

[0212] This embodiment provides a 3D printer, including a processing unit, a collection unit, and an execution unit.

[0213] The acquisition unit includes an image acquisition device and a matching lighting component to collect RGB or RGBD information of the object, and the processing unit runs the methods of the above embodiments 1-11.

[0214] The execution unit includes a print head, a motion component, and a printing base plate. The motion component enables the print head to move to any position within the XYZ range of the printing base plate and the machine frame.

[0215] As a preferred technical solution, scale lines are printed on the printing base plate, and the scale lines represent scale information such as the angle, length, and width of the rotating object.

Claims

1. A method for interacting and generating a specified model in a scene obtained by shooting, characterized in that: The following steps are involved: In the first step, the user uses a terminal to shoot scene information. The scene information may be an outdoor or indoor scene. The terminal may be a smart phone, an AR / VR device, or other smart terminal with a camera function. The scene information obtained by shooting is RGB and / or RGBD (including depth) information. The second step is to display the scene information on the terminal and prompt the user to select an object of interest. The selection method can be to mark a bounding box of the object of interest, or, Displaying scene information on the terminal, prompting the user to select an object of interest, by specifying one or more points on the object area of ​​interest; The third step is to input the scene graph and the box or point hint obtained in the second step into a segmentation neural network to obtain a mask belonging to the object, and use the mask to segment the RGB image and / or RGBD image containing the object. Alternatively, directly intercept the object in the marked box as the RGB image and / or RGBD image containing the object; The fourth step is an optional step, in which the mask and / or the segmented object RGB image and / or RGBD image is displayed to the user. When the user identifies an inappropriate segmentation, the third step is repeated or additional point prompts are added; In the fifth step, the RGB image and / or RGBD image I of only a single object is fed into the monocular 3D reconstruction algorithm to obtain the corresponding 3D model, which can be superimposed on the real scene display on the interactive interface or displayed on a single page.

2. The method for interacting and generating a specified model in a scene obtained by shooting according to claim 1, characterized in that: The following subsequent steps may also be included: The sixth step is an optional step, in which the user views the model obtained in the fifth step; the user moves the camera to another new perspective, and the model is also rotated to the new perspective, and the model can be displayed to the user at the new perspective; The seventh step is an optional step, wherein the three-dimensional model obtained in the fifth step can be optimized based on the RGB image of the new perspective at the new perspective; the optimization includes optimization of the three-dimensional model structure and / or optimization of the three-dimensional model texture and / or optimization of the model shape; Steps 6 and 7 can be repeated to obtain a sufficiently refined model; The above steps can be repeated for multiple objects in the scene.

3. The method for interacting and generating a specified model in a scene obtained by shooting according to claim 2, characterized in that: The angle difference of the view rotation can be obtained based on the posture information read by the IMU of the mobile phone, or can be obtained using an estimation based on optical flow or a Structure-from-Motion algorithm.

4. The method for interacting and generating a specified model in a scene obtained by shooting according to claim 1, characterized in that: The following subsequent steps may also be included: adding a step after the fifth step, the sixth step or the seventh step: Step 8: Convert the reconstructed 3D model into a compact surface-line combination, which the user can edit; Specifically, the compact surface-line combination can be a B-Rep representation, a CSG representation, or a CAD construct sequence. Specifically, the conversion of the B-Rep representation can be: The method converts the 3D model into a B-Rep representation, detects geometric primitives of different orders of the model, namely, vertices, edges and surface patches, and comprehensively models the correspondence between these primitives as a chain complex.

5. The method for interacting and generating a specified model in a scene obtained by shooting according to claim 4, characterized in that: The following subsequent steps may also be included: the eighth step specifically includes: 8.1 is used to generate geometric primitives and their mutual relations and estimate their probabilities. The extraction of the said relations can adopt a convolutional neural network, such as a sparse convolution-based encoder, and then use a transformer decoder to output corner points, curves and patch sets and their mutual topological relations; As a preferred technical solution, mutual communication between element groups is further introduced in the transformer decoder, which effectively transforms other element groups into the context for generating the current group; 8.2 Based on the probabilistic structure predicted in 8.1, a deterministic B-Rep chain complex is recovered by solving a global optimization problem to maximize the likelihood under the constraints of structural validity and perform geometric refinement; given the solved structure, the geometry of the elements is optimized by fitting their corresponding type geometric representations to the input points and constraining their shapes so that the topological structure is strictly satisfied; the closed curves (patches) solved (predicted) will be constructed accordingly; where, 8.2.1 Fitting using spline patches, fitting each patch as a spline (except planes and spheres) to the input points, curves, and corners; 8.2.2 Fit each curve to the patch and corner points; 8.2.3 Fit each corner point to the curve and patch; 8.2.4 Convert spline patches to type primitives; For each type of primitive patch, an iterative fitting process is again used to refine their geometry; 8.2.5 Fit each type of patch to the input points, curves, and corners; 8.2.6 Fit each curve to the patch and corner points; 8.2.7 Fit each corner point to a curve and patch.

6. The method for interacting and generating a specified model in a scene obtained by shooting according to claim 4, characterized in that: The following subsequent steps may also be included: the eighth step specifically includes: The method of converting the 3D model into CSG representation is adopted: 8.1 Receive a 3D shape as input and encode it using a convolutional neural network or a multi-layer perceptron MLP 8.2 Use MLP to reconstruct the shape by predicting a set of primitives from the encoded feature vector, which are combined into intermediate convex and concave shapes through the CSG tree. The primitives are represented by tetrahedron equations, ， This equation determines whether a query point qj = (x, y, z) is inside or outside the primitive; the tetrahedron equation has the ability to generate several useful primitives, including a sphere, a plane, or a cylinder; 8.3 CSG Tree Construction: The predicted primitives are then assembled into a CSG tree using a fixed order combination strategy; the CSG tree represents a 3D shape as a combination of the union and intersection of the predicted primitives; 8.4 Shape Reconstruction: Evaluate the CSG tree to reconstruct the 3D shape; this involves evaluating the tetrahedral equations of the primitives and combining them according to the CSG tree structure to obtain the final reconstructed shape; the CSG tree represents the 3D shape as a combination of the union and intersection of the predicted primitives; the assembly process starts with the first primitive and proceeds recursively until all primitives are combined into the final CSG tree; 8.5 Primitive dropout: To further improve the compactness of learned primitives, apply primitive dropout techniques based on importance metrics to remove redundant primitives; this step is optional.

7. The method for interacting and generating a specified model in a scene obtained by shooting according to claim 4, characterized in that: The following subsequent steps may also be included: the eighth step specifically includes: A method of converting a three-dimensional model into a CAD instruction is adopted, wherein the CAD instruction may be a representation method of sketch and extrude; 8.1 Encode the three-dimensional model into a feature vector, wherein the encoding may be a convolutional neural network or a multi-layer perceptron MLP; 8.2 Use a Transformer decoder to output the sketch and extrude instruction set one by one; 8.3 Use a trained Latent-Diffusion model or autoencoder or VQ-VAE or Transformer, with the input being a three-dimensional model and the output being the three-dimensional model, to constrain the transformer decoder generation in 8.2; this step is optional.

8. The method for interacting and generating a specified model in a scene obtained by shooting according to any one of claims 1, 4-7, characterized in that it can also include the following subsequent steps: Add a step after step 5 or 6 or 7 or 8: The ninth step is to disassemble the generated 3D model. Users can edit each component after disassembly, including replacement of components, size change, displacement of points, lines and surfaces inside components, etc. Specifically, the disassembly step may be: 9.1 Use a component instance segmentation network to segment the 3D model and assign each face to the instance to which it belongs; or, 9.1 CNN-based segmentation network divides the shape into a set of coarse parts. The coarse parts are used as input to the recurrent network; 9.2 Recursive component decomposition: The recursive network will take the component from 9.1 as input and recursively decompose it into smaller sub-components; this involves applying a set of RNN-based decomposition modules to the input component in a bottom-up manner; each decomposition module takes as input a point cloud representing the component and outputs a set of sub-components; then, the sub-components are recursively decomposed until the desired level of granularity is reached; 9.3 Hierarchical Output: Output a hierarchical segmentation of the shape, where each component is assigned a label representing its semantic category and a set of subcomponents; the hierarchical segmentation is represented as a tree structure, where the root node represents the entire shape and each child node represents a component of the shape; Training is performed using the steps of supervised learning method 9.1-9.3, where the true segmentation is used as the target output; the loss function used during training is a combination of classification loss and geometric consistency loss; the classification loss penalizes errors in the predicted part labels, while the geometric consistency loss penalizes errors in the predicted part shapes.

9. A method for interacting and generating a specified model in a scene obtained by shooting, characterized in that it includes the following steps: In the first step, the user uses a terminal to shoot scene information, which may be an outdoor or indoor scene, and the terminal may be a smart phone, an AR / VR device, or other smart terminal with a camera function; The second step is to display the scene information on the terminal and prompt the user to select an object of interest. The selection method can be to mark a bounding box of the object of interest, or, Displaying scene information on the terminal, prompting the user to select an object of interest, by specifying one or more points on the object area of ​​interest; The third step is to input the scene graph and the box or point hint obtained in the second step into a segmentation neural network to obtain the mask belonging to the object, and use the mask to segment the RGB image containing the object. Alternatively, directly capture the area in the marked box as the RGB image containing the object; The fourth step is an optional step, in which the mask and / or the segmented object RGB image are displayed to the user. When the user identifies an inappropriate segmentation, the third step is repeated or additional point prompts are added; Step 5: Use the segmented RGB image I to estimate the pose of an object in space, obtain the pose of the object relative to the camera (6D Pose), and compare the pose with a pre-set pose range; the pre-set pose range can be a fixed interval R1-R2, or a probability distribution in the whole space, that is, there is a "fit probability" for each pose; Step 6: If the current shooting posture in step 5 is not suitable (such as the posture exceeds the posture range, or the "suitability probability" of the posture range is lower than a threshold), prompt the user and guide the user to rotate the camera on the interactive interface so that the object is presented in a suitable posture; the judgment of being unsuitable is that the posture of the object does not belong to the range of the set posture interval R1-R2, or the probability value of belonging to the posture is lower than a certain threshold; preferably, the threshold is 0.86; In the seventh step, the RGB image of a single object in a suitable posture is sent to the monocular 3D reconstruction algorithm to obtain the corresponding 3D model, which can be superimposed and displayed on the interactive interface; as a preferred technical solution, the guidance on the interactive interface can be an indication of the direction in which the camera needs to be rotated (such as an arrow superimposed on the real scene).

10. The method for interacting and generating a specified model in a scene obtained by shooting according to claim 9, characterized in that: The following subsequent steps may also be included: Step 8: Rotate the model to the view corresponding to the first step. The model is also rotated to the viewing angle when the model was taken in the first step. The model is optimized based on the RGB image in the third step. The ninth step is an optional step, in which the obtained model uniformly samples N pictures within a viewing angle range, and the N pictures are sent to a discriminator. According to the result output by the discriminator, a viewing angle combination K is selected to determine whether the user needs to take additional photos to obtain a better modeling effect, that is, the user needs to take photos in the K viewing angle combinations to have a chance of obtaining a better modeling effect; The discriminator can be a pre-trained CLIP encoder, or a pre-trained Diffusion-Inversion method that calculates the loss of natural images that conform to diffusion generation, or an encoder that calculates VGG Loss; The tenth step is an optional step, in which the user views the model; the user moves the camera to another new perspective, and the model is also rotated to the new perspective, and the model can be displayed to the user at the new perspective; the new perspective can be a combination of perspectives in the ninth step, in which case the interface prompts the user to shoot at a suitable position, and the new perspective can also be any perspective that the user can rotate freely; The eleventh step is an optional step, in which the model can be optimized based on the RGB image of the new perspective at the new perspective; the optimization includes optimization of the three-dimensional model structure and / or optimization of the three-dimensional model texture and / or optimization of the model shape; Steps 10 and 11 can be repeated to obtain a sufficiently detailed model and obtain more clues about texture information.

11. A method for interacting and generating a specified model in a scene obtained by shooting, characterized in that: The following steps are involved: Step 1: The user uses a terminal to shoot scene information, which can be an outdoor or indoor scene, and the terminal can be a smart phone, AR / VR device or other smart terminal with a camera function; when shooting, the user is prompted to shoot only a single object. As a preferred technical solution, a fixed-size frame is displayed on the shooting interface and superimposed on the real-time shooting interface, prompting the user to place the object to be photographed in the area within the frame; or, the framing space is a fixed-size square frame, so an additional fixed-size frame is not displayed separately; Optionally, the user may be guided to take a photo with a solid color background, and then a color-based segmentation model may be used to obtain a mask of the area occupied by the object, and the mask is used to obtain a picture with only one object as input I. Alternatively, a saliency segmentation model or other segmentation model may be used to obtain a mask of the area occupied by the object, and the mask is used to obtain a picture with only one object as input I. Alternatively, the picture taken in this step is directly used as input I; The second step is to use the segmented RGB image I to estimate the pose of an object in space, obtain the pose of the object relative to the camera (6D Pose), and compare the pose with a pre-set pose range; the pre-set pose range can be a fixed interval R1-R2, or a probability distribution in the whole space, that is, there is a "fit probability" for each pose; Step 3: If the current shooting posture in step 5 is not suitable (such as the posture exceeds the posture range, or the "suitability probability" of the posture range is lower than a threshold), prompt the user and guide the user to rotate the camera on the interactive interface so that the object is presented in a suitable posture; the judgment of being not suitable is that the posture of the object does not belong to the range of the set posture interval R1-R2, or the probability value of belonging to the posture is lower than a certain threshold; preferably, the threshold is 0.86; In the fourth step, the RGB image of a single object in a suitable posture is sent to the monocular 3D reconstruction algorithm to obtain the corresponding 3D model, which can be superimposed and displayed on the interactive interface; as a preferred technical solution, the guidance on the interactive interface can be an indication of the direction in which the camera needs to be rotated (such as an arrow superimposed on the real scene).

12. A method for interacting and generating a specified model in a scene obtained by shooting, characterized in that: The following steps are involved: Step 1: The user uses a terminal to shoot scene information, the scene information may be an outdoor or indoor scene, and the terminal may be a smart phone, an AR / VR device or other smart terminal with a camera function; when shooting, the user is prompted to shoot only a single object; as a preferred technical solution, a real-time shooting interface is displayed on the shooting interface, prompting the user to place the object to be shot in the area within the frame; or, the framing space is a square frame of a fixed size, so an additional square frame of a fixed size is not displayed separately; Optionally, the user may be guided to take a photo with a pure color background, and then a color-based segmentation model may be used to obtain a mask of the area occupied by the object, and the mask is used to obtain a picture with only one object as input I. Alternatively, a saliency segmentation model or other segmentation semantic model may be used to obtain a mask of the area occupied by the object, and the mask is used to obtain a picture with only one object as input I. Alternatively, the picture taken in this step is directly used as input I; The second step is to use the segmented RGB image I to estimate the pose of an object in space, obtain the pose of the object relative to the camera (6D Pose), and compare the pose with a pre-set pose range; the pre-set pose range can be a fixed interval R1-R2, or a probability distribution in the whole space, that is, there is a "fit probability" for each pose; Step 3: If the current shooting posture in step 5 is not suitable (such as the posture exceeds the posture range, or the "suitability probability" of the posture range is lower than a threshold), prompt the user and guide the user to rotate the camera on the interactive interface so that the object is presented in a suitable posture; the judgment of being not suitable is that the posture of the object does not belong to the range of the set posture interval R1-R2, or the probability value of belonging to the posture is lower than a certain threshold; preferably, the threshold is 0.86; In the fourth step, the RGB image of a single object in a suitable posture is sent to the monocular 3D reconstruction algorithm to obtain the corresponding 3D model, which can be superimposed and displayed on the interactive interface; as a preferred technical solution, the guidance on the interactive interface can be an indication of the direction in which the camera needs to be rotated (such as an arrow superimposed on the real scene).

13. A method for interacting and generating a specified model in a scene obtained by shooting according to any one of claims 1 to 12, characterized in that: Add a step before the first step, or before the fifth step, or before the seventh step: The user inputs or selects a hint about the photographed object, which can be a natural text description, such as "a table with long legs", or a category selection box or a category text description, or a voice description; the category information will be input into the monocular 3D reconstruction algorithm and / or the pose estimation (6D Pose estimation) algorithm to enable the algorithm to perform better; optionally, separate monocular 3D reconstruction algorithms are trained for different categories, that is, the hint of the object input is used to select a suitable 3D reconstruction network model or model weight.

14. A method for interacting and generating a specified model in a scene obtained by shooting according to any one of claims 1 to 12, characterized in that: Add a step before the first step, or before the fifth step, or before the seventh step: The user inputs a hint about the texture of the photographed object, which can be a natural text description, such as "a table with a retro-style texture", or a voice description, or a style reference picture; the texture description is input into a monocular 3D reconstruction algorithm or a separate texture generation algorithm, so that the generated 3D model has a complete texture.

15. A method for three-dimensional reconstruction of multiple objects in a scene, characterized in that: The method described in any one of claims 1 to 14 is used to perform the operations on any multiple objects in the shooting scene to obtain multiple model combinations in the scene.

16. A method for rapid processing and manufacturing of parts, characterized in that: The following steps are involved: The first step is to obtain a three-dimensional digital file of one or more models or multiple parts using the method according to claim 15; The second step is to input the 3D model into a parser and convert it into machine-readable instructions; for example, the parser is a slicing software that converts the 3D model into a machine-readable Gcode file; In the third step, the processing device performs processing according to the machine-readable instructions; the processing device can be a 3D printer, a laser engraving machine, a milling machine, or a CNC engraving machine.

17. A 3D printer, characterized in that: It includes at least one processing unit, one collection unit, and one execution unit; The acquisition unit includes an image acquisition device and a matching lighting component to collect RGB or RGBD information of the object. The processing unit executes the method according to any one of claims 1 to 15; The execution unit includes a print head, a motion component, and a printing base. The motion component enables the print head to move to any position within the XYZ range of the printing base and the machine frame; As a preferred technical solution, scale lines are printed on the printing base plate, and the scale lines represent scale information such as the angle, length, and width of the rotating object.

Images