Three-dimensional reconstruction method and device for object

Abstract

The application discloses a kind of object three-dimensional reconstruction method and device, method includes: robot is carried out around movement to the object to be reconstructed, and data acquisition is carried out using depth camera at current best acquisition viewpoint, and the viewpoint selection set V in the reachable space of mechanical arm after robot moves a step is precalculated at current best acquisition viewpoint iter , the acquisition data of current best acquisition viewpoint and the viewpoint selection set V iter Input best viewpoint model, the best viewpoint model calculates next best acquisition viewpoint, until the end of around one week;Based on the acquisition pose of depth camera, splice the point cloud acquisition data of all depth cameras in the whole around process, obtain the three-dimensional reconstruction model of the object to be reconstructed.The application simultaneously considers viewpoint evaluation and sequencing, proposes a set of original double-branch neural network, simultaneously considers the evaluation and sequencing of viewpoint, improves feature extraction accuracy.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and apparatus for three-dimensional reconstruction of objects. Background Technology

[0002] 3D object reconstruction has always been a crucial application of computer vision. Researchers have previously proposed two different solutions for 3D object reconstruction: passive vision, which focuses on acquiring as much information as possible from a fixed perceptual environment; and active vision, which attempts to enable the sensor to actively adjust its parameters and adapt to the environment, acquiring more effective data to suit different tasks. With the improvement of robotics and sensor performance, active vision is receiving increasing attention, and many related viewpoint planning tasks, such as viewpoint path planning and finding the next optimal viewpoint, are gradually being developed.

[0003] The common workflow of active vision is as follows: Figure 1 As shown. The entire process includes four stages: sensor viewpoint selection, robot path planning, sensor information acquisition, and known information update and correction. These four stages form a closed loop, which the robot executes repeatedly during the visual task until a termination condition is met.

[0004] Viewpoint planning in active vision systems is based on detected workspace information. The workspace refers to the entire three-dimensional space containing the target object and free space. Active vision systems use newly acquired information to update their environment model, thereby guiding viewpoint planning. Therefore, the choice of workspace data representation has a significant impact on the planning calculation method and efficiency.

[0005] Voxels, due to their simplicity and ease of use, and their balance between computational cost and result quality, have become the preferred data format for many algorithms. They use occupancy grids or octrees to discretize the workspace. Although some precision is sacrificed, voxels can still simulate complex object surfaces to a certain extent, and the computational complexity of ray propagation for viewpoint planning in discrete space is acceptable. Therefore, voxel representation offers a good balance between quality and efficiency.

[0006] Triangular meshes are a common model storage format for 3D models. They are also used in some viewpoint planning studies to represent intermediate states of the environment. Triangular meshes can capture fine details of the surface of scanned objects very well. However, richer detail means higher computational cost. Furthermore, triangular meshes only describe the surface of the scanned object and cannot model blank or unknown spaces. Therefore, triangular mesh representation is not as widely used as voxel representation.

[0007] However, neither voxel nor triangular patch data can meet real-time requirements. Point clouds, being the native data collected by most existing depth sensors, can achieve real-time performance. Each point in a point cloud contains not only spatial information but also reflectivity and color information. Compared to voxel representation, point cloud data can reconstruct more surface details of an object. Furthermore, point cloud data is more direct and simpler than triangular patch representation, but extracting effective features for active vision problems from point cloud data remains a significant challenge.

[0008] In recent years, deep learning methods have shone brightly in many fields, such as image classification. However, there is very little research on using deep learning methods for optimal viewpoint prediction.

[0009] Deep learning-based methods are computationally fast and meet real-time requirements, but the potential of these methods in active vision problems has not yet been fully explored through the mining, representation, and design of input information and neural networks, and accuracy is still lacking.

[0010] In terms of building physical robots, current implementation plans require a lot of time to calculate the best viewpoint and plan motion, making it difficult to meet real-time requirements. Summary of the Invention

[0011] This invention solves the problems of poor real-time performance, insufficient stability, and inadequate reconstruction accuracy in current robot active vision systems. The technical solution adopted in this invention is as follows.

[0012] A method for 3D reconstruction of an object, comprising:

[0013] The robot performs orbital motion around the object to be reconstructed, using a depth camera to acquire data from the current optimal viewpoint. It also pre-calculates the set of viewpoints V that the robot's arm can reach in space after moving one step from the current optimal viewpoint. iter The data collected from the current best viewpoint and the viewpoint selection set V are combined. iter Input the optimal viewpoint model, which calculates the next optimal acquisition viewpoint and acquires data until the cycle is completed;

[0014] Based on the pose acquired by the depth camera, the point cloud data acquired by all depth cameras during the entire orbiting process are stitched together to obtain a three-dimensional reconstruction model of the object to be reconstructed.

[0015] Optionally, the acquisition data of the current best acquisition viewpoint and the viewpoint selection set V are combined. iter Input the optimal viewpoint model, including:

[0016] Viewpoints that do not belong to the selected viewpoint set V selected And belongs to the viewpoint selection set V iterThe viewpoint, along with the data collected from the current best acquisition viewpoint, is input into the best viewpoint model for evaluation, thereby obtaining the next best acquisition viewpoint.

[0017] Alternatively, the process of constructing the optimal viewpoint model is as follows:

[0018] The following formula is used to evaluate the input point cloud set P and the complete point cloud set P sampled from the object to be reconstructed. c The coverage rate is calculated using the point cloud set P as input, and the coverage rate C of the reconstruction model at the current best acquisition viewpoint relative to the original model as the label. The mean squared error loss is used to train the evaluation branch of the best viewpoint model.

[0019]

[0020] Where U is the Heaviside step function, and ∈ is the distance threshold;

[0021] Using point cloud set P as input and coverage C as label, the following loss is designed using cross-entropy:

[0022]

[0023] i, j are viewpoint numbers, P ij To estimate the probability of ranking accuracy, the actual probability of ranking accuracy is calculated based on the labels of the optimal viewpoint model. R is a sorted set.

[0024] Cross-entropy of viewpoint pairs

[0025] By summing the cross-entropy of all viewpoint pairs, a ranking loss (LR) is designed to train the ranking branch of the optimal viewpoint model.

[0026]

[0027] i and j are viewpoint pairs, which are random pairs of two viewpoints from all available viewpoints.

[0028] The prediction results of the ranking branch are combined with the features of the evaluation branch to train the best viewpoint model based on the two branches.

[0029] Optionally, after obtaining the three-dimensional reconstruction model of the object to be reconstructed, the model is further constructed based on the point cloud of the object to be reconstructed and the complete point cloud set P. c To address the differences, point cloud completion and registration post-processing are performed.

[0030] Optionally, the step of collecting data at the current optimal acquisition viewpoint includes:

[0031] Based on inverse dynamics, the optimal acquisition viewpoint is calculated into the joint values ​​of the robotic arm to determine the target pose of the robotic arm.

[0032] Using a sampling-based planning algorithm, the trajectory of the robotic arm to reach the target pose is calculated and executed.

[0033] The depth camera completes the data acquisition process.

[0034] Optionally, the first optimal acquisition viewpoint can be selected by using different object models in a simulation environment to evaluate the value of the viewpoint based on the viewpoint and its coverage, and then selecting the viewpoint that gives the highest initial average coverage of the different object models as the first optimal acquisition viewpoint.

[0035] Optionally, the robot includes a mobile chassis, a single-line radar, a robotic arm, and a depth camera mounted on the robotic arm.

[0036] Optionally, the mobile chassis drives the robot to move around the object to be reconstructed, senses the distance to the object through a single-line radar, and automatically moves to the optimal acquisition viewpoint.

[0037] Optionally, the viewpoints that do not belong to the selected viewpoint set V selected And belongs to the viewpoint selection set V iter The evaluation of the optimal viewpoint model for viewpoint input includes:

[0038] All viewpoints surrounding the object to be reconstructed are categorized as follows, with only viewpoints in category 2 being evaluated as the best viewpoint model:

[0039]

[0040] Combine the point cloud of the object to be reconstructed captured from the current best acquisition viewpoint with v state The set of viewpoint states with a value of 2 is input into the optimal viewpoint model to obtain the next optimal acquisition viewpoint.

[0041] The present invention also provides an electronic device, including a processor and a memory, wherein a three-dimensional object reconstruction program is stored in the memory, and the three-dimensional object reconstruction program, when executed by the processor, performs the three-dimensional object reconstruction method as described above.

[0042] This invention, in the implementation of a real-time robot active vision system, establishes a unique robot system and designs an active vision object reconstruction process for a single object, realizing a complete real-time and stable process for viewpoint calculation, motion planning, and acquisition and reconstruction. This invention has the following beneficial effects:

[0043] (1) In the robot active vision method based on generation and testing, the selection of candidate viewpoints is not only an evaluation problem, but also a ranking problem. This invention proposes an original dual-branch neural network that takes into account both the evaluation and ranking of viewpoints, thereby improving the efficiency and accuracy of feature extraction from point cloud data in active vision problems.

[0044] (2) A real-time robot active vision system based on point cloud data is realized, which enables the system to meet the real-time requirements and the calculation frame rate can reach 20 to 30.

[0045] (3) A system design including a mobile chassis, robotic arm, single-line radar and depth camera was designed, and a real-time, stable and high reconstruction accuracy robot active vision system was realized. Attached Figure Description

[0046] Figure 1 A flowchart for active vision object reconstruction;

[0047] Figure 2 This is a flowchart of the object three-dimensional reconstruction method according to an embodiment of the present invention;

[0048] Figure 3 This is a side view of the robot according to an embodiment of the present invention;

[0049] Figure 4 This is a top view of the robot according to an embodiment of the present invention. Detailed Implementation

[0050] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] The object 3D reconstruction method of this embodiment includes the following steps:

[0052] Step S1: The robot performs a circular motion around the object to be reconstructed, collects data from the current optimal acquisition viewpoint, and pre-calculates the set of viewpoints V that the robotic arm can reach in the space after moving a certain step from the current optimal acquisition viewpoint. iter The data collected from the current best viewpoint and the viewpoint selection set V are combined. iter Input the optimal viewpoint model, which calculates the next optimal acquisition viewpoint, until the cycle is completed.

[0053] Step S1 includes:

[0054] Step S11: Pre-calculate the reachable space of the robotic arm after the robot moves a certain step length from the current optimal acquisition viewpoint, and obtain the viewpoint selection set V based on the reachable space. iter .

[0055] For example, for the object to be reconstructed, the total candidate viewpoint set V around it in one revolution contains 240 viewpoints. These 240 points are uniformly sampled within an angle range of [30, 60] on the upper hemisphere centered on the centroid of the object to be reconstructed. Based on the arm span of the robotic arm and the pre-calculated reachable space, each viewpoint selects a set V. iter It includes 40 viewpoints (for a mobile robot on a circle, the robotic arm can reach 40 out of 240 points), and it iterates around it 9 times. The robot moves a fixed angle of 40 degrees each time, so it is equivalent to the Viter set rotating uniformly with the number of iterations. Each rotation coincides with 40 out of the 240 points in V.

[0056] Step S12, based on whether it belongs to the already selected viewpoint set V selected Whether a viewpoint belongs to the selectable viewpoint range in the current iteration is determined by Viter, which categorizes all viewpoints within the range of the object to be reconstructed into three classes. Only viewpoints in class 2 require evaluation using the best viewpoint model in the current iteration.

[0057]

[0058] Step S13: Combine the point cloud of the object to be reconstructed acquired from the current best acquisition viewpoint with v state The set of viewpoint states with a value of 2 is input into the optimal viewpoint model to obtain the next optimal acquisition viewpoint.

[0059] The process of constructing the optimal viewpoint model is as follows:

[0060] Step SA generates training data for the optimal viewpoint model, for each viewpoint v. s The effect label is defined as the coverage C of the reconstructed model to the original model. The input point cloud set P and the complete point cloud set P obtained by upsampling from the original model are evaluated according to the following formula. c The coverage is used as input to the point cloud set P and the coverage C is used as the label. The mean squared error loss is used to train the evaluation branch of the best viewpoint model.

[0061]

[0062] Where U is the Heaviside step function, and ∈ is the distance threshold, set to 0.1cm.

[0063] SB, taking the point cloud set P as input and the coverage C as the label, combines the characteristics of the best viewpoint task and cross-entropy, and designs the following loss:

[0064]

[0065] i, j are viewpoint numbers, P ij To estimate the probability of ranking accuracy, the actual probability of ranking accuracy is calculated based on the labels of the training set of the optimal viewpoint model. R is a sorted set.

[0066] Cross-entropy of viewpoint pairs

[0067] Summing the cross-entropy of all viewpoint pairs, we design a ranking loss (LR) to train the ranking branch of the optimal viewpoint model.

[0068]

[0069] i, where i is a viewpoint pair, is a pair of viewpoints randomly selected from all viewpoints.

[0070] SC combines the prediction results of the ranking branch with the feature splicing and processing of the evaluation branch to achieve the combination of two branches and train the best viewpoint model based on the two branches.

[0071] Step S2: Based on the pose acquired by the depth camera, the point cloud data acquired by all depth cameras during the entire orbiting process are stitched together to form a complete object reconstruction model, thereby completing the active visual reconstruction of the object to be reconstructed.

[0072] Step S3: Output the complete object reconstruction model, which can be determined based on the point cloud of the object to be reconstructed and the complete point cloud set P. c Depending on the needs, decide whether to perform post-processing operations such as point cloud completion and point cloud registration.

[0073] Furthermore, the data acquisition at the current optimal acquisition viewpoint includes:

[0074] S111, based on inverse dynamics, the current optimal acquisition viewpoint is calculated as the 6 joint values ​​of the robotic arm, and the target pose of the robotic arm is determined.

[0075] S112 uses a sampling-based planning algorithm to calculate the trajectory of the robotic arm to the target pose and executes the trajectory.

[0076] S113, the depth camera completes the acquisition action and inputs the acquired point cloud data into the current model to be reconstructed, which will then be used as input data for the next iteration into the optimal viewpoint model.

[0077] Furthermore, the method for selecting the first optimal acquisition viewpoint (optimal fixed initial acquisition point) is to evaluate the value of the initial acquisition point in advance in the simulation environment using different object models in the ShapeNet database based on the viewpoint and its coverage, and select the viewpoint that gives the highest initial average coverage of the different object models as the first optimal acquisition viewpoint.

[0078] Furthermore, such as Figure 3 , Figure 4 As shown, the robot includes a mobile chassis 10, a single-line radar 20, a robotic arm 30, and a depth camera 40 mounted on the robotic arm 30. The mobile chassis 10 drives the robot to move around the object to be reconstructed, ensuring the robot faces the object directly. The single-line radar 20 senses the distance to the object and automatically moves to the optimal fixed initial acquisition point. The single-line radar's field of view is a 20-degree range directly in front of the robot. The 20-degree range is merely illustrative; different radar field of view ranges can be used depending on different requirements.

[0079] The present invention also provides an electronic device, which is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it may be a smartphone, tablet computer, laptop computer, desktop computer, server, etc. The electronic device includes, but is not limited to, a memory and a processor that are communicatively connected to each other. Specifically, the memory includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the electronic device, such as the hard disk or RAM of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory may include both internal storage units and external storage devices of the electronic device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the electronic device, such as the code for the 3D reconstruction of the object. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0080] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the electronic device, such as performing control and processing related to data interaction or communication with the electronic device. In this embodiment, the processor is used to run program code stored in the memory or process data, such as running the object 3D reconstruction program.

[0081] The memory containing the readable storage medium may include an operating system, a 3D object reconstruction program, etc. When the processor executes the 3D object reconstruction program in the memory, it performs the steps described above, which will not be repeated here. In this embodiment, the 3D object reconstruction program stored in the memory can be divided into one or more program modules. These one or more program modules are stored in the memory and can be executed by one or more processors to complete this application; specific details will not be repeated here.

[0082] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications are all within the protection scope of the claims of the present invention.

Claims

1. A method for three-dimensional reconstruction of an object, characterized in that, include: The robot orbits the object to be reconstructed, using a depth camera to collect data from the current optimal viewpoint. It also pre-calculates the set of viewpoints within the space reachable by the robot arm after moving one step from the current optimal viewpoint. The data collected from the current best viewpoint and the viewpoint selection set are combined. Input the optimal viewpoint model, which calculates the next optimal acquisition viewpoint and acquires data until the cycle is completed; Based on the pose acquired by the depth cameras, the point cloud data acquired by all depth cameras during the entire orbiting process are stitched together to obtain a 3D reconstructed model of the object to be reconstructed. The process of constructing the optimal viewpoint model is as follows: The input point cloud set is evaluated using the following formula. and the complete point cloud set obtained by sampling from the object to be reconstructed Coverage, in point cloud set As input, the coverage of the reconstructed model over the original model at the current best acquisition viewpoint. Using the labels, the mean squared error loss is used to train the evaluation branch of the optimal viewpoint model. Where U is the Heaviside step function, Distance threshold Indicates viewpoint; Indicates the category of viewpoint division; Point cloud aggregation For input, coverage For the labels, and combining cross-entropy, the following loss is designed: i and j are the viewpoint numbers. This is a probabilistic estimate of the sorting accuracy. Given a sorted set, calculate the actual probability of sorting accuracy based on the labels of the optimal viewpoint model. , Sum the cross-entropy of all viewpoint pairs and design a ranking loss. To train the ranking branch of the optimal viewpoint model. A viewpoint pair is a pair of viewpoints randomly selected from all viewpoints. By combining the ranking branch with the evaluation branch, a best viewpoint model based on two branches is trained.

2. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The data collected from the current best acquisition viewpoint and the viewpoint selection set are then... Input the optimal viewpoint model, including: Viewpoints that do not belong to the selected viewpoint set And belongs to the viewpoint selection set The viewpoint, along with the data collected from the current best acquisition viewpoint, is input into the best viewpoint model for evaluation, thereby obtaining the next best acquisition viewpoint.

3. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, After obtaining the 3D reconstruction model of the object to be reconstructed, the model is further constructed based on the point cloud of the object to be reconstructed and the complete point cloud set. To address the differences, point cloud completion and registration post-processing are performed.

4. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The process of collecting data from the current optimal acquisition viewpoint includes: Based on inverse dynamics, the optimal acquisition viewpoint is calculated into the joint values ​​of the robotic arm to determine the target pose of the robotic arm. Using a sampling-based planning algorithm, the trajectory of the robotic arm to reach the target pose is calculated and executed. The depth camera completes the data acquisition process.

5. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The method for selecting the first optimal acquisition viewpoint is to evaluate the value of the viewpoint based on the viewpoint and its coverage using different object models in a simulation environment, and then select the viewpoint that has the highest initial average coverage of the different object models as the first optimal acquisition viewpoint.

6. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The robot includes a mobile chassis, a single-line radar, a robotic arm, and a depth camera mounted on the robotic arm.

7. The method for three-dimensional reconstruction of an object according to claim 6, characterized in that, The mobile chassis drives the robot to move around the object to be reconstructed, using a single-line radar to sense the distance to the object and automatically move to the optimal acquisition viewpoint.

8. The method for three-dimensional reconstruction of an object according to claim 2, characterized in that, The viewpoints that do not belong to the selected viewpoint set And belongs to the viewpoint selection set The optimal viewpoint model is evaluated based on the input viewpoint to obtain the next optimal acquisition viewpoint, including: All viewpoints surrounding the object to be reconstructed are categorized as follows, with only viewpoints in category 2 being evaluated as the best viewpoint model: Combine the point cloud of the object to be reconstructed captured from the current best acquisition viewpoint with... The set of viewpoint states with a value of 2 is input into the optimal viewpoint model to obtain the next optimal acquisition viewpoint.

9. An electronic device, characterized in that, The device includes a processor and a memory, wherein the memory stores a three-dimensional object reconstruction program, which, when executed by the processor, performs the three-dimensional object reconstruction method according to any one of claims 1 to 8.

Images