Method and system for transparent object reconstruction based on rgb-d camera consistent depth prediction

Abstract

The application provides a transparent object reconstruction method and system based on RGB-D camera consistency depth prediction, acquires an RGB image, a depth image and camera internal parameter information of an RGB-D camera containing a transparent object; uses a transparent object segmentation network to obtain a mask of the transparent object based on the RGB image; applies the mask on the depth image to acquire a depth value of a non-transparent object region, and uses the camera internal parameter information to acquire point cloud of the image in a three-dimensional space; uses a pre-trained consistency depth prediction neural network to perform depth image prediction based on the mask, the RGB image and the three-dimensional space point cloud to obtain a restored depth image; and performs three-dimensional reconstruction based on the restored depth image, the RGB image and the camera internal parameter information to obtain a final result. The application can accurately reconstruct a scene with a transparent object as a foreground.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and the technical field of three-dimensional reconstruction, and particularly relates to a transparent object reconstruction method and system based on RGB-D camera consistent depth prediction. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] The three-dimensional reconstruction task refers to reconstructing a corresponding object or scene given the data captured by a camera under a partial view of the object or scene. Existing methods can reconstruct a scene by using an RGB-D camera and a SLAM algorithm.

[0004] Three-dimensional reconstruction is a classic problem in the research of computer vision, robots, AR / VR, etc. The existing three-dimensional reconstruction method based on the classic SLAM algorithm has a long research history. This kind of method is based on RGB-D images, and completes the reconstruction of a scene or object through motion estimation, multi-frame optimization and mapping. However, this kind of method cannot handle defective data, that is, when the depth image obtained by the camera has errors, this kind of algorithm cannot reconstruct the correct result. The depth estimation algorithm based on deep learning has a short research history. This kind of method uses the fitting ability of a neural network to obtain the depth value of the input RGB image through a large amount of data. However, the existing methods only focus on indoor scenes or single images containing transparent objects, and cannot give multi-view consistent depth estimation for continuous data containing transparent objects. Inconsistent depth will cause serious reconstruction errors in RGB-D reconstruction.

[0005] In summary, although the existing reconstruction method can well handle the reconstruction of a conventional indoor scene, it cannot obtain correct results for scenes with transparent objects as the foreground. SUMMARY

[0006] In order to solve the above problems, the present application proposes a transparent object reconstruction method and system based on RGB-D camera consistent depth prediction, which can accurately reconstruct scenes with transparent objects as the foreground.

[0007] According to some embodiments, the present application adopts the following technical solution:

[0008] A transparent object reconstruction method based on RGB-D camera consistent depth prediction, comprising the following steps:

[0009] Obtaining an RGB image, a depth image and camera intrinsic information of an RGB-D camera containing a transparent object;

[0010] obtaining a mask of the transparent object based on the RGB image by using a transparent object segmentation network;

[0011] applying the mask on the depth image to obtain the depth value of the non-transparent object region, and obtaining the point cloud of the image in the three-dimensional space by using the camera intrinsic information;

[0012] obtaining the restored depth image by using a pre-trained consistency depth prediction neural network to perform depth image prediction based on the mask, the RGB image and the point cloud in the three-dimensional space;

[0013] performing three-dimensional reconstruction based on the restored depth image, the RGB image and the camera intrinsic information to obtain the final result.

[0014] As an optional implementation, the depth image is obtained at the same time as the RGB image, representing the distance of the object relative to the camera at the current view angle as a depth image.

[0015] As an optional implementation, the specific process of obtaining the mask of the transparent object based on the RGB image by using the transparent object segmentation network includes training a pre-constructed deep convolutional neural network by using a semantic segmentation dataset and a dataset containing pre-segmented transparent object RGB images and masks to obtain the transparent object segmentation network, and inputting the obtained RGB image frame by frame into the transparent object segmentation network to obtain the mask frame by frame.

[0016] As an optional implementation, the step of predicting the depth image includes:

[0017] projecting the image into the point cloud in the three-dimensional space by using the camera intrinsic and the depth information excluding the transparent object region;

[0018] extracting feature information from adjacent images by using a convolutional neural network with a pyramid structure;

[0019] dividing the maximum and minimum depth values of the scene into a fixed number to form different depth spaces, projecting the feature information extracted from the current image in each space into the coordinate system of the adjacent image to obtain fused features fused with the information of the adjacent frame image, and splicing the fused features and the point cloud features;

[0020] outputting the probability of the real depth value of the current image in each different space by the convolutional neural network, and obtaining the final depth value estimated by the network by stacking the probabilities through an accumulation function.

[0021] Further, the calculation process of the point cloud in the three-dimensional space includes:

[0022]

[0023] wherein represents the horizontal position and the vertical position in the image coordinate system, respectively, represents the point cloud coordinate calculated at the position, represents the inverse of the camera intrinsic matrix, represents the position of the point in the image coordinate system, represents the depth value at the point, the depth value of the transparent object region is set to 0 by the mask, and the depth value of other regions remains unchanged.

[0024] Further, the feature extraction part of the image is represented as a mapping function:

[0025]

[0026] wherein is the feature of the image is the mapping function composed of the convolutional neural network.

[0027] Or, the formula for space division is wherein is the maximum depth and the depth is the number of spaces defined artificially.

[0028] Or, the formula for projecting the current image to the adjacent image is:

[0029]

[0030] wherein , is the projection coordinate system and the original coordinate system, is the depth value of each space, is the transformation matrix of the adjacent image pose under the Lie algebra, is the coordinate of the point cloud under the current depth space, the [X, Y, Z] of the point cloud coordinate is converted into [X, Y, Z, 1], that is, 1 is supplemented in the last position to align the matrix, and the projection of the feature is composed of the bilinear interpolation of the feature map in the projection coordinate system.

[0031] As an optional implementation, the training process of the consistent depth prediction neural network comprises: simulating the process of a real scene RGB-D camera shooting a transparent object by using software, and recording the depth information of each picture;

[0032] predicting a depth image;

[0033] constructing a loss function with the real depth value, and training the depth prediction network by using the mechanism of back propagation of the neural network.

[0034] Further, the loss function is:​

[0035]

[0036] wherein , denote the real value of the depth and the real value of the surface normal, is a set weighting parameter, and denote the loss value of the mask region and the loss value of the non-mask region, respectively;

[0037]

[0038] wherein is a mapping function composed of a convolutional neural network, , distribution represents the and , denote the vector outer product;

[0039]

[0040] is the accumulated depth value, is the probability, is the depth value of each space.

[0041] As an optional implementation, the specific process of three-dimensional reconstruction includes: for each frame of input data, reading the data of its adjacent frames, performing depth image prediction, obtaining the recovery data of the model in the current frame and caching the results for the prediction of the next frame;

[0042] The obtained original depth data is replaced by the depth data recovered by the current frame, and the scene is reconstructed by using the instant positioning and map construction technology.

[0043] A transparent object reconstruction system based on RGB-D camera consistent depth prediction, comprising:

[0044] An acquisition module configured to acquire an RGB image, a depth image and camera intrinsic information of an RGB-D camera containing a transparent object;

[0045] An image mask calculation module configured to obtain a mask of the transparent object based on the RGB image by using a transparent object segmentation network;

[0046] A point cloud calculation module configured to apply the mask to the depth image to obtain the depth value of the non-transparent object region, and obtain the point cloud of the image in the three-dimensional space by using the camera intrinsic information;

[0047] ​The depth image prediction module is configured to perform depth image prediction based on the mask, the RGB image and the three-dimensional space point cloud, using a pre-trained consistency depth prediction neural network, to obtain a recovered depth image.

[0048] The three-dimensional reconstruction module is configured to perform three-dimensional reconstruction based on the recovered depth image, the RGB image and camera intrinsic information, to obtain a final result.

[0049] Compared with the prior art, the present application has the following beneficial effects:

[0050] The present application can perform real-time reconstruction on a scene containing transparent objects, and through the consistency constraint structure in the neural network, the problem of inconsistent predicted depth values between adjacent frames can be avoided, so that a continuous view consistency result is recovered, and the entire scene is reconstructed.

[0051] The present application first obtains the mask of the transparent object, then applies the mask to the original depth image to obtain the depth value of the non-transparent object region, projects the image into a three-dimensional space point cloud using the camera intrinsic parameters and the depth information excluding the transparent object region, extracts feature information from adjacent images respectively to form different depth spaces, projects the feature information extracted from the current image in each space into the coordinate system of the adjacent image to obtain fused features fused with the information of adjacent frame images, and the probability of the real depth value of the current image in each different space, and the network estimates the final depth value by superimposing these probabilities through an accumulation function, and the characteristics of the transparent object are particularly set in the point cloud calculation and depth image prediction processes, so that the method can be applied to the scene containing transparent objects in the captured image.

[0052] To make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0053] The drawings accompanying the specification of the present application form a part of the present application and are used to provide a further understanding of the present application, the illustrative embodiments of the present application and the description thereof serve to explain the present application, and do not constitute an improper limitation of the present application.

[0054] Figure 1 A flowchart of a transparent object reconstruction method based on RGB-D camera consistency depth prediction in an embodiment. DETAILED DESCRIPTION

[0055] The present application will be further described below in combination with the drawings and embodiments.

[0056] It should be noted that the following detailed description is illustrative only, and is intended to provide further description in connection with the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0057] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, as the scope of the exemplary embodiments of this application will be limited only by the appended claims. Unless defined otherwise, all technical and scientific terms and any acronyms used herein have the same meanings as commonly understood by one of ordinary skill in the art in the field of the application. Although any methods and materials similar or equivalent to those described herein can be used in the practice of the present application, the preferred methods and materials are described herein.

[0058] Example One

[0059] A transparent object reconstruction method based on RGB-D camera consistency depth prediction, as shown in Figure 1 includes the following steps:

[0060] Through the "RealSense D435i RGB-D" handheld camera, the picture of the scene containing the transparent object is shot, and at the same time the distance of the object relative to the camera under the current view is obtained through the sensor, which is represented as a depth image.

[0061] Through pre-camera calibration, the internal parameter information of the camera is determined, which is a fixed attribute of the camera and will not change, and can be directly applied in the shooting of different scenes.

[0062] An open-source semantic segmentation dataset is used, and a depth convolutional neural network is trained using a pre-segmented dataset containing transparent object RGB images and mask. The RGB image shot is input into the network frame by frame to obtain the mask frame by frame. This step needs to be pre-processed outside the system algorithm.

[0063] Further, the mask output by the network sometimes cannot correctly contain only the transparent object, but as long as the transparent object is contained in the mask area with a value of 1, the subsequent algorithm can also predict its depth value.

[0064] According to the transparent object reconstruction method based on RGB-D camera consistency depth prediction provided by the application, the step of predicting the depth image comprises:

[0065] The camera internal parameter and the depth information excluding the transparent object region are used to project the image into a point cloud in three-dimensional space.

[0066] The convolutional neural network with a pyramid structure is used to extract feature information from adjacent images.

[0067] The scene is divided into a fixed number of maximum and minimum depth values ​​to form different depth spaces. By projecting the feature information extracted from the current image in each space onto the coordinate system of the adjacent images, a fused feature that integrates the information of adjacent frames is obtained. The fused feature is then stitched together with the feature of the point cloud.

[0068] The fused features are passed through a convolutional neural network to output probabilities, representing the probability of the true depth value of the current image in each different space. These probabilities are then summed using an accumulation function to obtain the network's final estimated depth value.

[0069] Furthermore, the number of depth spaces to be divided is determined manually. More spaces will result in more accurate results, but will also bring greater computational load and affect the real-time performance of reconstruction.

[0070] According to the present invention, a method for reconstructing transparent objects based on consistent depth prediction using an RGB-D camera is provided, wherein the steps for reconstructing the three-dimensional object include:

[0071] For each input frame, data from its adjacent frames is read, the model's reconstructed data for the current frame is obtained, and the results are cached for prediction in the next frame. The original data acquired by the camera is replaced with the depth data reconstructed from the current frame, and the scene is reconstructed using the classic SLAM framework ElasticFusion.

[0072] According to the transparent object reconstruction method based on RGB-D camera consistent depth prediction provided by the present invention, the point cloud computing formula in three-dimensional space is as follows:

[0073]

[0074] in These represent the horizontal and vertical positions in the image coordinate system, respectively. This represents the point cloud coordinates calculated at that location. Represents the inverse of the camera intrinsic parameter matrix. This represents the position of a point in the image coordinate system. This represents the depth value at that point. The depth value of transparent objects is set to 0 through the use of a mask, while other areas remain unchanged.

[0075] The method for reconstructing transparent objects based on consistent depth prediction using an RGB-D camera, according to the present invention, includes the following specific representations of image feature extraction, depth space partitioning, probability calculation, and accumulation formula:

[0076] The extracted features of an image are represented as a mapping function:

[0077]

[0078] in is a feature of the image , is a mapping function composed of a convolutional neural network.

[0079] The formula for spatial division is , is the maximum depth and depth obtained according to the original depth information, is the number of division spaces defined artificially.

[0080] The formula for projecting the current image into the adjacent image is:

[0081]

[0082] wherein , is the projection coordinate system and the original coordinate system, is the depth value of each space, is the transformation matrix of the adjacent image pose under the Lie algebra. Since the matrix needs to be aligned, is the coordinate of the point cloud in the current depth space. Since the matrix needs to be aligned, the [X, Y, Z] of the point cloud coordinate is converted to [X, Y, Z, 1], that is, 1 is supplemented in the last position. The projection of the feature is composed of the bilinear interpolation of the feature map in the projection coordinate system.

[0083] The formula for probability calculation is:

[0084]

[0085] wherein is a mapping function composed of a convolutional neural network, represents a set composed of the point cloud of the above-mentioned current image, is a mapping function from the point cloud set to the point cloud feature. represents splicing the feature with the projected feature.

[0086] Using the depth value of each space and its probability, the accumulated depth value is obtained, and the formula is:

[0087]

[0088] The SLAM reconstruction of the embodiment includes the following steps:

[0089] The point cloud calculated using the adjacent image depth information is calculated by the ICP algorithm to calculate the target difference function, and the matching difference calculated by using the RGB feature point matching method is added at the same time, the convex optimization algorithm is used to minimize the difference to solve the relative operation of the camera, the BA optimization and pose graph optimization method is used to optimize the camera pose of multiple continuous frames, the image of each frame is projected into the three-dimensional space using the camera pose and camera intrinsic parameter, and the final reconstruction result is obtained by fusing multiple frames of images.

[0090] Further, the result obtained by the SLAM reconstruction is not a point cloud map, but a dense reconstruction result using a patch. Each patch contains radius, normal vector and other information, and the center of the patch is derived to obtain the reconstruction result of the point cloud.

[0091] It should be noted that the transparent object reconstruction method based on the consistency depth prediction of the RGB-D camera is based on the depth neural network to recover the depth information of the scene and then perform real-time reconstruction. The running speed of the reconstruction is about 15FPS. The network needs to be trained in advance before reconstruction, that is, the real depth data is used to supervise the training of the model.

[0092] According to the transparent object reconstruction method based on the consistency depth prediction of the RGB-D camera provided by the embodiment, the method for constructing the loss function when training the network is:

[0093] When constructing the loss function, the model needs an additional surface normal constraint branch. The calculation formula is

[0094]

[0095] Among them The mapping function composed of the convolutional neural network is , The distribution represents the under the and , represents the vector outer product.

[0096] The final loss function of the model is:

[0097]

[0098] Among them , represents the real value of the depth and the real value of the surface normal. is a weighting parameter set artificially, and represent the loss value of the mask area and the loss value of the non-mask area respectively.

[0099] To train the model, a dataset was built for testing the reconstruction method based on the RGB-D camera consistency depth prediction of the present application, the dataset including 20 synthetic scenes and 5 real scenes. Since the real depth and model of the displayed scene are not known, all the training processes are based on the synthetic dataset.

[0100] Further, the synthetic dataset is partially based on the open source software Blender, the three-dimensional model using various common three-dimensional models of the prior art, placed in the center of a preset room scene, and the camera is uniformly circled around the room during rendering, the camera always facing the central object of the room, and 300-400 images with a resolution of 640x480 are rendered. Among the 20 rendered scenes, 5 scenes are divided for verification and index comparison, and the remaining 15 scenes are used for training.

[0101] In one embodiment, the method for shooting real data is to use an RGB-D camera for shooting, using common transparent objects such as glass cups and glass vases as regular objects, and placing some flat occluders behind the objects as much as possible to determine the depth boundary of the scene; the camera and the object are kept at a certain distance during shooting to prevent the background depth value obtained by the depth camera from being wrong, and a semicircular circumferential motion is performed around the object.

[0102] To prove the reconstruction method based on the RGB-D camera consistency depth prediction provided in the present embodiment, the most advanced transparent object depth prediction method (only considering a single image) is compared with the present method in depth estimation and three-dimensional reconstruction on the self-created synthetic dataset, wherein the three-dimensional reconstruction comparison uses a common reconstruction algorithm, but replaces the depth prediction algorithm with the corresponding method, and the reconstruction results are compared. These methods include ClearGrasp (estimating surface normal, occlusion boundary, and mask, respectively), LIDF (using an implicit depth function to estimate depth), and TransCG (using an encoder-decoder structure).

[0103] The indexes for comparison of depth prediction include:

[0104] RMSE: root mean square error of predicted depth value and real depth value

[0105] REL: relative error of predicted depth value and real depth value

[0106] MAE: mean absolute error of predicted depth value and real depth value

[0107] Threshold 1.05 / 1.10 / 1.25: percentage number of pixel points with relative error within 1.05, 1.10, and 1.25

[0108] The metrics used for comparison in 3D reconstruction include:

[0109] Chamfer distance: A larger distance indicates a greater difference between the two point cloud sets; a smaller distance indicates a better reconstruction. This metric measures the accuracy of the reconstruction results. However, it has limitations in measuring reconstruction results at different scales.

[0110] Prec, Recall, and F-Score: Precision is calculated by sampling 200k points from both the reconstructed and ground truth models. The distance between these points is compared to a threshold, typically set to 5cm. Matching points are used to find the corresponding points in the predicted and ground truth models. Precision (TP) is defined as both the predicted and ground truth models being less than the threshold; FP is defined as both the predicted and ground truth models being greater than the threshold; FN is defined as both the predicted and ground truth models being less than the threshold; and TN is defined as both the predicted and ground truth models being greater than the threshold. Precision is the ratio of TP to TP+FP, recall is the ratio of TP to TP+FN, and the F-score is calculated by combining precision and recall.

[0111] Table 1. Comparison of depth prediction results between this embodiment and existing advanced methods on synthetic datasets.

[0112]

[0113] Table 1 shows the average results of the depth prediction metrics of the reconstruction method based on RGB-D camera consistency depth prediction provided in this embodiment and other methods on all synthetic datasets. This comparison only considers the depth prediction metrics. It can be seen that our method has better results in most cases compared with the numerical metrics of the true values. However, the depth recovery metrics only consider the error of a single image. When using depth information for reconstruction, the depth consistency of consecutive images is also very important. Therefore, we also compared the errors of 3D reconstruction.

[0114] Table 2 shows the average results of the reconstruction metrics of the RGB-D camera-consistent depth prediction method provided in this invention and other methods reconstructed on the same synthetic dataset. Since the LIDF method does not provide a corresponding depth prediction interface, it is ignored in the comparison of reconstruction results.

[0115] Table 2. Comparison of reconstruction results of this embodiment and existing advanced methods on synthetic datasets.

[0116]

[0117] The smaller the chamfer distance is, the more accurate the result is. The smaller result of the method indicates that the average error between the reconstructed result and the real result is smaller, and the position is closer to the real result. However, the method is sensitive to the scale of the model, and a small scene itself will also obtain a small chamfer distance, so the indicators of point and point matching, Prec, Recall and F-Score, are still needed to evaluate. , The larger the value is, the better it is. From the better result of the method, it can be seen that the overall shape of the model obtained by the method provided in the application is more accurate.

[0118] According to Tables 1 and 2, it can be seen that the effect of the reconstruction method based on consistency depth prediction of an RGB-D camera provided in the embodiment is obviously better than that of other existing technical methods.

[0119] In other embodiments, the values of the above parameters can be modified or adjusted according to the situation.

[0120] Embodiment Two

[0121] A transparent object reconstruction system based on consistency depth prediction of an RGB-D camera, comprising:

[0122] An acquisition module configured to acquire an RGB image, a depth image and camera intrinsic information of an RGB-D camera containing a transparent object;

[0123] An image mask calculation module configured to obtain a mask of the transparent object based on the RGB image by using a transparent object segmentation network;

[0124] A point cloud calculation module configured to apply the mask to the depth image to obtain a depth value of a non-transparent object region, and obtain a point cloud of the image in a three-dimensional space by using the camera intrinsic information;

[0125] A depth image prediction module configured to perform depth image prediction by using a pre-trained consistency depth prediction neural network based on the mask, the RGB image and the three-dimensional space point cloud, to obtain a restored depth image;

[0126] A three-dimensional reconstruction module configured to perform three-dimensional reconstruction based on the restored depth image, the RGB image and the camera intrinsic information, to obtain a final result.

[0127] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, various software modules in accordance with embodiments of the present application are stored in a memory such as a computer program product (e.g., a disk storage) and executed by a computer processor. As such, various computer program products code, when executed, enable the computer to function as a special purpose computer programmed to carry out the steps described herein. The software modules, when executed, enable the computer to provide various example embodiments of the present application as discussed herein. CD - ROM

[0128] The present application is described in reference to the drawings using a flowchart and / or a block diagram of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 Figure 1

[0129] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 Figure 1

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 Figure 1

[0131] The above description is intended to be illustrative and not restrictive. Many other changes and modifications can occur to those skilled in the art without departing from the scope of the application. The scope of the application should be determined by the appended claims along with the full scope of equivalents to which such claims are entitled.​​​​​​​

Claims

1. A method for reconstructing transparent objects based on consistent depth prediction using an RGB-D camera, characterized in that, Includes the following steps: Acquire RGB images, depth images, and camera intrinsic information from an RGB-D camera, including transparent objects; Using a transparent object segmentation network, a mask for the transparent object is obtained based on the RGB image; The mask is applied to the depth image to obtain the depth value of the non-transparent object region, and the point cloud of the image in three-dimensional space is obtained using the camera intrinsic parameter information. Based on the mask, RGB image and 3D spatial point cloud, a pre-trained consistent depth prediction neural network is used to predict the depth image and obtain the recovered depth image. The training process of the consistent depth prediction neural network includes: using software to simulate the process of shooting transparent objects with a real-world RGB-D camera, and recording the depth information of each image; Predict depth images; Construct a loss function for the true depth value and train a depth prediction network using the backpropagation mechanism of a neural network; Based on the recovered depth image, RGB image, and camera intrinsic parameter information, 3D reconstruction is performed to obtain the final result.

2. The transparent object reconstruction method based on RGB-D camera consistent depth prediction as described in claim 1, characterized in that, The depth image is obtained by acquiring the distance of the object relative to the camera from the current viewpoint while acquiring the RGB image, and is represented as a depth image.

3. The transparent object reconstruction method based on RGB-D camera consistent depth prediction as described in claim 1, characterized in that, The specific process of obtaining the mask of a transparent object based on the RGB image using a transparent object segmentation network includes training a pre-constructed deep convolutional neural network using a semantic segmentation dataset and a pre-segmented dataset containing RGB images of transparent objects and masks to obtain a transparent object segmentation network. The acquired RGB images are then input into the transparent object segmentation network frame by frame to obtain the mask frame by frame.

4. The transparent object reconstruction method based on RGB-D camera consistent depth prediction as described in claim 1, characterized in that, The steps for predicting depth images include: By utilizing camera intrinsics and depth information from regions where transparent objects are removed, the image is projected into a point cloud in three-dimensional space. A pyramid-structured convolutional neural network is used to extract feature information from adjacent images separately; The scene is divided into a fixed number of maximum and minimum depth values ​​to form different depth spaces. By projecting the feature information extracted from the current image in each space onto the coordinate system of the adjacent images, a fused feature with information from adjacent frames is obtained. The fused feature is then stitched together with the point cloud feature. The concatenated features are passed through a convolutional neural network to output probabilities, representing the probability of the true depth value of the current image in each different space. The network estimates the final depth value by summing the probabilities through an accumulation function.

5. The transparent object reconstruction method based on RGB-D camera consistent depth prediction as described in claim 4, characterized in that, The calculation process of point clouds in three-dimensional space includes: in These represent the horizontal and vertical positions in the image coordinate system, respectively. This represents the point cloud coordinates calculated at that location. Represents the inverse of the camera intrinsic parameter matrix. This represents the position of a point in the image coordinate system. This represents the depth value at that point. The depth value of transparent objects is set to 0 through the use of a mask, while other areas remain unchanged.

6. The transparent object reconstruction method based on RGB-D camera consistent depth prediction as described in claim 4, characterized in that, The feature extraction part of an image can be represented by a mapping function: in For image Features This is a mapping function constructed from a convolutional neural network; Alternatively, the formula for spatial partitioning is: ,in The maximum depth and depth are obtained based on the original depth information. It is an artificially defined number of partitioned spaces; Alternatively, the formula for projecting the current image onto neighboring images is: in , For the projected coordinate system and the original coordinate system, For the depth value of each space, Let be the transformation matrix of adjacent image poses under Lie algebra. Given the coordinates of the point cloud in the current depth space, the [X, Y, Z] coordinates of the point cloud are transformed into [X, Y, Z, 1], that is, 1 is added to the last position to align the matrix. The projection of the feature is formed by bilinear interpolation of the feature map in the projection coordinate system.

7. The transparent object reconstruction method based on RGB-D camera consistent depth prediction as described in claim 1, characterized in that, The loss function is: in , The true values ​​represent depth and surface normal. These are the weighting parameters that are set. and These represent the loss values ​​for the masked region and the non-masked region, respectively. in Mapping functions composed of convolutional neural networks , Distribution representation of the first In a space and , Represents the cross product of vectors; The accumulated depth value. For probability, The depth value for each space.

8. The transparent object reconstruction method based on RGB-D camera consistent depth prediction as described in claim 1, characterized in that, The specific process of 3D reconstruction includes: for each frame of input data, reading the data of its adjacent frames, predicting the depth image, obtaining the model's recovery data in the current frame and caching the results for prediction in the next frame; The original depth data is replaced with depth data recovered from the current frame, and the scene is reconstructed using real-time localization and mapping techniques.

9. A transparent object reconstruction system based on consistent depth prediction using an RGB-D camera, characterized in that, include: The acquisition module is configured to acquire RGB images, depth images, and camera intrinsic information, including transparent objects, from an RGB-D camera; The image mask calculation module is configured to use a transparent object segmentation network to obtain the mask of the transparent object based on the RGB image; The point cloud computing module is configured to apply a mask to the depth image, obtain the depth value of the non-transparent object region, and use camera intrinsic information to obtain the point cloud of the image in three-dimensional space. The depth image prediction module is configured to perform depth image prediction based on the mask, RGB image and three-dimensional spatial point cloud using a pre-trained consistent depth prediction neural network to obtain the recovered depth image. The training process of the consistent depth prediction neural network includes: using software to simulate the process of shooting transparent objects with a real-world RGB-D camera, and recording the depth information of each image; Predict depth images; Construct a loss function for the true depth value and train a depth prediction network using the backpropagation mechanism of a neural network; The 3D reconstruction module is configured to perform 3D reconstruction based on the recovered depth image, RGB image, and camera intrinsic information to obtain the final result.

Images