Object posture estimation method, method for obtaining dense depth image, and corresponding device

Abstract

Provided are an object posture estimation method, a method for obtaining a dense depth image, and a corresponding device. The object posture estimation method comprises: obtaining a two-dimensional (2D) image corresponding to the object to be estimated; extracting global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated; and performing three-dimensional (3D) posture estimation on the object to be estimated based on the extracted global visual features and / or local geometric features.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an object posture estimation method, a method for obtaining a dense depth image, and corresponding devices. Background Art

[0002] In augmented reality scenarios, dense depth images are needed to determine the location and presentation of virtual content in order to provide users with an augmented reality scene that is closer to reality. Dense means that each pixel in the object image area has a depth estimation value.

[0003] One way to obtain dense depth images is to use specific sensors (such as depth sensors) to obtain dense depth images. The limitation of this method is that it requires special hardware sensors, and the structured light source required to measure depth consumes energy, which reduces the use time of the device. In addition, since structured light cannot illuminate distant objects and is affected by strong light such as the sun, the method based on specific sensors is only applicable to indoor scenes, and the use scenarios are very limited.

[0004] Another existing method for obtaining a dense depth image is to obtain multiple images by moving the camera, and calculate the depth image by fusing the information of the multi-view images. This technology requires the user to move, the operation is cumbersome, and the user experience is reduced.

[0005] Due to the above problems, it is necessary to improve the existing methods to obtain dense depth images more conveniently. Summary of the invention

[0006] In view of the problems of the prior art, the present invention proposes the following solutions:

[0007] The present invention proposes that global visual features and / or local geometric features can be extracted from a single two-dimensional (2D) image, and the three-dimensional (3D) posture of the object can be estimated based on the extracted features, and then a dense depth image can be obtained based on the 3D posture of the object. The above method provided by the present invention does not require the use of a specific sensor (such as a depth sensor), nor does it require the user to move the camera to obtain multiple images. A dense depth image can be obtained using a single 2D image that is easily obtained, which greatly improves the convenience of obtaining a dense depth image.

[0008] The present invention also proposes that a deep learning network can be used to extract global visual features and / or local geometric features. When training a deep learning network, not only 2D images can be used for training, but also depth information or 3D voxel information corresponding to the 2D images can be used for training at the same time. The trained deep learning network uses 2D images and depth information (or 3D voxel information) as guidance information, so the features extracted have higher resolution and higher robustness.

[0009] By using a feature approximation strategy during the training process, the trained network model can estimate the 3D pose information of an object using only a single 2D image, but can achieve performance similar to that of using both 2D images and depth information (or 3D voxel information), thereby improving the accuracy of object pose estimation.

[0010] The present invention proposes to extract global visual features of an image when estimating the 3D posture of an object, estimate the 3D posture of the object based on the global visual features, and further obtain a dense depth image.

[0011] The present invention proposes to extract local geometric features of an image when estimating the 3D posture of an object, estimate the 3D posture of the object based on the local geometric features, and further obtain a dense depth image.

[0012] The present invention proposes that when estimating the 3D posture of an object, global visual features and local geometric features can also be extracted simultaneously, and the 3D posture of the object can be estimated by combining these two features, and a dense depth image can be further obtained.

[0013] This method can be used to obtain a dense depth image from a single image. The obtained dense depth image can be used to display highly realistic virtual images in augmented reality application scenarios, and can also be used for other applications that require depth images, such as autonomous driving.

[0014] Specifically, according to one aspect of the present invention, a method for object posture estimation is provided, comprising: acquiring a 2D image corresponding to the object to be estimated; extracting global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated; and performing 3D posture estimation on the object to be estimated based on the extracted global visual features and / or local geometric features.

[0015] The local geometric features are features that characterize local key components or key points of an object.

[0016] Among them, extracting the global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated includes: extracting the global visual features of the 2D image corresponding to the object to be estimated through a first deep learning network; and / or extracting the local geometric features of the 2D image corresponding to the object to be estimated through a second deep learning network.

[0017] Among them, the first deep learning network is trained in the following manner: a third deep learning network is trained based on an input 2D image and depth information corresponding to the 2D image; and the first deep learning network is trained based on the input 2D image and the third deep learning network through a first feature approximation strategy.

[0018] Among them, the first deep learning network is trained through the first feature approximation strategy, including: setting a first auxiliary loss term in the loss function, the first auxiliary loss term is a difference function of the characteristics of the first deep learning network and the characteristics of the third deep learning network; through the first auxiliary loss term, the first deep learning network is trained so that the characteristics of the first deep learning network approach the characteristics of the third deep learning network.

[0019] Among them, the second deep learning network is trained in the following manner: a fourth deep learning network is trained based on the input 2D image and the 3D voxel information corresponding to the 2D image; and a second deep learning network is trained based on the input 2D image and the fourth deep learning network through a second feature approximation strategy.

[0020] Among them, the second deep learning network is trained through the second feature approximation strategy, including: setting a second auxiliary loss term in the loss function, the second auxiliary loss term is a difference function between the characteristics of the second deep learning network and the characteristics of the fourth deep learning network; through the second auxiliary loss term, the second deep learning network is trained so that the characteristics of the second deep learning network approach the characteristics of the fourth deep learning network.

[0021] The step of obtaining a 2D image corresponding to the object to be estimated includes: performing object region segmentation on the 2D image to be processed to obtain a 2D image corresponding to the object to be estimated.

[0022] According to another aspect of the present invention, a method for obtaining a dense depth image by the above-mentioned object posture estimation method is provided, comprising: determining a dense depth image corresponding to the object to be estimated in a 2D image according to the 3D posture of the object to be estimated in the 2D image; and determining a dense depth image corresponding to the 2D image to be processed according to the dense depth images corresponding to each object to be estimated in the 2D image to be processed.

[0023] According to another aspect of the present invention, there is provided an object posture estimation device, comprising: an image acquisition module, used to acquire a two-dimensional 2D image corresponding to the object to be estimated; a feature extraction module, used to extract global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated; and a posture estimation module, used to perform three-dimensional 3D posture estimation on the object to be estimated based on the extracted global visual features and / or local geometric features.

[0024] The local geometric features are features that characterize local key components or key points of an object.

[0025] Among them, the feature extraction module includes: a component for extracting global visual features of a 2D image corresponding to an object to be estimated through a first deep learning network; and / or a component for extracting local geometric features of a 2D image corresponding to an object to be estimated through a second deep learning network.

[0026] Among them, the first deep learning network is trained by the following components: components of the third deep learning network are trained according to the input 2D image and the depth information corresponding to the 2D image; components of the first deep learning network are trained according to the input 2D image and the third deep learning network through the first feature approximation strategy.

[0027] Among them, the components of the first deep learning network obtained by training the first feature approximation strategy include: a component that sets a first auxiliary loss term in the loss function, wherein the first auxiliary loss term is a difference function between the characteristics of the first deep learning network and the characteristics of the third deep learning network; and a component that trains the first deep learning network through the first auxiliary loss term so that the characteristics of the first deep learning network approximate the characteristics of the third deep learning network.

[0028] Among them, the second deep learning network is trained by the following components: components of the fourth deep learning network are trained according to the input 2D image and the 3D voxel information corresponding to the 2D image; components of the second deep learning network are trained according to the input 2D image and the fourth deep learning network through the second feature approximation strategy.

[0029] Among them, the component of the second deep learning network obtained by training the second feature approximation strategy includes: a component that sets a second auxiliary loss term in the loss function, wherein the second auxiliary loss term is a difference function between the characteristics of the second deep learning network and the characteristics of the fourth deep learning network; and a component that trains the second deep learning network through the second auxiliary loss term so that the characteristics of the second deep learning network approximate the characteristics of the fourth deep learning network.

[0030] The image acquisition module includes: a component for segmenting the object region of the 2D image to be processed to obtain a 2D image corresponding to the object to be estimated.

[0031] According to another aspect of the present invention, there is provided a device for obtaining a dense depth image by means of the above-mentioned object posture estimation device, comprising: a sub-image determination module, for determining a dense depth image corresponding to the object to be estimated in a 2D image according to the 3D posture of the object to be estimated in the 2D image; and a total image determination module, for determining a dense depth image corresponding to the 2D image to be processed according to the dense depth images corresponding to each object to be estimated in the 2D image to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or other aspects will become apparent and more readily understood through the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 It is a flowchart of obtaining the 3D posture of an object by taking the first deep learning network as GAppNet and the second deep learning network as LGeoNet as an example according to the present invention;

[0034] Figure 2 is a simplified flow chart of object posture estimation according to the present invention;

[0035] Figure 3 A training process of global visual features based on deep learning according to the present invention, wherein a network model for extracting global visual features of an image is obtained;

[0036] Figure 4 A training process of local geometric features based on deep learning according to the present invention, wherein a network model for extracting local geometric features of an image is obtained;

[0037] Figure 5 This is the process of training GAppNet based on deep learning according to the present invention (taking the VGG16 model as an example);

[0038] Figure 6 is a flowchart of training LGeoNet based on deep learning according to the present invention (taking the VGG16 model as an example);

[0039] Figure 7 It is a flowchart of performing posture estimation by extracting only global visual features based on deep learning according to the present invention (taking the VGG16 model as an example);

[0040] Figure 8 It is a flowchart of performing posture estimation by extracting only local geometric features based on deep learning according to the present invention (taking the VGG16 model as an example);

[0041] Fig. 9 It is a flowchart of performing posture estimation by combining global visual features based on deep learning and local geometric features based on deep learning according to the present invention (taking the VGG16 model as an example);

[0042] Fig.10 is a flow chart of generating a depth image based on a 3D posture of an object according to the present invention;

[0043] Fig.11 is a block diagram of an object posture estimation apparatus according to the present invention; and

[0044] Fig.12is a block diagram of an apparatus for obtaining a dense depth image according to the present invention. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present invention will now be described in detail, examples of which are shown in the accompanying drawings, wherein like reference numerals refer to like parts. The embodiments will be described below by referring to the drawings in order to explain the present invention.

[0046] With respect to the above two methods of obtaining dense depth images described in the background technology part, the present invention proposes that it is possible to estimate the 3D posture of the object from the single image without relying on specific sensors or multiple images, and then further obtain a dense depth image based on the 3D posture of the object by using only a single 2D image as input.

[0047] The 3D pose estimation of an object is to calculate the angle information of the object's six degrees of freedom relative to the camera or person, which can be decomposed into a translation matrix T and a rotation matrix R. The six degrees of freedom are azimuth a, elevation e, plane rotation angle θ, depth d, and the object's principal point (u, v). The azimuth a, elevation e, and plane rotation angle θ constitute the rotation matrix R, and the azimuth a, elevation e, and depth information d constitute the translation matrix T. The translation matrix and the rotation matrix constitute the external parameters of the camera, and the object's principal point (u, v), focal length f, and window size α constitute the internal parameters of the camera.

[0048] There are several difficulties in estimating the 3D pose of an object from a single 2D image. For example, the images of the same object may be very different visually (the same chair seen from the front and back are completely different). In addition, the object may be blocked by other objects in the image, or the image may only contain a part of the object (the object is truncated by the image). If the above problems can be solved, a higher-precision 3D pose of the object can be obtained, which can better meet the needs of augmented reality scenarios.

[0049] Due to the above problems, it is necessary to improve the existing methods to obtain more accurate object 3D pose estimation and dense depth images.

[0050] Hereinafter, the improved method and apparatus of the present invention will be described in detail with reference to the accompanying drawings.

[0051] According to the method for obtaining the 3D posture of an object by extracting image features of the present invention, the posture of the 3D object can be obtained by extracting only global visual features, only local geometric features, or both global visual features and local geometric features.

[0052] The so-called global visual features are features extracted from the entire image (color image or grayscale image), which can represent the overall characteristics of the object. The so-called local geometric features refer to features that can characterize the local key components or key points of the object. Taking a car as an example, it includes key components such as the left front wheel, right front wheel, left rear wheel, right rear wheel, headlights, doors, etc. Local geometric features can distinguish the differences of objects when they are blocked or truncated.

[0053] Since the different postures of objects in actual images are usually very different, and objects may be occluded or truncated in the image, the global visual features cannot reflect the local features of the object. Therefore, when the object is occluded or truncated, the local geometric features can better reflect the posture of the object. In this case, the local geometric features are more robust and can effectively improve the accuracy of object posture estimation.

[0054] If global visual features and local geometric features are combined for pose estimation, it can reflect not only the overall characteristics of the object, but also the local characteristics of the object. Therefore, they can be combined into more robust and discriminative features, making the 3D pose estimation of the object more accurate.

[0055] The present invention provides an object posture estimation method, comprising the following steps:

[0056] In step 00, a 2D image corresponding to the object to be estimated is obtained;

[0057] In step 02, extracting global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated;

[0058] In step 04, 3D pose estimation is performed on the object to be estimated based on the extracted global visual features and / or local geometric features.

[0059] Wherein, step 00 further includes: segmenting the 2D image to be processed into object regions to obtain a 2D image corresponding to the object to be estimated.

[0060] Wherein, step 02 further comprises:

[0061] Extracting global visual features of the 2D image corresponding to the object to be estimated through a first deep learning network; and / or

[0062] The local geometric features of the 2D image corresponding to the object to be estimated are extracted through the second deep learning network.

[0063] In step 04, if the features extracted in step 02 are global visual features and local geometric features, the extracted global visual features and local geometric features can be combined to obtain combined image features, and 3D pose estimation is performed on the object to be estimated based on the combined image features. The combined image features can be input into a subsequent network to obtain the 3D pose of the object.

[0064] When training the first deep learning network, the 2D image and the depth information can be used as guidance information for training. Specifically, the third deep learning network can be first trained based on the input 2D image and the depth information corresponding to the 2D image, and then the first deep learning network can be trained based on the input 2D image and the third deep learning network through the first feature approximation strategy. Among them, a first auxiliary loss term can be added to the loss function, and the first auxiliary loss term is a difference function between the features of the first deep learning network and the features of the third deep learning network. By adding the first auxiliary loss term, the first deep learning network is trained so that the features of the first deep learning network are close to the features of the third deep learning network.

[0065] When training the second deep learning network, the 2D image and the 3D voxel information can be used as guidance information for training. Specifically, the fourth deep learning network is first trained based on the input 2D image and the 3D voxel information corresponding to the 2D image, and the second deep learning network is trained based on the input 2D image and the fourth deep learning network through the second feature approximation strategy. Among them, a second auxiliary loss term can be added to the loss function, and the second auxiliary loss term is a difference function between the features of the second deep learning network and the features of the fourth deep learning network. By adding the second auxiliary loss term, the second deep learning network is trained so that the features of the second deep learning network are close to the features of the fourth deep learning network.

[0066] The first deep learning network and the second deep learning network trained in the above manner can, during actual online testing, use only a single 2D image to extract image features and estimate the 3D posture information of the object, but can achieve performance similar to that of using 2D images and depth information (or 3D voxel information) at the same time, thereby improving the accuracy of object posture estimation.

[0067] Figure 1 This is a flowchart of obtaining the 3D posture of an object, taking the first deep learning network as GAppNet and the second deep learning network as LGeoNet as an example. Figure 1 A method for obtaining the 3D posture of an object by combining global visual features and local geometric features is shown in FIG.

[0068] like Figure 1As shown, the specific steps of the method for extracting image features to obtain the posture of a 3D object according to the present invention are as follows:

[0069] In step 10, a 2D image is input, and the global visual features of the image are obtained through the deep learning network GAppNet. The 2D image can be a color image or a grayscale image.

[0070] In step 12, a 2D image is input and the local geometric features of the image are obtained through the deep learning network LGeoNet.

[0071] In step 14, the local geometric features after convolution of the LGeoNet network and the global visual features after convolution of the GAppNet network are combined, that is, feature combination is performed to obtain the final image feature f as the input of the subsequent network.

[0072] In step 16, the obtained image feature f is used as the input of the subsequent network, that is, it is processed by one or more fully connected layers as the subsequent network.

[0073] Finally, in step 18, the final 3D pose of the object is output according to the image feature f.

[0074] Generally, the deep learning network structure for estimating the posture of an object includes at least an input layer, a hidden layer, and an output layer, wherein the last layer of the hidden layer is a fully connected layer, which serves as a feature extraction layer. Figure 2 As shown, Figure 2 is a simplified flow chart of object posture estimation according to the present invention.

[0075] exist Figure 2 In step 20, color (or grayscale) image data, such as an RGB image, is input, corresponding to the input layer of the deep learning network structure;

[0076] In step 22, extract the image feature f, corresponding to the hidden layer of the deep learning network structure;

[0077] In step 24, the 3D posture information of the object is estimated based on the image feature f, corresponding to the output layer of the deep learning network structure.

[0078] Deep learning is a perceptron containing multiple hidden layers. It uses low-level features to form high-level features, and uses more abstract high-level features to represent categories, thereby ultimately improving the accuracy of classification or prediction. Convolutional neural networks have been successfully applied to handwriting recognition, face detection and other related research due to their local subsampling, global training, and shared weights. They also show good anti-interference ability and are highly robust to light changes and foreground occlusions. To this end, the present invention also uses deep learning methods to learn and extract features. Figure 3is a training process of a deep learning network (the first deep learning network) for extracting global visual features according to the present invention; Figure 4 is the training process of the deep learning network (the second deep learning network) for extracting local geometric features according to the present invention. Figure 3 The network model obtained by training is used to extract the global visual features of the image. Figure 4 The network model obtained by training is used to extract local geometric features of the image. Figure 3 , 4 The functional steps of obtaining the network model for extracting global visual features and local geometric features are shown respectively.

[0079] like Figure 3 As shown, during training, the deep learning network CDNet (corresponding to the third deep learning network mentioned above) is first trained, and the input is image data, which includes both color (or grayscale) channels (such as RGB images) and depth channels (such as depth images). Among them, the image data of the object and the corresponding depth data must have consistent posture information. In step 30-1, the image data is operated through several convolutional layers, activation function layers (not shown), and pooling layers (not shown), and then in step 32-1, it is connected to several fully connected layers to obtain the feature f cd Finally, in step 34-1, the 3D posture information of the object is output to obtain the network model Model cd .

[0080] Then, the deep learning network network GAppNet (corresponding to the first deep learning network mentioned above) is trained. The input image data only includes the color (or grayscale) channel (i.e., 2D image). The image data is also operated through several convolution layers, activation function layers (not shown), and pooling layers (not shown) in step 30-2 to obtain the image features after convolution transformation. Then, the image features after convolution are connected to several fully connected layers in step 32-2 to obtain the features f c Finally, in step 34-2, the 3D posture information of the object is output to obtain the network model Model GApp .

[0081] In the training process of the deep learning network GAppNet, a feature approximation strategy is adopted. The so-called feature approximation strategy refers to making the features output by one network as consistent as possible with the features output by another network. GApp The features obtained are as close as possible to the ones obtained through the network model cd Get consistent features.

[0082] Specifically, when training the deep network GAppNet, the trained network model Model cd Supervise the training process of the GAppNet network, that is, add an auxiliary loss term to the loss function: loss cd =f c -f cd Therefore, through the network model Model GApp The features obtained will continue to approach the network model cd The features obtained. Due to the network model Model cd Since both color and depth data are used, we add an auxiliary loss term loss cd After that, the network model Model GApp Implicitly learn deep information to increase the discriminability and robustness of learned features.

[0083] pass Figure 4 Train a deep learning network to obtain local geometric features of images, and Figure 3 The training method is similar to that of , and also adopts the feature approximation strategy.

[0084] like Figure 4 As shown, firstly, the deep learning network VolNet (corresponding to the fourth deep learning network mentioned above) is trained, and in step 40-1, 3D Voxel data generated from the 3D model database is input, and the 3D voxel data is a kind of 3D depth data, and then in step 42-1, several convolution layers, activation function layers (not shown), and pooling layers (not shown) are operated to obtain the feature f vol Then, in step 44-1, several deconvolution layers are connected, and finally, in step 46-1, the key component information of the object is output to obtain the network model Model vol .

[0085] Then, the deep learning network LGeoNet is trained. In step 40-2, a color or grayscale image (such as an RGB image) is input from the 3D model database. Similarly, in step 42-2, a plurality of convolutional layers, activation function layers (not shown), and pooling layers (not shown) are operated to obtain the feature f LGeo Then, in step 44-2, several deconvolution layers are connected, and finally, in step 46-2, the key component information of the object is output to obtain the network model Model LGeo .

[0086] In the training process of LGeoNet network, the feature approximation strategy should also be adopted, that is, the trained network model should be referred to. vol , so that the network model Model LGeo The features obtained are as close as possible to the network model Modelvol Get consistent features.

[0087] Specifically, during network training, the trained network model Model is used vol Supervise the training process of the LGeoNet network, that is, add an auxiliary loss term to the loss function: loss LGeo =f LGeo -f vol Therefore, the network model Model LGeo The obtained features will continue to approach the network model Model vol The features obtained. Due to the network model Model vol 3D voxel data is used as 3D depth data, so an auxiliary loss term loss is added LGeo After that, the network model Model LGeo Implicitly learn depth information and increase the expressiveness of local geometric features.

[0088] Through the trained network model Model GApp and network model LGeo , input a single image of the object, extract the global visual features and / or local geometric features of the image, and estimate the 3D posture of the object through the global visual features and / or local geometric features.

[0089] Embodiments of the present invention can classify or regress object postures based on convolutional neural networks. The input information includes color channels, depth channels, and 3D voxel data, and the feature approximation strategy is used to enhance the feature expression capability of color images. The network structure can adopt a variety of models. The present invention takes the VGG16 model (a very classic deep learning network structure) as an example for illustration. The VGG16 model includes 16 hidden layers, of which the first 13 are convolutional layers, each of which is followed by an activation function layer (not shown) and a pooling layer (not shown), and the last three are fully connected layers, wherein the first two fully connected layers are followed by an activation function layer (not shown) and a Dropout layer (not shown), respectively, and the last fully connected layer is the output layer, which outputs the estimated 3D posture of the object.

[0090] Figure 5 This is the process of training GAppNet based on deep learning according to the present invention (taking the VGG16 model as an example); Figure 6 This is a flowchart of training LGeoNet based on deep learning according to the present invention (taking the VGG16 model as an example). Figure 5 The method uses color information and depth information to train a deep learning network model to estimate the 3D pose of an object. Figure 6The method uses color information and 3D Voxel to train a deep learning network model to segment the key components of the object and estimate the 3D posture of the object. Figure 5 The trained network model obtains the global visual features of the image. Figure 6 The trained network model obtains the local geometric features of the image. Figure 5 , 6 The functional steps of training the network model to acquire global visual features and local geometric features are shown respectively.

[0091] like Figure 5 As shown, first train CDNet. The specific steps are as follows:

[0092] In step 50-1, the color and depth data of the image are subjected to 13 layers of convolution operations to obtain the image convolution feature f cd_5_3 . Among them, the input data size is [64,4,224,224], where 64 is the batch size, 4 is the number of channels, including data of four channels: red, green, blue, and depth, and 224 is the height and width of the input image. The convolution kernel size is 3×3, the step size is set to 1, the edge expansion value is 1, the convolution kernel is convolved with the input data, and the number of output channels is the number of neurons of the current convolution kernel. Due to edge expansion, the convolution operation does not change the size of the feature map. The feature map after the convolution operation is processed by the nonlinear activation function relu function. The calculation formula of the relu function is max(0,x). The use of the relu function can reduce the amount of error gradient calculation during back propagation, and at the same time increase the sparse rows of the network, reduce the interdependence of parameters, and alleviate the occurrence of overfitting problems. The Relu layer neither changes the size of the feature map nor the number of feature maps. The pooling layer is the downsampling layer. The pooling window size is set to 2×2, the pooling step is 2, and the image is subjected to non-overlapping maximum pooling. The pooling operation only changes the size of the feature map, not the number of feature maps. After each pooling operation, the feature map becomes a quarter of the original size. Pooling operations are performed after 2, 4, 7, 10, and 13 convolution operations, respectively. After 5 pooling operations, the width and height of the feature map become 1 / 32 of the input image.

[0093] In step 52-1, the feature f after the convolution operation cd_5_3 Then two layers of fully connected layers are operated. The feature maps after the two layers of fully connected layers are processed by the nonlinear activation function relu function and then dropout operation is performed. The probability of discarding data is set to 0.5. The main purpose of setting the dropout layer is to prevent the occurrence of overfitting. Finally, the image feature f is obtained. cd_7 , image feature f cd_7 correspond Figure 3 f cd .

[0094] In step 54-1, the image feature f cd_7 After another fully connected layer operation, the final 3D posture information is output.

[0095] like Figure 5 As shown, then train GAppNet. The specific steps are as follows:

[0096] In step 50-2, a 13-layer convolution operation is performed on the color image to obtain the image feature f after the convolution operation. c_5_3 . Among them, the input data size is [64,3,224,224], where 64 is the batch size, 3 is the number of channels, including data for the three channels of red, green, and blue, and 224 is the height and width of the input image. The convolution kernel size is 3×3, the step size is set to 1, the edge expansion value is 1, the convolution kernel is convolved with the input data, and the number of output channels is the number of neurons of the current convolution kernel. The feature map after the convolution operation is processed by the nonlinear activation function relu function. After 2, 4, 7, 10, and 13 layers of convolution operations, the feature map is downsampled, the pooling window size is set to 2×2, the pooling step size is 2, and the image is subjected to non-overlapping maximum pooling.

[0097] In step 52-2, the feature f obtained by convolution in step 50-2 is c_5_3 Two layers of fully connected layers are operated. The feature map after the fully connected layer operation is processed by the nonlinear activation function relu function and then the dropout operation is performed. The probability of discarding data is set to 0.5. Finally, the image feature f is obtained. c_7 ,correspond Figure 3 The image features f in c .

[0098] In step 54-2, the image feature f c_7 After another fully connected layer operation, the final 3D posture information is output.

[0099] In the process of GAppNet training, the feature approximation strategy is adopted, that is, the image feature f obtained by the network structure GAppNet c_7 The image feature f obtained by the network structure CDNet cd_7 To this end, add an auxiliary loss term to the loss function, loss cd =f c_7 -f cd_7 , so that when the network is trained, the image feature f c_7 Continuously approaching f cd_7 In this step of training, the network model Model cdThe parameters of are unchanged and are only used to assist the training of the GAppNet network.

[0100] like Figure 6 As shown, first train VolNet. The specific steps are as follows:

[0101] In step 60 - 1 , 3D Voxel data is generated from a 3D model database.

[0102] In step 62-1, a 13-layer convolution operation is performed on the generated 3D Voxel data to obtain the feature f vol . The input data size is [64,n,224,224], where 64 is the batch size, n is the number of channels of the 3D voxel data, and 224 is the height and width of the input data. The convolution kernel size is 3×3, the step size is set to 1, the edge expansion value is 1, the convolution kernel is convolved with the input data, and the number of output channels is the number of neurons of the current convolution kernel. The feature map after the convolution operation is processed by the nonlinear activation function relu function. After 2, 4, 7, 10, and 13 layers of convolution operations, the feature map is downsampled, the pooling window size is set to 2×2, the pooling step size is 2, and the image is subjected to non-overlapping maximum pooling.

[0103] In step 64-1, for feature f vol Perform several deconvolution operations to output the key component information of the object. Deconvolution is also called transposed convolution, and image upsampling can be achieved by using deconvolution. After the deconvolution operation, the key component information of the object with the same width and height as the input image is output in step 66-1.

[0104] like Figure 6 As shown, then train LGeoNet. The specific steps are as follows:

[0105] In step 60 - 2 , color image data is generated from the 3D model database, and the generated color image and the corresponding 3D Voxel data must have consistent posture information.

[0106] In step 62-2, a 13-layer convolution operation is performed on the generated color image to obtain the local geometric features f of the image. LGeo. Among them, the input data size is [64,3,224,224], where 64 is the batch size, 3 is the number of channels, including data for the three channels of red, green, and blue, and 224 is the height and width of the input image. The convolution kernel size is 3×3, the step size is set to 1, the edge expansion value is 1, the convolution kernel is convolved with the input data, and the number of output channels is the number of neurons of the current convolution kernel. The feature map after the convolution operation is processed by the nonlinear activation function relu function. After 2, 4, 7, 10, and 13 layers of convolution operations, the feature map is downsampled, the pooling window size is set to 2×2, the pooling step size is 2, and the image is subjected to non-overlapping maximum pooling.

[0107] In step 64-2, for feature f LGeo Several deconvolution operations are performed, and key component information of the object is output in step 66-2.

[0108] In the process of LGeoNet training, the feature approximation strategy is adopted, that is, the image feature f obtained by the network structure LGeoNet LGeo The image feature f obtained by the network structure VolNet vol To this end, add an auxiliary loss term to the loss function, loss LGeo =f LGeo -f vol , so that when the network is trained, the image feature f LGeo Continuously approaching f vol In this step of training, the network model Model vol The parameters of are unchanged and are only used to assist the training of the LGeoNet network.

[0109] In practical applications, for the overall training of the network, we can use only the deep learning-based GAppNet network to extract the global visual features of the image and perform 3D pose estimation of the object, or we can use only the deep learning-based LGeoNet network to extract the local geometric features of the image and perform 3D pose estimation of the object, or we can combine the two to perform 3D pose estimation of the object.

[0110] When only using the deep learning-based GAppNet network for object pose estimation, no additional training is required because the output layer is the 3D pose of the object. Figure 7 As shown, Figure 7 This is a flowchart of posture estimation based only on global visual features extracted by deep learning according to the present invention (taking the VGG16 model as an example), and the specific steps are as follows:

[0111] In step 70, a color image is input, and a global visual feature f is obtained according to the trained deep learning-based network model GAppNet. c_7 , f c_7 As the input of the subsequent network, it passes through a fully connected layer and outputs the 3D posture of the object.

[0112] When only the LGeoNet network based on deep learning is used for object pose estimation, since the output layer of the network structure is the key components or key points of the object, further training is required to achieve the purpose of estimating the object pose. Figure 8 As shown, Figure 8 This is a flowchart of posture estimation based only on local geometric features extracted by deep learning according to the present invention (taking the VGG16 model as an example), and the specific steps are as follows:

[0113] In step 80, a color image is input, and a deep learning-based network model LGeoNet is trained through a 13-layer convolution operation, and a local geometric feature f is obtained through the trained deep learning-based network model LGeoNet. LGeo .

[0114] In step 82, the local geometric feature f LGeo As the input of the subsequent network, it passes through three fully connected layers and finally outputs the 3D posture of the object in step 84.

[0115] During this training process, the learning rate of the network structure for extracting local geometric features is 0, that is, the network parameters remain unchanged, and it is only used to assist the training of the LGeoNet network.

[0116] When combining the global visual features and local geometric features of images extracted based on deep learning to estimate the pose of an object, further training is required to estimate the 3D pose of the object because the two features need to be synthesized into a new feature. Fig. 9 As shown, Fig. 9 This is a flowchart of combining global visual features based on deep learning and local geometric features based on deep learning for posture estimation according to the present invention (VGG16 is taken as an example). The specific steps are as follows:

[0117] Input a color image, then, in steps 90-1 and 90-2, respectively, train the deep learning-based network models GAppNet and LGeoNet through 13 layers of convolution operations, and obtain the global visual features f after the convolution operation based on the trained deep learning-based network models GAppNet and LGeoNet. c_5_3 and local geometric features f LGeo .

[0118] In step 92, the global visual feature f after the convolution operation isc_5_3 and local geometric features f LGeo The features of the corresponding positions are added together to obtain the total feature f.

[0119] In step 94 , feature f is used as input to the subsequent network, i.e., passes through three fully connected layers, and finally outputs the 3D pose of the object in step 96 .

[0120] During this training process, the learning rate of the convolutional layer of the network structure that extracts global visual features and local geometric features is 0, that is, the network parameters remain unchanged, and only the network parameters of the following three fully connected layers are changed.

[0121] In practical applications, for online network testing, only global visual features, only local geometric features, or a combination of both features can be used for the final object 3D pose estimation.

[0122] The process of pose estimation using only global visual features is as follows Figure 7 As shown, the above steps are refined as follows:

[0123] In step 70, a color image is input and the image feature f is obtained through GAppNet. c_7 The input data size is [1,3,224,224], where 1 is the batch size, 3 is the number of channels, including data for the red, green, and blue channels, and 224 is the height and width of the input image, that is, one image is tested each time.

[0124] In step 72, according to the feature f c_7 , after a fully connected layer, the 3D posture information of the object is output.

[0125] The process of posture estimation using only local geometric features is as follows Figure 8 As shown, the above steps are refined as follows:

[0126] In step 80, a color image is input and the image feature f is obtained through LGeoNet. LGeo The input data size is [1,3,224,224], where 1 is the batch size, 3 is the number of channels, including data for the red, green, and blue channels, and 224 is the height and width of the input image, that is, one image is tested each time.

[0127] In steps 82 and 84, the local geometric feature f LGeo As the input of the subsequent network, the 3D posture information of the object is output through three fully connected layers.

[0128] The process of combining global visual features and local geometric features for pose estimation is as follows Fig. 9 As shown, the above steps are refined as follows:

[0129] In steps 90-1 and 90-2, a color image is input and the image features f are obtained by LGeoNet and GAppNet respectively. LGeo and f c_5_3 The input data size is [1,3,224,224], where 1 is the batch size, 3 is the number of channels, including data for the red, green, and blue channels, and 224 is the height and width of the input image, that is, one image is tested each time.

[0130] In step 92, the global visual feature f after the convolution operation is c_5_3 and local geometric features f LGeo The features of the corresponding positions are added together to obtain the total feature f.

[0131] In steps 94 and 96, the feature f is used as the input of the subsequent network, and the 3D posture information of the object is finally output.

[0132] The depth image can be calculated by the object 3D posture information estimated by the present invention. Specifically, the dense depth image corresponding to the object to be estimated can be determined according to the 3D posture of the object to be estimated in the 2D image; and the dense depth image corresponding to the 2D image to be processed can be determined according to the dense depth images corresponding to each object to be estimated in the 2D image to be processed.

[0133] Fig.10 It is a flowchart of generating a depth image based on the 3D posture of an object according to the present invention, and the specific steps are as follows:

[0134] For the acquired t-th frame 2D image:

[0135] In step 1000, image segmentation is performed in the tth frame to segment the image into different object regions. That is, the 2D image to be processed is segmented into object regions to form a 2D image corresponding to the object to be estimated.

[0136] In step 1002, 3D pose estimation is performed on each segmented object.

[0137] In step 1004 , for each object, a dense depth image is generated according to the corresponding 3D model and estimated pose information.

[0138] In step 1006, the depth maps of all objects in the image are collected to generate a dense depth value of the current image.

[0139] Frame t+dt1:

[0140] In step 1010, motion parameters of the six degrees of freedom of the camera are obtained.

[0141] In step 1012 , it is detected whether the camera is moving. If not, the image depth map remains unchanged and the process jumps to step 1019 . If the camera is moving, step 1014 is executed.

[0142] At step 1014, the distance of the object is updated based on the triangulation of the two views.

[0143] In step 1016, the motion parameters of the camera with six degrees of freedom from the tth frame to the t+dt1th frame are obtained.

[0144] In step 1018, the object posture information of the current frame is calculated based on the object posture information of the tth frame and the motion parameters of the camera.

[0145] In step 1019, based on the object posture of the current frame, a dense depth value of the current t+dt1th frame is obtained, and a depth map of the current t+dt1th frame is generated.

[0146] Frame t+dt2:

[0147] Execute steps 1020, 1022, 1026, 1028, and 1029, corresponding to steps 1010, 1012, 1016, 1018, and 1019.

[0148] This can yield a depth image.

[0149] The present invention can be used in the fields of augmented reality, autonomous driving, home intelligent robots, etc. According to the estimated 3D posture information, the corresponding 3D model is back-projected onto the image to obtain a dense depth image.

[0150] Fig.11 is a block diagram of an object posture estimation device according to the present invention. Fig.11 As shown, the object posture estimation device according to the present invention includes an image acquisition module 1101 , a feature extraction module 1103 , and a posture estimation module 1105 .

[0151] In the image acquisition module 1101, a two-dimensional (2D) image corresponding to the object to be estimated is acquired; in the feature extraction module 1103, global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated are extracted; in the posture estimation module 1105, 3D posture estimation is performed on the object to be estimated based on the extracted global visual features and / or local geometric features.

[0152] Fig.12 is a block diagram of an apparatus for obtaining a dense depth image according to the present invention. Fig.12 As shown, the apparatus for obtaining a dense depth image according to the present invention includes a sub-image determination module 1201 and a total image determination module 1203 .

[0153] The image segment determination module 1201 determines the image segment according to the Fig.11 The 3D posture of the object to be estimated in the 2D image obtained by the object posture estimation device is used to determine the dense depth image corresponding to the object to be estimated; the total image determination module 1203 determines the dense depth image corresponding to the 2D image to be processed according to the dense depth images corresponding to each object to be estimated in the 2D image to be processed.

[0154] Through the strategy of feature approximation learning, the deep learning network GAppNet can learn features similar to the network structure CDNet, and the deep learning network LGeoNet can learn features similar to the network structure VolNet. Since the network models CDNet and VolNet use the depth information (or 3D voxel information) of the image, the method implicitly learns the deep features of the image, making the learned features more robust and more discriminative. The network structures GAppNet and LGeoNet both use color (or grayscale) images as input, so in actual prediction, using the technology of the present invention, the 3D posture information of the object can be estimated from a single image, and the results can be similar to those using image and depth data at the same time. In addition, in actual testing, the two features can be combined or used separately for 3D posture estimation of the object.

[0155] To facilitate understanding of the exemplary embodiments, some exemplary embodiments of the method for obtaining the posture and dense depth image of an object based on a single color (or grayscale) image according to the present invention have been described and shown in the accompanying drawings. However, it should be understood that these exemplary embodiments are intended only to illustrate the exemplary embodiments, not to limit the exemplary embodiments. It should also be understood that the exemplary embodiments are not limited to the exemplary embodiments illustrated and described. A person of ordinary skill in the art may make various modifications to the exemplary embodiments.

Claims

1. A method for estimating an object posture, comprising: Obtain a two-dimensional 2D image corresponding to the object to be estimated; Extracting global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated; According to the extracted global visual features and / or local geometric features, three-dimensional 3D posture estimation is performed on the object to be estimated, The step of extracting the global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated includes: Extracting global visual features of a 2D image corresponding to an object to be estimated through a first deep learning network, wherein the first deep learning network is trained through a first feature approximation strategy based on an input 2D image and features obtained through a third deep learning network, so that the features obtained through the first deep learning network approximate features obtained through a third deep learning network, wherein the third deep learning network is trained based on the input 2D image and depth information corresponding to the 2D image; and / or Through the second deep learning network, local geometric features of the 2D image corresponding to the object to be estimated are extracted, wherein the second deep learning network is trained by a second feature approximation strategy based on the input 2D image and the features obtained through the fourth deep learning network, so that the features obtained through the second deep learning network approximate the features obtained through the fourth deep learning network, wherein the fourth deep learning network is trained based on the input 2D image and the 3D voxel information corresponding to the 2D image.

2. The method as claimed in claim 1, wherein the global visual feature is a visual feature that characterizes the entire object, and the local geometric feature is a geometric feature that characterizes a local part of the object.

3. The method of claim 1, wherein: The first deep learning network is obtained by the first feature approximation strategy using the following method: Setting a first auxiliary loss term in the loss function, wherein the first auxiliary loss term is a difference function between a feature obtained through the first deep learning network and a feature obtained through the third deep learning network; The first deep learning network is trained through the first auxiliary loss term so that the features obtained through the first deep learning network are close to the features obtained through the third deep learning network.

4. The method of claim 1, wherein: The second deep learning network is obtained by the second feature approximation strategy using the following method: Setting a second auxiliary loss term in the loss function, where the second auxiliary loss term is a difference function between features obtained through the second deep learning network and features obtained through the fourth deep learning network; The second deep learning network is trained through the second auxiliary loss term so that the features obtained through the second deep learning network are close to the features obtained through the fourth deep learning network.

5. The method according to claim 1, wherein obtaining a 2D image corresponding to the object to be estimated comprises: The 2D image to be processed is segmented into object regions to obtain a 2D image corresponding to the object to be estimated. The method of claim 1 , wherein another portion of the object in the local geometric feature is occluded or truncated.

7. The method according to claim 2, wherein the local geometric features are features of local key components or key points of the object.

8. A method for obtaining a dense depth image, comprising: Determine a dense depth image corresponding to the object to be estimated according to the 3D pose of the object to be estimated in the 2D image obtained by the object pose estimation method according to any one of claims 1 to 7; By gathering dense depth images corresponding to the objects to be estimated in the 2D image to be processed, a dense depth image corresponding to the 2D image to be processed is determined.

9. An object posture estimation device, comprising: An image acquisition module is used to acquire a two-dimensional 2D image corresponding to the object to be estimated; A feature extraction module, used to extract global visual features and / or local geometric features of the 2D image corresponding to the object to be estimated; a posture estimation module, configured to perform three-dimensional 3D posture estimation on the object to be estimated based on the extracted global visual features and / or local geometric features, The feature extraction module includes: a component for extracting global visual features of a 2D image corresponding to an object to be estimated through a first deep learning network; and / or a component for extracting local geometric features of a 2D image corresponding to an object to be estimated through a second deep learning network. The first deep learning network is trained by a first feature approximation strategy based on an input 2D image and features obtained by a third deep learning network, so that the features obtained by the first deep learning network approximate the features obtained by the third deep learning network, wherein the third deep learning network is trained based on an input 2D image and depth information corresponding to the 2D image, The second deep learning network is trained by a second feature approximation strategy based on the input 2D image and the features obtained by the fourth deep learning network, so that the features obtained by the second deep learning network are close to the features obtained by the fourth deep learning network, wherein the fourth deep learning network is trained based on the input 2D image and the 3D voxel information corresponding to the 2D image.

10. The device of claim 9, wherein: The global visual feature is a visual feature that characterizes the entire object, and the local geometric feature is a geometric feature that characterizes a local part of the object.

11. The device of claim 9, wherein: The first deep learning network is obtained by using the first feature approximation strategy using the following method: setting a first auxiliary loss term in the loss function, wherein the first auxiliary loss term is a difference function of features obtained by the first deep learning network and features obtained by the third deep learning network; and training the first deep learning network using the first auxiliary loss term so that the features obtained by the first deep learning network approximate the features obtained by the third deep learning network.

12. The device of claim 9, wherein: The second deep learning network is obtained by the second feature approximation strategy using the following method: setting a second auxiliary loss term in the loss function, wherein the second auxiliary loss term is a difference function of features obtained by the second deep learning network and features obtained by the fourth deep learning network; the second deep learning network is trained by the second auxiliary loss term so that the features obtained by the second deep learning network approximate the features obtained by the fourth deep learning network.

13. The device of claim 9, wherein: The image acquisition module includes: a component for segmenting the object region of the 2D image to be processed to obtain a 2D image corresponding to the object to be estimated. The apparatus of claim 9 , wherein another portion of the object in the local geometric feature is occluded or truncated. The device according to claim 10 , wherein the local geometric features are features of local key components or key points of the object.

16. A device for obtaining a dense depth image, comprising: A sub-image determination module, configured to determine a dense depth image corresponding to the object to be estimated according to the 3D posture of the object to be estimated in the 2D image obtained by the apparatus according to any one of claims 9 to 15; The total image determination module is used to determine the dense depth image corresponding to the 2D image to be processed by collecting the dense depth images corresponding to the objects to be estimated in the 2D image to be processed.

Images