Camera posture estimation device, learning method and program
The camera pose estimation device uses high-dimensional transformations and learning to accurately determine relative poses and shapes between cameras, addressing errors in conventional methods.
Patent Information
- Application Number
- JP2020188644
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-11-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-11-12
AI Technical Summary
Conventional methods struggle to accurately estimate the relative pose between cameras, especially when there are large camera posture transformations or no overlap in image appearance, leading to errors in feature point association and inability to determine relative poses directly.
A camera pose estimation device that uses high-dimensional transformations of features extracted from images or 3D point clouds to find similar features and estimate relative poses, incorporating a learning method to refine parameters for accurate pose and shape estimation.
Enables precise estimation of camera poses and shapes by learning from errors, improving accuracy in relative pose determination and shape reconstruction.
Smart Images

Figure 0007590162000001 
Figure 0007590162000002 
Figure 0007590162000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a camera posture estimation device, a learning method, and a program for estimating a relative posture between cameras. [Background technology]
[0002] Estimating the relative pose between cameras is an important issue in many computer vision applications such as Structure from Motion (SfM) and Simultaneous Localization and Mapping (SLAM). To address this issue, a method is generally used to estimate the relative pose between cameras using local features extracted from two images (for example, Patent Document 1). In this method, pose estimation and 3D shape estimation are performed simultaneously using corresponding feature points. That is, a hypothesis of a matrix that represents the pose transformation of the camera in a 3D space, called a fundamental matrix or a fundamental matrix, is constructed, and it is verified whether the corresponding feature points comply with the epipolar constraint defined by this hypothesis. This hypothesis and verification process is performed multiple times to select the most appropriate hypothesis. Then, the selected hypothesis, i.e., the fundamental matrix or fundamental matrix, is decomposed to obtain the relative pose between the cameras.
[0003] In recent years, a method has been proposed for simultaneously estimating the relative camera pose and estimating the three-dimensional shape from a single image using deep learning. For example, Non-Patent Document 1 proposes a method for extracting features from a single image using a convolutional neural network (CNN) as input, and estimating the camera pose and the three-dimensional shape from the features. In this method, the accuracy of both estimations is improved by performing multitask learning for these two estimations. The camera pose here is the camera pose relative to a standard camera pose (for example, a camera pose that photographs a chair from the front) defined during learning. In this method, the camera pose is estimated individually from two images, and then the relative pose between the cameras is calculated, thereby obtaining the relative pose between the cameras. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-021291 A [Non-patent literature]
[0005] [Non-Patent Document 1] Sun, Xingyuan, et al. "Pix3d: Dataset and methods for single-image 3d shape modeling." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. Summary of the Invention [Problem to be solved by the invention]
[0006] However, the above-described conventional techniques have problems in terms of accurately estimating the relative orientation between the cameras.
[0007] In the method of Patent Document 1, when it is difficult to correctly match feature points, it becomes difficult to estimate the relative pose between the cameras. For example, when the camera pose change is large, the number of errors in matching feature points increases, resulting in a large estimation error in the relative pose between the cameras. Also, when there is no overlap in appearance between the two images (for example, when the same object is photographed from opposite directions), there are no correctly corresponding feature points, and therefore it is not possible to estimate the relative pose between the cameras.
[0008] In the method of Non-Patent Document 1, although the camera poses can be estimated individually, the relative pose cannot be obtained directly. In other words, the similarity of the 3D shapes estimated from two images cannot be utilized during learning, and there is still room for improvement in the accuracy of relative pose estimation.
[0009] In view of the problems with the conventional techniques described above, an object of the present invention is to provide a camera posture estimation device and program capable of estimating the relative posture between cameras with high accuracy, and a method for learning parameters of the device. [Means for solving the problem]
[0010] In order to achieve the above object, the present invention provides a camera posture estimation device, comprising: a feature extraction unit for extracting a first feature and a second feature as representing a spatial feature from first data and second data obtained by photographing or measuring an identical object with different camera arrangements, a feature transformation unit for applying a high-dimensional transformation to the first feature to obtain a first transformed feature and applying a high-dimensional transformation and a plurality of pose transformations to the second feature to obtain a second transformed feature for each pose transformation, and a transformation search unit for searching for a second transformed feature for each pose transformation that is determined to be similar to the first transformed feature, thereby obtaining a pose transformation corresponding to the determined second transformed feature as a relative pose between the cameras of the first data and the second data, or a program for causing a computer to function as the camera posture estimation device. Also, the present invention provides a learning method, comprising: learning parameters used by the feature extraction unit using an error between the relative pose obtained by the camera posture estimation device for the first data and the second data for learning and a relative pose given as a true value.
[0011] A second feature of the camera posture estimation device is that it further comprises a shape estimation unit that estimates a first shape and a second shape as shapes of objects in the first data and the second data from the first transformation feature and a second transformation feature determined to be similar to the first transformation feature by the transformation search unit, or a program that causes a computer to function as the camera posture estimation device.A learning method is characterized in that it learns parameters used by the shape estimation unit using errors between the first shape and the second shape obtained by the camera posture estimation device for the first data and the second data for learning and the first shape and the second shape given as true values. Effect of the Invention
[0012] According to the camera posture estimation device or program of the first feature, the camera posture can be estimated with high accuracy by estimating the relative posture via a high-dimensional transformation feature. According to the learning method of the first feature, the parameters of the feature extraction process as the front-stage processing at this time can be learned.
[0013] According to the camera posture estimation device or program of the second feature, it is possible to estimate the camera posture with high accuracy and perform shape estimation by estimating the relative posture via a high-dimensional transformation feature. According to the learning method of the second feature, it is possible to learn parameters for the shape estimation process. [Brief description of the drawings]
[0014] [Figure 1] FIG. 2 is a functional block diagram of a camera posture estimation device and a learning device according to an embodiment. [Diagram 2] 4 is a flowchart of the operation of a camera posture estimation device and a learning device according to an embodiment. [Diagram 3] 2 is a diagram showing a camera posture estimation device and a learning device having a configuration in which the shape estimation unit is omitted, as an alternative embodiment to the configuration in FIG. 1. FIG. [Figure 4] FIG. 13 is a diagram showing a schematic example of high-dimensional transformation by a feature transformation unit. [Diagram 5] FIG. 1 illustrates an example of a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] 1 is a functional block diagram of a camera pose estimation device and a learning device according to an embodiment. As shown in the figure, a camera pose estimation device 10 includes a feature extraction unit 1, a transformation generation unit 2, a feature transformation unit 3, a transformation search unit 4, and a shape estimation unit 5. A learning device 20 is configured such that a learning unit 6 is further provided in addition to the camera pose estimation device 10.
[0016] 2 is a flowchart of the operation of the camera posture estimation device 10 and the learning device 20 according to an embodiment. In step S1, the learning device 20 acquires a large number of pairs (D1, D2) of first data D1 and second data D2 as learning data. k (k is an identifier of an individual learning data pair, k=1, 2, 3, ...) is read and learning is performed to learn the parameters of the camera pose estimation device 10 (more specifically, the parameters of the feature extraction unit 1 and the shape estimation unit 5 of the camera pose estimation device 10, each of which is configured as a deep learning network in one embodiment), and then the process proceeds to step S2. In step S2, the camera pose estimation device 10 to which the parameters learned in step S1 are set reads a pair (D1, D2) of the first data D1 and the second data D2, estimates the relative posture of the data pair (D1, D2) as an inference process, and ends the flow of FIG. 2. In step S2, the camera pose estimation device 10 may further estimate a shape (SH1, SH2) corresponding to the data pair (D1, D2) as an inference process.
[0017] The overall framework of the flow in Figure 2 above is similar to that generally performed in machine learning such as existing deep learning technologies, in that a learning process (step S1) is performed using training data to obtain network parameters, and then an inference process (step S2) is performed using the learned parameters. The learning method can be an existing method in deep learning, such as backpropagation.
[0018] The processing of the camera pose estimation device 10 during forward propagation to obtain an error during learning in step S1 (error evaluation and parameter update by error back propagation are performed in the learning unit 6) is the same as the processing of the camera pose estimation device 10 during inference in step S2. (However, the former uses parameters during learning, and the latter uses parameters that have already been learned.) The outline of the common processing of the camera pose estimation device 10 during learning and inference (the details of which will be described later) is as follows.
[0019] The feature extraction unit 1 extracts features from each pair of first and second data (D1, D2) input (as learning data or inference data), obtains a first feature F1 and a second feature F2, and outputs them to the feature conversion unit 3. The conversion generation unit 2 generates a plurality of conversions T i (i=1,2,…,N) (N is the number of transformations T i The feature conversion unit 3 generates a feature vector (the total number of features) and outputs it to the feature conversion unit 3.
[0020] In addition, the multiple transformations T i (i=1, 2, ..., N) is a list of candidates for the relative orientation to be estimated as the correct answer when estimating the relative orientation between the input data pair (D1, D2) in the camera orientation estimation device 10. i The process of searching for the transformation T corresponding to the relative orientation that is the correct answer in the transformation search unit 4 is performed in the latter stage. (As already mentioned, the process of the camera pose estimation device 10 is common to both learning and inference. i The process of searching for is performed both during training and inference.)
[0021] The feature conversion unit 3 performs feature conversion to increase the dimensionality of each of the feature data pairs (F1, F2) obtained from the feature extraction unit 1, and outputs the converted features to the conversion search unit 4. At this time, for the first feature F1, which is one of the feature data pair (F1, F2), only the feature conversion to increase the dimensionality is performed, to obtain a first converted feature T1, which is output to the conversion search unit 4. On the other hand, for the second feature F2, which is the other of the feature data pair (F1, F2), in addition to the feature conversion to increase the dimensionality similar to that applied to the first feature F1, multiple conversions T2 generated as candidates by the conversion generation unit 2 are also performed. i (i=1,2,…,N) are applied to generate multiple (N) second transformation features T2 i (i=1, 2, . . . , N) is output to the conversion search unit 4.
[0022] That is, the feature transformation to increase the dimension in the feature transformation unit 3 is expressed as a function high(), and similarly, the multiple transformations T i (i=1,2,…,N) to function T i If the expression is written as (), the processing of the feature conversion unit 3 can be formally expressed as the following equation (1) and equation (2a) or (2b). T1=high(F1) …(1) T2 i =high(T i (F2)) …(2a) T2 i =T i (high(F2)) …(2b)
[0023] As shown in the above formula (2a) or (2b), the feature conversion unit 3 converts the second feature F2 into a conversion T i (The transformation T i ) is applied to the high-dimensional transformation feature T2 i In obtaining the transformation T i Then, the high-dimensional transformation can be applied, or conversely, the high-dimensional transformation can be applied as shown in equation (2b) before the transformation T i may be applied.
[0024] In the conversion search unit 4, the first conversion feature T1 obtained in the feature conversion unit 3 is converted into a plurality (N) of second conversion features T2 obtained in the feature conversion unit 3. i (i=1, 2, ..., N) that is similar to the first transformed feature T1. The second transformed feature (feature with index i=j) that is determined to be similar to the first transformed feature T1 is determined to be similar to T2. j The conversion search unit 4 obtains the second conversion feature T2 that is determined to be similar. j (Not only one, but multiple j=i1, i2, ... may be judged to be similar) j (generated as candidates by the transformation generation unit 2) represents the relative orientation between the input data pair (D1, D2), and an estimation result of the relative orientation can be obtained.
[0025] In addition to outputting the relative pose estimation result, the conversion search unit 4 outputs the first conversion feature T1 obtained from the feature conversion unit 3 to the shape estimation unit 5 as is, and also outputs the second conversion feature T2 obtained from the feature conversion unit 3 and subjected to similarity judgment to the shape estimation unit 5. j (there may be multiple) to the shape estimation unit 5.
[0026] The shape estimation unit 5 calculates the first transformation feature T1 and the second transformation feature T2 obtained from the transformation search unit 4. j (There may be a plurality of input data) and a shape estimation (shape estimation of an object photographed in the original input data D1, D2) is performed, and a first shape SH1 and a second shape SH2 are obtained as the estimation results. j Output (multiple numbers are allowed).
[0027] In addition, in the shape estimation unit 5, the first shape SH1 and the second shape SH2 are used during learning. jto the learning unit 6 at the subsequent stage, and can be output as the estimation result in camera posture estimation device 10 during inference (as an additional estimation result to the estimation result of the relative posture in conversion search unit 4). Depending on the application of camera posture estimation device 10, during inference, only the processing up to conversion search unit 4 may be performed, and the processing of shape estimation unit 5 may be omitted. (In the configuration of FIG. 1, the processing of shape estimation unit 5 is not omitted during learning. Note that in the configuration of FIG. 4 described later, the shape estimation unit 5 itself is omitted.)
[0028] The learning unit 6 (used only during learning) calculates the first shape SH1 and the second shape SH2 obtained by the shape estimation unit 5. j The error between them is evaluated by the backpropagation method, and parameters based on the evaluation error are updated for the shape estimation unit 5 and the feature extraction unit 1 by the backpropagation method. Any existing method such as mini-batch learning may be used for this. Note that during forward propagation, processing is performed in the order of the following equation (3) as in the case of inference, and during backpropagation, processing is performed in the reverse order (and the transformation search unit 4 is skipped) as in the order of the following equation (4) (backpropagation of the error and parameter update of the shape estimation unit 5 and the feature extraction unit 1 due to this). Note that during backpropagation, the pair input corresponding to the error may be used in the same way as during forward propagation. (D1, D2) → [Feature extraction unit 1] → (F1, F2) → [Feature conversion unit 3] → (T1, T2 i ) →[Conversion search part 4] →(T1,T2 j ) → [Shape estimation unit 5] → (SH1, SH2 j ) (Relative attitude, SH1, SH2 j ) → [Learning unit 6] → (Error) … (3) (Error) → [Shape estimation unit 5] → (T1, T2 j ) → [Feature transformation part 3 (reverse transformation)] → (F1, F2) (F1, F2) → [Feature Extraction Unit 1] … (4)
[0029] The camera posture estimation device 10 as a whole uses a deep learning network, but among them, only the feature extraction unit 1 and the shape estimation unit 5 have parameters set as learning targets. The transformation generation unit 2 generates a transformation T i The feature conversion unit 2 generates parameters for the feature transformation unit 3, which performs fixed conversion processing (this inverse conversion processing is performed during reverse propagation of learning, but since the calculation has already been performed during forward propagation and data before and after the feature transformation unit 3 can be referenced, there is no need to actually perform a new inverse conversion processing), and the conversion search unit 4 performs search processing. These units 2, 3, and 4 do not have parameters set as the learning target.
[0030] As another embodiment, the camera pose estimation device 10 and the learning device 20 may be configured to omit the shape estimation unit 5 (FIG. 3). In the case of the configuration of FIG. 3, the difference from the configuration shown in FIG. 1 is that the output of the conversion search unit 4 (relative pose and conversion features T1, T2 j ) can be received by the learning unit 6. Then, during forward propagation and backward propagation, calculations can be performed according to the flow of the following equations (5) and (6) instead of the above equations (3) and (4). (D1, D2) → [Feature extraction unit 1] → (F1, F2) → [Feature conversion unit 3] → (T1, T2 i ) (T1, T2 i ) → [Transformation search part 4] → (relative orientation, T1, T2 j ) (Relative posture, T1, T2 j ) → [Learning unit 6] → (Error) … (5) (Error, relative attitude, T1, T2 j ) → [Feature transformation part 3 (reverse transformation)] → (F1, F2) (F1, F2) → [Feature extraction unit 1] … (6) As with equation (4), the inverse transformation of equation (6) in the feature transformation unit 3 has already been calculated during forward propagation, so there is no need to perform the inverse transformation again.
[0031] Above, we have explained the outline of the processing of each part in Figure 1. The details of each part will be explained below.
[0032] [Feature extraction part 1] The feature extraction unit 1 extracts features (spatial features) from the input data D1 and D2 to obtain feature data F1 and F2.
[0033] In one embodiment of the present invention, an image is given as input data, and an arbitrary CNN model is used to extract features (2D spatial features for each channel in W×H) represented by a 3D (W×H×C) tensor, where W, H, and C are the width, height, and number of channels of the tensor, respectively.
[0034] In one embodiment of the present invention, the input data may be a 3D point cloud or voxels. The 3D point cloud can be measured by a camera equipped with a depth sensor, and the voxels can be obtained by converting the 3D point cloud. If the input data is a 3D point cloud, it is first converted into voxels. Then, features expressed by a 4-dimensional (W×H×D×C) tensor (3D spatial features in each channel in W×H×D) are extracted from the voxels using an arbitrary 3D CNN model. Here, D is the depth of the tensor.
[0035] In addition, even when the two input data D1, D2 (for learning and for inference) are prepared in any of the above formats, the two input data D1, D2 may be prepared as images of the same object photographed or measured at two different camera postures (which may coincidentally be the same posture) (so that the relative posture between them can be estimated in the camera posture estimation device 10 from the data D1, D2). In addition, when the first data D1 is prepared as an image of the same object photographed from the front and the second data D2 is prepared as an image of the same object photographed from the opposite rear, for example, when the data D1, D2 are images, it is possible that no common surface is photographed even for the same object. However, the camera posture estimation device 10 of this embodiment can estimate the relative posture using three-dimensional information of the object even for the data D1, D2 in such a state.
[0036] [Conversion generation part 2] The transformation generator 2 generates multiple transformations T i (i=1,2,...,N) are generated. The parameters for transforming the camera posture can be composed of all or some of the roll angle, pitch angle, and yaw angle, which represent rotation in three-dimensional space, and the parallel translation in the x, y, and z directions, but the parameters may be given as data in other formats representing rotation and translation.
[0037] In one embodiment of the present invention, transformation parameters for camera postures that represent various relative postures of cameras are comprehensively generated. Specifically, values are sampled from a range specified for each parameter element, and the transformation parameters are generated by combining these values. As an example, for the roll angle, angles are sampled at 10° intervals in the range of [0°, 360°]. Other elements are sampled in the same manner. All sampled values may be used, or only a portion of them may be used.
[0038] For example, during learning and inference, if it is known that the relative orientation between the input data D1 and D2 falls within a certain range, parameters may be generated only within that certain range.
[0039] In one embodiment of the present invention, the camera posture relative to the standard camera posture is calculated by regression using the features F1 and F2 extracted by the feature extraction unit 1 during learning and inference. The relative posture between the input data D1 and D2 is calculated by regression between the features F1 and F2, and the posture estimation in the regression may use, for example, an existing method called RPNetFC in the following non-patent document 2. It may be regarded as a generated transformation parameter. (Since this embodiment calculates one true value of the relative posture by regression, it may be used for shape estimation in the shape estimation unit 5, not for estimating the relative posture as an inference process in the camera posture estimation device 10, or it may be used as a true value during learning for estimating the relative posture as an inference process.) [Non-Patent Document 2] En, S., Lechervy, A., & Jurie, F. (2018). Rpnet: An end-to-end network for relative camera pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 0-0).
[0040] In addition, in one embodiment of the present invention, during learning, the true values of the relative orientation of the cameras corresponding to the input data D1 and D2 may be regarded as the generated transformation parameters, and the search process in the transformation search unit 4 may be omitted.
[0041] [Feature conversion part 3] The feature conversion unit 3 converts the features F1 and F2 extracted by the feature extraction unit 1 into the above-mentioned formulas (1) and (2a) or (2b).
[0042] When a three-dimensional (W×H×C) tensor extracted from an image is given, it is converted into a four-dimensional (W×H×D×C) tensor by, for example, the method (unprojection) described in Non-Patent Document 3 below. FIG. 4 shows an outline of the conversion of a two-dimensional (W×C) tensor t2d into a three-dimensional (W×H×C) tensor t3d for one channel C for simplicity, as a schematic example of the high-dimensional conversion by the feature conversion unit 3. In this method, a virtual camera Cam is first installed at an arbitrary position. Then, a one-dimensional lattice consisting of the width direction of the two-dimensional tensor t2d is placed in front of the camera Cam, and a two-dimensional lattice is placed behind it. A light ray (three light rays L1, L2, and L3 are illustrated in FIG. 2) emitted from the camera Cam passes through a one-dimensional lattice (two-dimensional tensor t2d) and then a two-dimensional lattice (three-dimensional tensor t3d). At this time, the two-dimensional lattice is assigned the characteristics of the one-dimensional lattice through which the same light ray has passed. The same process can be used to convert a three-dimensional tensor into a four-dimensional tensor. In other words, the three-dimensional lattice is assigned the characteristics of the two-dimensional lattice through which the same light ray has passed.
[0043] □[Non-Patent Document 3] Kar, Abhishek, Christian Hane, and Jitendra Malik. "Learning a multi-view stereo machine." Advances in neural information processing systems. 2017.
[0044] Since the features represented by this three-dimensional lattice represent the three-dimensional shape of an object, the relative orientation of the virtual camera corresponds to the relative orientation of the camera in the real world. Therefore, if the relative orientation of the camera that acquired the input data D1 and D2 is reproduced using two virtual cameras, the two obtained features are considered to be similar. Here, unprojection (also called back projection) is performed on features F1 and F2 using two virtual cameras with various relative orientations. (Note that projection is generally a low-dimensional transformation, and its inverse, back projection, is a high-dimensional transformation.) Specifically, first, a three-dimensional lattice is positioned so that its center is located at the origin. Then, for feature F1, a virtual camera is placed in front of the three-dimensional lattice (for example, the negative point on the axis representing the depth direction) and unprojection is performed. The orientation of the camera at this time is p1. For the other feature F2, various orientation transformations T are performed using the orientation p1 as a reference. i (i=1, 2, ..., N) is used to perform unprojection. (That is, it is done as in the above-mentioned formula (2a).) In this posture transformation, the transformation T i (i=1,2,…,N).
[0045] Alternatively, after converting a three-dimensional (W×H×C) tensor into a four-dimensional (W×H×D×C) tensor, the four-dimensional tensor may be subjected to orientation transformation, for example, by the method described in Non-Patent Document 4 (rigid-body transformation). That is, features F1 and F2 are both transformed into a four-dimensional tensor using orientation p1. Then, the transformation generated by transformation generation unit 2 is applied only to the four-dimensional tensor obtained from feature F2. Specifically, the four-dimensional tensor is regarded as a three-dimensional lattice in which a C-dimensional vector is stored inside each lattice.
[0046] Then, the virtual camera is considered to be located at the origin, and this three-dimensional lattice is placed in front of the virtual camera (for example, the positive point on the axis representing the depth direction). The three-dimensional lattice at this time is called G1. Next, the 3D coordinates of the lattice G1 are subjected to posture transformation using the transformation generated by the transformation generation unit 2, and this is called G2. Then, to each lattice of G1, the C-dimensional features stored in each lattice of G2 that matches the position are assigned, and this is called G3. Here, since it is considered that the positions of the lattice of G1 and the lattice of G2 do not completely match, the assignment is performed using the nearest neighbor interpolation method or the trilinear interpolation method. If there is no lattice of G2 that matches the position of the lattice of G1, a C-dimensional vector with all elements set to 0 is assigned. The four-dimensional tensor represented by this G3 is called the posture-transformed tensor. (That is, it is as shown in the above-mentioned formula (2b).)
[0047] [Non-patent document 4] Nguyen-Phuoc, Thu, et al. "Hologan: Unsupervised learning of 3d representations from natural images." Proceedings of the IEEE International Conference on Computer Vision. 2019.
[0048] Furthermore, when the input data D1, D2 are not images but three-dimensional point clouds or voxels, the input data D1, D2 are already obtained as information on three-dimensional spatial features, so the process of high-dimensional transformation of the features F1, F2 by the feature transformation unit 3 may be omitted, and posture transformation may be performed on the four-dimensional tensor (features F1, F2 already obtained as three-dimensional spatial features) obtained by the feature extraction unit 1. Similarly, the transformation generated by the transformation generation unit 3 is applied only to the four-dimensional tensor obtained from F2. That is, in this case, the processing of the feature transformation unit 3 is expressed as a process in which the high-dimensional transformation high() in the above-mentioned formulas (1), (2a), and (2b) is replaced with an identity transformation, and formulas (2a) and (2b) are the same.
[0049] [Conversion search part 4] The conversion search unit 4 uses the conversion parameters T i From (i=1, 2, . . . , N), a search is made for features that are similar to those converted by the feature conversion unit 3, and an estimation result of the relative orientation between the input data D1 and D2 is obtained.
[0050] As already mentioned, only one feature (only the transformation feature T1) is obtained from the input data D1, but the number of transformation features T2 generated by the transformation generation unit 2 from the input data D2 is N. i (i=1, 2, ..., N) are obtained. Therefore, the transformed feature T1 obtained from the input data D1 and the transformed feature T2 obtained from the input data D2 are obtained. i Calculate the error for (i=1,2,…,N) and calculate the transformation feature T2 obtained from the input data D2. i (i=1,2,…,N) and select the k items with the smallest error (the top k items on the side with the smallest error (i.e., the side with the highest similarity), k≧1), and the search result T2 j Alternatively, the k items whose errors are judged to be small by the threshold judgment are output as search results T2 j The transformation feature T1 obtained from the input data D1 is output as is.
[0051] Here, the error measure is, for example, Euclidean distance, but other measures may be used. In one embodiment of the present invention, a coarse-to-fine search method may be used to improve the efficiency of the search. That is, the above search is first performed only on the transformation features corresponding to the transformation parameters sampled at coarse intervals. After that, the above search is performed on the transformation features corresponding to the transformation parameters sampled at fine intervals based on the transformation parameters corresponding to the obtained features.
[0052] [Shape estimation unit 5] The shape estimation unit 5 uses the second transformation feature T2 found by the transformation search unit 4. j (and the first transformed feature T1 output as is) to estimate the 3D shape of the object.
[0053] The three-dimensional shape may be expressed, for example, as voxels or a three-dimensional point cloud, or may be expressed as a depth image or a silhouette image estimated from a virtual camera using them. j If there are multiple similar features (for example, if the conversion search unit 4 searches for the top k similar features with k≧2), the three-dimensional shapes may be estimated for all of them. Then, the three-dimensional shapes SH1 and SH2 corresponding to the input data D1 and D2 are j Calculate the shape error between the two and select the one with the smallest error (multiple second shapes SH2 j In another embodiment, the three-dimensional shapes SH1, SH2 may be selected and output (output during inference or output to the learning unit 6 during learning) without being limited to the one with the smallest error during learning and inference. j and their respective errors may be output.
[0054] When applying the embodiment in which the one with the smallest error is output, the output during inference is the one three-dimensional shape SH2 in which the shape corresponding to the first input data D1 (the shape of the object included in the data D1) is the three-dimensional shape SH1 and the shape corresponding to the second input data D2 (the shape of the object included in the data D2) is the one with the smallest error. j may be output as
[0055] Here, the measure of shape error may be, for example, binary cross entropy in the case of voxels, Chamfer distance in the case of 3D point clouds, mean squared error of pixel values in the case of depth images or silhouette images, or other measures may be used.
[0056] [Learning Section 6] The learning unit 6 learns the model parameters of the neural network. Various embodiments are possible depending on how the error is evaluated and what kind of network configuration is learned (FIGS. 1 and 3). (Note that while the learning unit 6 evaluates the error, for convenience, FIG. 1 and FIG. 3 only depict a portion (or in some embodiments, the entirety) of the data flow referenced for error evaluation in some embodiments. For example, there are embodiments in which the learning unit 6 evaluates the error using features F1, F2, etc., but the flow of inputting features F1, F2, etc. to the learning unit 6 in this way is omitted from the perspective of preventing the drawings from becoming too complicated.)
[0057] For example, in the method LA, when the true values of the three-dimensional shapes corresponding to the input data D1 and D2 are given according to the supervised learning framework in the configuration of FIG. 1 and formulas (3) and (4), the three-dimensional shapes SH1 and SH2 estimated by the shape estimation unit 5 are j and true value SH1 真値 ,SH2 真値 The error between e(SH1,SH1 真値 ), e(SH2,SH2 真値 ) may be calculated to learn parameters of only the shape estimation unit 5, or parameters of both the shape estimation unit 5 and the feature extraction unit 1, or parameters of only the feature extraction unit 1. (e is an error function, and the same applies below.)
[0058] As another example, in method LB, when a true value of the relative orientation between input data D1 and D2 is given by the framework of supervised learning in the configuration of Fig. 3 and formulas (5) and (6), the parameters of feature extraction unit 3 may be learned by calculating error e (relative orientation, true value) between the relative orientation searched for by conversion search unit 3 and the true value. The true value of the relative orientation may be calculated, for example, by the above-mentioned regression method.
[0059] As another example, as the method LC, an evaluation method that can be used in the unsupervised learning framework in the configuration of FIG. 3 and formulas (5) and (6) (and can also be used in combination with the supervised learning framework) is to measure the consistency of the estimation results for each input data D1, D2 by extracting the feature F2 from the input data D2 by the feature extraction unit 1 and comparing it with the relative pose (T2 ja ) to transform the feature T2 ja (F2), and the error e(T2 ja (F2), F1) to learn the parameters of the feature extraction unit 1. Furthermore, when the relationship between the input data D1 and D2 is exchanged (relative pose T2 jb is searched) error e(T2 jb (F1), F2) may also be calculated to learn the parameters of the feature extraction unit 1. For example, the sum of these e(T2 ja (F2),F1) + e(T2 jb The parameters of the feature extraction unit 1 may be learned using (F1), F2) as errors. Here, the relative orientation searched for by the transformation search unit 4 may be transformed in the opposite direction (for example, by using the inverse matrix of the matrix representing the relative orientation) to perform the process of obtaining feature F1 from transformed feature T1 and the process of obtaining feature F2 from transformed feature T2 as the inverse process of the feature transformation unit 3. (The results of those already calculated during forward propagation may be stored, and it may be omitted to newly perform the inverse transformation.)
[0060] Similarly, a method for evaluating the consistency of the data D1 and D2 may be used in the above-mentioned method LA, and the input data D2 is input and the three-dimensional shape SH2 estimated by the shape estimation unit 5 is j is converted using the relative orientation searched for by the conversion search unit 4, and the error e (the converted SH2 j , SH1) is calculated. Also, the error e (posture-transformed SH1) when the relationship between the input data D1 and D2 is swapped is calculated. j , SH2) are also calculated. For example, the sum of these e (posture transformed SH2j , SH1) + e(posture-transformed SH1 j , SH2) may be used as an error to learn parameters of the shape estimation unit 5 and / or the feature extraction unit 1. Here, the shapes SH1, SH2 j Since the data also contains position information (posture is also expressed as a distribution of positions), and the position (posture) may differ even if the shape is the same, it is sufficient to calculate contrastive loss or triplet loss in order to learn a feature space in which the distance between features representing the same shape and the same posture is close, and the distance between features representing the same shape but different postures is large.
[0061] In either case of supervised or unsupervised learning, the input data pair D1, D2 is prepared by photographing or measuring the same object with different camera poses.
[0062] A loss function is defined as a weighted sum of all or some of these, and a model is trained to minimize this loss function. When updating parameters through repeated training, the loss function to be used may be switched (e.g., switched periodically). For example, methods LB and LC are used to train parameters of the feature extraction unit 1 in the configuration of Fig. 3, but methods LB and LC may be used in addition to or instead of method LA when training the feature extraction unit 1 in the configuration of Fig. 1.
[0063] As described above, according to each embodiment of the present invention, it is possible to extract discriminative features regarding the three-dimensional shape and orientation of an object, and by comparing these, it is possible to correctly estimate the relative orientation between cameras.
[0064] Supplementary and additional examples will be described below.
[0065] (1) In the above, as an explanatory example, two data sets D1 and D2 obtained by photographing or measuring the same object at different camera poses during learning and inference are used as a pair, but a similar set of three or more data sets D1, D2, and D3 may also be used. When three data sets are used, three pairs (D1, D2), (D2, D3), and (D3, D1) are possible, so all or some of these can be treated in the same way as in the case of two data pairs, and the sum of the errors for each data pair may be used when calculating the error.
[0066] (2) Although the feature extraction unit 1 and the shape estimation unit 5 have been described as being implemented using a deep learning network such as CNN, any configuration that can perform similar processing using a learning framework may be used. For example, a neural network that is not necessarily deep may be used, or other general machine learning methods may be used.
[0067] (3) FIG. 5 is a diagram showing an example of a hardware configuration of a general computer device 70. Each of the camera posture estimation device 10 and the learning device 20 can be realized as one or more computer devices 70 having such a configuration. When each of the camera posture estimation device 10 and the learning device 20 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, a keyboard, a touch panel, or the like, a camera 78, and a bus BS for transmitting and receiving data between them.
[0068] Each functional unit of the camera posture estimation device 10 and the learning device 20 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the above, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the above. Processing results by the camera posture estimation device 10 and the learning device 20 may be displayed and output on the display 76. All or part of the image used as an input to each of the camera posture estimation device 10 and the learning device 20 may be obtained by photographing it with a camera 78. [Explanation of symbols]
[0069] 10...Camera pose estimation device, 20...Learning device 1... feature extraction unit, 2... transformation generation unit, 3... feature transformation unit, 4... transformation search unit, 5... shape estimation unit, 6... learning unit
Claims
1. a feature extraction unit that extracts a first feature and a second feature, each of which is given as a three-dimensional tensor and represents a spatial feature, from first data and second data obtained by photographing or measuring the same object with different camera arrangements; a feature transformation unit that applies a high-dimensional transformation to the first features to obtain first transformed features as four-dimensional tensors representing at least a three-dimensional shape and a posture of an object, and applies a high-dimensional transformation and a plurality of posture transformations prepared in advance to the second features to obtain second transformed features as four-dimensional tensors representing at least a three-dimensional shape and a posture of the object for each posture transformation; a transformation search unit that searches for second transformation features for each pose transformation that are determined to be similar to the first transformation feature, and obtains a pose transformation corresponding to the determined second transformation feature as a relative pose between the cameras of the first data and the second data.
2. a feature extraction unit that extracts a first feature and a second feature representing a spatial feature from first data and second data obtained by photographing or measuring the same object at different camera arrangements, respectively; a feature transformation unit that applies a high-dimensional transformation to the first features to obtain first transformed features, and applies a high-dimensional transformation and a plurality of posture transformations to the second features to obtain second transformed features for each posture transformation; a transformation search unit that searches for a second transformation feature determined to be similar to the first transformation feature from among second transformation features for each of the posture transformations, and obtains a posture transformation corresponding to the determined second transformation feature as a posture corresponding to a relative posture between the cameras of the first data and the second data, the feature transformation unit performs the high-dimensional transformation when the first feature and the second feature are given as three-dimensional tensors, and omits performing the high-dimensional transformation when the first feature and the second feature are given as four-dimensional tensors, since the first feature and the second feature have already been made high-dimensional.
3. a feature extraction unit that extracts a first feature and a second feature representing a spatial feature from first data and second data obtained by photographing or measuring the same object at different camera arrangements, respectively; a feature transformation unit that applies a high-dimensional transformation to the first features to obtain first transformed features, and applies a high-dimensional transformation and a plurality of posture transformations to the second features to obtain second transformed features for each posture transformation; a transformation search unit that searches for a second transformation feature determined to be similar to the first transformation feature from among second transformation features for each of the posture transformations, and obtains a posture transformation corresponding to the determined second transformation feature as a posture corresponding to a relative posture between the cameras of the first data and the second data, A camera posture estimation device further comprising a shape estimation unit that estimates a first shape and a second shape as the shapes of objects in the first data and the second data, respectively, based on the first conversion feature and a second conversion feature determined to be similar to the first conversion feature by the conversion search unit.
4. The conversion search unit searches for a plurality of second conversion features having a high degree of similarity; 4. The camera posture estimation device according to claim 3, wherein the shape estimation unit searches for a most similar second shape among a plurality of first shapes for one first shape, and outputs the one first shape as the shape of an object included in the first data, and the one most similar second shape as the shape of an object included in the second data.
5. a feature extraction unit that extracts a first feature and a second feature representing a spatial feature from first data and second data obtained by photographing or measuring the same object at different camera arrangements, respectively; a feature transformation unit that applies a high-dimensional transformation to the first features to obtain first transformed features, and applies a high-dimensional transformation and a plurality of posture transformations to the second features to obtain second transformed features for each posture transformation; a transformation search unit that searches for a second transformation feature determined to be similar to the first transformation feature from among second transformation features for each of the posture transformations, and obtains a posture transformation corresponding to the determined second transformation feature as a posture corresponding to a relative posture between the cameras of the first data and the second data, the feature conversion unit performs the high-dimensional conversion when the first data and the second data are given as images, and when the first data and the second data are given as three-dimensional point clouds or voxels, the camera posture estimation device omits performing the high-dimensional conversion, assuming that the first data and the second data have already been made high-dimensional.
6. 6. The camera posture estimation device according to claim 1, wherein each of the plurality of posture transformations is given as parameters representing a rotation and / or a translation in a three-dimensional space.
7. 7. The camera posture estimation device according to claim 1, wherein the conversion search unit searches for an object determined to be similar in terms of shape.
8. A learning method for learning parameters used by a feature extraction unit in a camera posture estimation device according to claim 1, comprising the steps of: extracting first and second features from the first and second learning data using the feature extraction unit to obtain first and second features as three-dimensional tensors; applying a high-dimensional transformation to the first feature and the second feature by the feature transformation unit to obtain a first transformation feature as a four-dimensional tensor representing at least a three-dimensional shape and orientation of the object and a second transformation feature as a four-dimensional tensor representing at least a three-dimensional shape and orientation of the object for each of a plurality of previously prepared orientation transformations; A search is performed by the transformation search unit on the first transformation feature and a second transformation feature for each pose transformation to obtain a relative pose between the cameras of the first data and the second data; a learning method for learning parameters used by the feature extraction unit using an error between the obtained relative orientation and a relative orientation given as a true value.
9. A learning method for learning parameters used by a feature extraction unit in a camera posture estimation device according to any one of claims 1 to 7, comprising the steps of: obtaining a first feature and a second feature by performing extraction by the feature extraction unit on the first data and the second data for learning; applying a high-dimensional transformation to the first feature and the second feature by the feature transformation unit to obtain a first transformation feature and a second transformation feature for each posture transformation; A search is performed by the transformation search unit on the first transformation feature and a second transformation feature for each pose transformation to obtain a relative pose between the cameras of the first data and the second data; a first error between a second transformed feature obtained by applying the obtained relative orientation to a second feature and the first transformed feature; A learning method comprising: learning parameters used by the feature extraction unit using the first error and a second error obtained by interchanging the first data and the second data.
10. A learning method for learning parameters used by a shape estimation unit in a camera posture estimation device according to claim 3 or 4, comprising the steps of: obtaining a first feature and a second feature by performing extraction by the feature extraction unit on the first data and the second data for learning; applying a high-dimensional transformation to the first feature and the second feature by the feature transformation unit to obtain a first transformation feature and a second transformation feature for each posture transformation; a search is performed by the conversion search unit on the first conversion feature and a second conversion feature for each posture conversion to obtain a second conversion feature that is determined to be similar to the first conversion feature; estimating shapes by the shape estimation unit for the first conversion feature and the second conversion feature determined to be similar to obtain a first shape and a second shape; A learning method comprising: learning parameters used by the shape estimation unit using errors between the obtained first and second shapes and the first and second shapes given as true values.
11. the transformation search unit obtains a pose transformation corresponding to the second transformation feature determined to be similar during the search as a relative pose between the cameras of the first data and the second data, Instead of using errors between the obtained first and second shapes and the first and second shapes given as true values, a first error between the obtained first shape and a shape obtained by applying the transformation of the relative pose to the second shape; 11. The learning method according to claim 10, further comprising: learning parameters used by the shape estimation unit using the first error and a second error obtained by interchanging the first data and the second data.
12. A program for causing a computer to function as the camera posture estimation device according to any one of claims 1 to 7.
Citation Information
Patent Citations
Single-camera rotary splicing method based on deep learning
CN111899174A
Database generation device, camera attitude estimation device, database generation method, camera attitude estimation method and program
JP2014164483A
Image processing device and control method therefor
JP2017059049A
Camera pose estimating method and system
JP2019021291A
Information processing apparatus, information processing method, and program
JP2020126590A
