Method for obtaining camera pose by utilizing relative pose constraint of adjacent and long-distance cameras covering scene space-time space

Abstract

An image processing apparatus for global camera relocation, the image processing apparatus comprising one or more processors to: receive an input image representing at least a portion of a scene; the input image is input into a trained weighted processing network, the trained weighted processing network is trained by using two sequences of input images and relative poses determined for each input image, and the trained weighted processing network comprises one or more parameters; based on the input image, using the trained weighted processing network, one or more of: estimated image coordinates in a global frame of the scene; estimated image coordinates in a camera frame of the scene; one or more weights associated with the coordinates within the scene; a pose of the input image is estimated based on one or more of the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, the one or more weights associated with the coordinates within the scene, and / or absolute image coordinates.

Description

Technical Field

[0001] This invention relates to an image processing apparatus and method for determining global camera repositioning. The invention also relates to an apparatus and method for training a weighted processing network to determine global camera repositioning. Background Technology

[0002] In many fields such as robotics and augmented / virtual reality, estimating the position and orientation of a camera based on a single image plays a crucial role, and therefore remains a core research topic in computer vision. With the emergence of deep learning in computer vision, recent methods have begun to utilize neural networks for data-driven camera pose estimation, hereinafter referred to as pose estimation.

[0003] The current problem is that it is difficult to perform global camera pose estimation from a single image. To address this problem, the training data consists of a set of images and their corresponding poses in a given reference frame.

[0004] Previous research has proposed solving the global localization problem by formulating a scheme based on a regression problem, where the input 2D image is mapped to a 6-degree-of-freedom (DoF) pose. In this case, a computational neural network (CNN) learns a function that encodes the input into latent features. Then, one or more regression layers can be used to map the latent image representation to pose (position and orientation). This method employs direct supervision, where each image is supervised by its corresponding pose.

[0005] To improve pose accuracy, previous studies have imposed various constraints on the learning process to better encode latent vectors, resulting in more accurate poses. One method in PoseNet addresses the loss imbalance between orientation and translation losses by learning weighting factors during training. Other studies have added constraints created from consecutive frames to improve localization. Another approach (AtLoc) employs an attention-based method to force the network to focus on robust objects and features, further leveraging image sequences to learn temporally consistent informative features. Finally, yet another approach (GeoPoseNet) further utilizes reprojection loss, which leverages a 3D model of the scene.

[0006] The methods described above address the global localization problem through regression, directly supervised by pose labels. While some of these methods utilize image sequences, they rely on consecutive and spatially neighboring images and still estimate pose via regression. Following this line of thought, this method takes an image as input and maps it to a 6-DoF pose (3 for position and 3 for orientation). However, because these methods do not utilize geometric constraints in a manner consistent with the geometry of pose estimation, their accuracy is measured in meters. In summary, the localization error of the methods described above is relatively high. Summary of the Invention

[0007] An image processing apparatus for global camera relocalization includes one or more processors configured to: receive an input image representing at least a portion of a scene; input the input image into a trained weighted processing network, wherein the trained weighted processing network is trained using two sequences of input images and a relative pose determined for each input image, the trained weighted processing network including one or more parameters; based on the input image, compute one or more of the following using the trained weighted processing network: estimated image coordinates in a global frame of the scene; estimated image coordinates in a camera frame of the scene; one or more weights associated with the coordinates within the scene; and estimate the pose of the input image based on one or more of the following: the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, the one or more weights associated with the coordinates within the scene, and / or absolute image coordinates. This provides the advantage that global camera relocalization can be performed based on a single input image, and that no additional labels are required due to the added geometric constraints.

[0008] In the image processing apparatus described above, the determined relative pose is based on comparing one or more first poses associated with a first image in a first sequence of image pose pairs with: another pose associated with another image in the first sequence and a pose associated with an image in a second sequence of image pose pairs. This provides the advantage of providing more accurate pose repositioning using images that are spatially and temporally distant.

[0009] In the image processing apparatus described above, the one or more processors are configured to: when estimating the pose of the input image, employ weighted aligned pose estimation by estimating the pose of the input image using one or more of the following: estimated image coordinates in the global frames of the scene, estimated image coordinates in the camera frames of the scene, and the one or more weights associated with the coordinates within the scene. This provides the advantage that the coordinates of the camera frames and the global frames can be appropriately mixed.

[0010] In the image processing apparatus described above, the one or more processors are configured to: when estimating the pose of the input image, employ unweighted aligned pose estimation by estimating the pose of the input image using one or more of the following: estimated image coordinates in the global frames of the scene and estimated image coordinates in the camera frames of the scene. This provides the advantage that the coordinates of the camera frames and the global frames can be combined in an approximate manner with less computation required.

[0011] In the image processing apparatus described above, the one or more processors are further configured to: employ perceptual point alignment when estimating the pose of the input image by using one or more of the following to estimate the pose of the input image: the estimated image coordinates in the global frame of the scene and the 2D image coordinates of the input image. This provides the advantage that only the coordinates of the global frame need to be calculated, and then the pose can be estimated using the 2D pixel information (image coordinates) of the input image, thereby reducing processing time.

[0012] In the image processing apparatus described above, the one or more processors are further configured to: use a random sampling consistency technique when estimating the pose of the input image. This provides a method for combining global frame coordinates of the input image with 2D image coordinates.

[0013] In the image processing apparatus described above, the one or more processors are configured to: employ weighted alignment pose estimation by estimating the pose of the input image using one or more of the following: estimated image coordinates in a global frame of the scene, estimated image coordinates in a camera frame of the scene, and one or more weights associated with the coordinates within the scene; and employ perceptual point alignment by estimating the pose of the input image using one or more of the following: estimated image coordinates in a global frame of the scene and 2D image coordinates of the input image. This provides the advantage that multiple estimated poses can be generated from the same input image, and that multiple estimated poses can be combined, if necessary, to produce a more accurate estimated pose.

[0014] In the image processing apparatus described above, the trained weighted processing network is trained by: receiving two or more sequences, wherein the two or more sequences include one or more image pose pairs of a first sequence and one or more image pose pairs of a second sequence; determining one or more relative poses for one or more image pose pairs in the first sequence and the one or more image pose pairs of the second sequence; iteratively calculating one or more estimated poses using the weighted processing network based on the determined one or more relative poses; determining the error between the one or more estimated poses and the one or more relative poses for each calculated estimated pose; and updating the one or more parameters of the weighted processing network based on the determined error. This provides the advantage that a minimal number of available labels (constraints) can be used to create additional signals for training, thus requiring only weak supervision to train the weighted processing network for accurate pose estimation.

[0015] In the image processing apparatus described above, the one or more processors are configured to train the parameters of the trained weighted processing network by: receiving a first sequence of image pose pairs, the first sequence of image pose pairs including a first set of images and an associated first absolute pose set; receiving a second sequence of image pose pairs, the second sequence of image pose pairs including a second set of images and an associated second absolute pose set; for the one or more image pose pairs in the first sequence, iteratively determining one or more relative poses based on comparing one or more first poses associated with a first image in the first sequence with: another pose associated with another image in the first sequence and a pose associated with an image in the second sequence; refining the one or more parameters of the weighted processing network by iteratively performing the following operations: calculating one or more estimated poses using the weighted processing network based on the one or more image pose pairs from the first sequence, the one or more image pose pairs from the second sequence, and the determined one or more relative poses; determining an error between the one or more estimated poses and the one or more relative poses for each calculated estimated pose; and updating the one or more parameters of the weighted processing network based on the determined error. This offers the advantage that additional signals for training can be created using minimal available labels (constraints), thus requiring only weak supervision to train the weighted processing network, enabling accurate pose estimation based on images with spatial and temporal distances.

[0016] In the image processing apparatus described above, the parameters of the weighted processing network are fine-tuned iteratively by performing the following operations: inputting one or more image location pairs into the image processing apparatus; calculating an estimated pose including location information; determining a fine-tuning error between the estimated pose and a known image location pair; and updating the one or more parameters of the weighted processing network based on minimizing the fine-tuning error. This provides the advantage that the weighted processing network can be modified in a customized manner based on limited data (location data in this example).

[0017] An image processing method for global camera relocalization includes: receiving an input image representing at least a portion of a scene; inputting the input image into a trained weighted processing network, wherein the trained weighted processing network is trained using two sequences of input images and a relative pose determined for each input image, the trained weighted processing network including one or more parameters; calculating, based on the input image, one or more of the following using the trained weighted processing network: estimated image coordinates in a global frame of the scene; estimated image coordinates in a camera frame of the scene; one or more weights associated with the coordinates within the scene; and estimating the pose of the input image based on one or more of the following: the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, the one or more weights associated with the coordinates within the scene, and / or absolute image coordinates. This provides the advantage that global camera relocalization can be performed based on a single input image, and that no additional labels are required due to the added geometric constraints.

[0018] In the image processing method described above, the determined relative pose is based on comparing one or more first poses associated with a first image in the first sequence with another pose associated with another image in the first sequence and a pose associated with an image in the second sequence. This provides the advantage of providing more accurate pose repositioning using images that are spatially and temporally distant.

[0019] In the image processing method described above, estimating the pose of the input image includes employing a weighted aligned pose estimation by estimating the pose of the input image using one or more of the following: estimated image coordinates in the global frame of the scene, estimated image coordinates in the camera frame of the scene, and one or more weights associated with the coordinates within the scene. This provides the advantage that the coordinates of the camera frame and the global frame can be appropriately mixed.

[0020] In the image processing method described above, estimating the pose of the input image includes employing unweighted aligned pose estimation by estimating the pose of the input image using one or more of the following: estimated image coordinates in the global frame of the scene and estimated image coordinates in the camera frame of the scene. This provides the advantage that the coordinates of the camera frame and the global frame can be combined in an approximate manner with less computation required.

[0021] In the image processing method described above, estimating the pose of the input image includes employing perceptual point alignment by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene and the 2D image coordinates of the input image. This provides the advantage that only the coordinates of the global frame need to be calculated, and then the pose can be estimated using the 2D pixel information (image coordinates) of the input image, thereby reducing processing time.

[0022] In the image processing method described above, estimating the pose of the input image includes using a random sampling consistency technique. This provides a method for combining the global frame coordinates of the input image with 2D image coordinates.

[0023] In the image processing method described above, estimating the pose of the input image includes: employing weighted alignment pose estimation by estimating the pose of the input image using one or more of the following: estimated image coordinates in the global frame of the scene, estimated image coordinates in the camera frame of the scene, and one or more weights associated with the coordinates within the scene; and employing perceptual point alignment by estimating the pose of the input image using one or more of the following: estimated image coordinates in the global frame of the scene and the 2D image coordinates of the input image. This provides the advantage that multiple estimated poses can be generated from the same input image, and that multiple estimated poses can be combined, if necessary, to produce a more accurate estimated pose.

[0024] The image processing method described above, wherein the trained weighted processing network is trained by: receiving two or more sequences, wherein the two or more sequences include one or more image pose pairs of a first sequence, and the two or more sequences further include one or more image pose pairs of a second sequence; for one or more image pose pairs in the image pose pairs, determining one or more relative poses based on the one or more image pose pairs of the first sequence and the one or more image pose pairs of the second sequence; iteratively calculating one or more estimated poses using the weighted processing network based on the determined one or more relative poses; for each calculated estimated pose, determining the error between the one or more estimated poses and the one or more relative poses; and updating the one or more parameters of the weighted processing network based on the determined error. This provides the advantage that the additional signals for training can be created using minimal available labels (constraints), thus requiring only weak supervision to train the weighted processing network for accurate pose estimation.

[0025] In the image processing method described above, the trained weighted processing network is trained by: receiving image pose pairs of a first sequence, wherein the image pose pairs include a first set of images and its associated first absolute pose set; receiving image pose pairs of a second sequence, wherein the image pose pairs include a second set of images and its associated second absolute pose set; for the one or more image pose pairs in the first sequence, iteratively determining one or more relative poses based on comparing one or more first poses associated with the first image in the first sequence with: another pose associated with another image in the first sequence and a pose associated with an image in the second sequence; refining one or more parameters of the weighted processing network by iteratively performing the following operations: calculating the one or more estimated poses using the weighted processing network based on the one or more image pose pairs from the first sequence, the one or more image pose pairs from the second sequence, and the determined one or more relative poses; determining the error between the one or more estimated poses and the one or more relative poses for each calculated estimated pose; and updating the one or more parameters of the weighted processing network based on the determined error. This offers the advantage that additional signals for training can be created using minimal available labels (constraints), thus requiring only weak supervision to train the weighted processing network, enabling accurate pose estimation based on images with spatial and temporal distances.

[0026] The image processing method described above further includes iteratively fine-tuning the trained weighted processing network by: inputting one or more image location pairs into an image processing device; calculating an estimated pose including location information; determining a fine-tuning error between the estimated pose and a known image location pair; and updating the one or more parameters of the weighted processing network based on minimizing the fine-tuning error. This provides the advantage that the weighted processing network can be modified in a customized manner based on limited data (location data in this example).

[0027] A training apparatus for training a weighted processing network to process images to determine camera positioning, wherein the apparatus includes one or more processors, the one or more processors being configured to: receive two or more sequences, wherein the two or more sequences include image pose pairs of a first sequence, the first sequence image pose pairs including a first set of images and its associated first absolute pose set, the two or more sequences further including image pose pairs of a second sequence, the second sequence image pose pairs including a second set of images and its associated second absolute pose set; and, for the one or more image pose pairs in the first sequence, based on one or more first poses associated with the first image in the first sequence... One or more relative poses are iteratively determined using another pose associated with another image in the first sequence and a pose associated with an image in the second sequence. One or more parameters of the weighted processing network are refined iteratively by performing the following operations: calculating one or more estimated poses using the weighted processing network based on the one or more image pose pairs from the first sequence, the one or more image pose pairs from the second sequence, and the determined one or more relative poses; determining the error between the one or more estimated poses and the one or more relative poses for each calculated estimated pose; and updating the one or more parameters of the weighted processing network based on the determined error. This provides the advantage that minimal additional signals for training can be created using available labels (constraints), thus requiring only weak supervision to train the weighted processing network for accurate pose estimation based on images with spatial and temporal distances.

[0028] In the training apparatus described above, the determined error includes one or more of the following: a minimization error based on the difference between the two or more image pose pairs in the first sequence; or a minimization error based on the difference between one or more image pose pairs in the first sequence and their associated absolute values ​​and estimates, and one or more image pose pairs in the second sequence. This provides the advantage of reducing errors between input images, thereby enabling more accurate pose estimation.

[0029] In the training apparatus described above, the determined error is further determined by: calculating one or more errors between two or more image pose pairs in the first sequence and / or the second sequence based on comparing the ground truth for each image pose pair with the estimated relative pose for each image pose pair; and summing and minimizing the one or more calculated errors. This provides the advantage that errors can be reduced based on known ground truth, thereby reducing the inaccuracy of the parameters of the weighted processing network.

[0030] The training apparatus described above, wherein the one or more image pose pairs of the first sequence are spatially and temporally distant from the one or more image pose pairs of the second sequence. This provides the advantage of providing more accurate pose relocalization using images that are spatially and temporally distant.

[0031] The training apparatus described above, wherein the one or more processors are configured to: determine the one or more relative poses of the image pose pairs by comparing the poses of the image pose pairs in the first sequence and / or the second sequence with another pose of one or more other image pose pairs. This provides the advantage of creating geometric constraints based on input image sequence data.

[0032] In the training apparatus described above, the one or more processors are further configured to iteratively perform the following operations: inputting one or more image location pairs into the image processing apparatus; calculating an estimated pose including location information; determining a fine-tuning error between the estimated pose and a known image location pair; and updating the one or more parameters of the weighted processing network based on minimizing the fine-tuning error. This provides the advantage that the weighted processing network can be modified in a customized manner based on limited data (location data in this example).

[0033] A method for training a weighted processing network to process images to determine camera localization, the method comprising: receiving two or more sequences, wherein the two or more sequences include image pose pairs of a first sequence, the first sequence of image pose pairs including a first set of images and an associated first absolute pose set, the two or more sequences further including image pose pairs of a second sequence, the second sequence of image pose pairs including a second set of images and an associated second absolute pose set; for the one or more image pose pairs in the first sequence, iteratively determining one or more relative poses based on one or more first poses associated with a first image in the first sequence, another pose associated with another image in the first sequence, and a pose associated with an image in the second sequence; refining one or more parameters of the weighted processing network by iteratively performing the following operations: calculating one or more estimated poses using the weighted processing network based on the one or more image pose pairs from the first sequence, the one or more image pose pairs from the second sequence, and the determined one or more relative poses; for each calculated estimated pose, determining the error between the one or more estimated poses and the one or more relative poses; and updating the one or more parameters of the weighted processing network based on the determined error. This offers the advantage that additional signals for training can be created using minimal available labels (constraints), thus requiring only weak supervision to train the weighted processing network, enabling accurate pose estimation based on images with spatial and temporal distances.

[0034] The method for training a weighted processing network as described above, wherein the determined error includes one or more of the following: a minimization error based on the difference between the two or more image pose pairs in the first sequence; or a minimization error based on the difference between one or more image pose pairs in the first sequence and their associated absolute values ​​and estimates, and one or more image pose pairs in the second sequence. This provides the advantage of reducing the error between input images, thereby achieving more accurate pose estimation.

[0035] The method for training the weighted processing network as described above, wherein the determined error is further determined by: calculating one or more errors between two or more image pose pairs in the first sequence and / or the second sequence based on comparing the ground truth for each image pose pair with the estimated relative pose for each image pose pair; and summing and minimizing the one or more calculated errors. This provides the advantage that errors can be reduced based on known ground truth, thereby reducing the inaccuracy of the parameters of the weighted processing network.

[0036] The method for training a weighted processing network as described above, wherein the one or more image pose pairs of the first sequence are spatially and temporally distant from the one or more image pose pairs of the second sequence. This provides the advantage of providing more accurate pose relocalization using images that are spatially and temporally distant.

[0037] The method for training a weighted processing network as described above, wherein the one or more processors are configured to: determine the one or more relative poses of the image pose pairs by comparing the poses of the image pose pairs in the first sequence and / or the second sequence with another pose of one or more other image pose pairs. This provides the advantage of creating geometric constraints based on the input image sequence data.

[0038] The method for training a weighted processing network as described above further includes iteratively fine-tuning the parameters of the weighted processing network by: inputting one or more image location pairs into the image processing device; calculating an estimated pose including location information; determining a fine-tuning error between the estimated pose and a known image location pair; and updating the one or more parameters of the weighted processing network based on minimizing the fine-tuning error. This provides the advantage that the weighted processing network can be modified in a customized manner based on limited data (location data in this example). Attached Figure Description

[0039] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings. In the drawings: Figure 1 An example of the training device of the present invention is shown; Figure 2 A general example of the fine-tuning process of the present invention is shown; Figure 3 An example of the image processing apparatus of the present invention is shown; Figure 4 An example of pose estimation using weighted rigid alignment and perspective n-point method in the RANSAC scheme is shown; Figure 5 An example of pose estimation via weighted rigid alignment is shown; Figure 6 An example of pose estimation using an unweighted rigid alignment module is shown; Figure 7 An example of pose estimation using only the perspective n-point module in the RANSAC framework is shown. Detailed Implementation

[0040] This invention proposes a novel apparatus and method for global camera relocalization based on a single image using deep learning. Addressing current problems, the proposed apparatus and method utilize a minimal number of available pose labels to create additional signals for training. The proposed apparatus operates in real-time, storing only the network weights and not relying on a reference 3D model for localization. In other words, this invention provides a solution to the problem of determining the camera's rotation and position relative to a global frame of the scene, given an input image.

[0041] In summary, this invention employs a network with relative geometric constraints derived from available images and their corresponding poses used for training. These geometric constraints are computed as relative poses between available absolute poses of the image training set. These constraints can be obtained from images that are spatially and temporally close as well as images that are far apart from each other. These constraints can be created randomly and applied simultaneously in each training iteration. These constraints can also be created from the available image and pose label set, potentially without the need to collect additional data.

[0042] These sets of relative geometric constraints can be used to train deep neural networks to learn the geometry of a scene. In one instance, the invention describes a trained network capable of accepting a single input (which may be a single input image) and outputting one or more quantities, preferably three. These three quantities include a set of weighting factors and two sets of 3D point clouds: one set in the image coordinate frame and the other in the global reference frame. A weighted rigid alignment module can take these three output quantities as input and use them to estimate the pose. During training, the pose is adjusted to match the ground truth pose. This is accomplished by using pose labels and a newly created relative geometric constraint-supervised deep neural network. These constraints are created from existing labels without additional manipulation.

[0043] These constraints support training by helping the neural network encode these geometric constraints in the network's weights. Therefore, during inference, the deep neural network estimates the pose constrained by these constraints. This improves pose accuracy. Furthermore, it eliminates the need for explicit 3D coordinates (e.g., explicit 3D coordinates from LiDAR or depth measurement units). Experiments further demonstrate that these constraint sets can help the network improve position and orientation localization when fine-tuned using only position labels.

[0044] During inference, the network can estimate the pose based on a single input image using its stored weights. The deep neural network estimates a set of weighting factors and two sets of 3D coordinates: one set of 3D point clouds in the global frame and the other set of 3D coordinates in the camera frame. Therefore, the method disclosed in this paper can compute the pose using a weighted rigid alignment module, an unweighted rigid alignment module, or a perspective n-point module employing a random sample consensus (RANSAC) scheme.

[0045] The apparatus and method for inference described in this paper utilize a single image for localization, represent the scene as a set of deep network parameters, save only the network weights (without requiring a 3D SFM model), and run in real time.

[0046] The apparatus and method of the present invention will now be described with reference to the accompanying drawings. First, the training apparatus 100 and method of the present invention will be described.

[0047] The general method described in this invention is a method for a localization pipeline 202 (which may include the weighted processing network 104 described herein). This method takes one or more images as input and outputs position and orientation (pose) labels after computation by the pipeline. For training purposes, the output position and orientation can be compared with known pose information of the input images, and the error between the two can be minimized to adjust the parameters of the pipeline so that the localization pipeline 202 outputs pose information with higher accuracy. To achieve this, the method of this invention utilizes novel weak supervision (e.g., non-user supervision) to train the parameters of the localization pipeline 202, which is formed as a trained weighted processing network 104. The weak supervision focuses on parameters of relative geometric constraints, such as relative pose labels between images.

[0048] Therefore, the proposed method can construct a set of relative geometric constraints obtained from neighboring images as well as distant images. Distant images are those that, in an image sequence, are spatially and / or temporally distant from their neighboring images. Examples of distant images are images taken by a camera located at another position in the scene, such as images taken in different corners of a room, possibly at different times of day. These constraints are used to help train the deep neural network 104 to better learn the geometry of the scene, thereby enabling global pose estimation based on a given image. Relative geometric constraints are defined as the relative pose between two poses 103.

[0049] One aspect of the present invention provides a training apparatus 100 for training a weighted processing network 104 to process images to determine camera positioning. The apparatus includes one or more processors for performing the following functions.

[0050] The apparatus and the one or more processors are configured to receive two or more sequences, which include multiple sequences of image pose pairs 201. As an example, two sequences of image pose pairs 201 are described herein. However, it should be understood that other sequences of image pose pairs 201 can be received and processed by the methods described below. Therefore, the one or more processors can be configured to receive image pose pairs 201 of a first sequence 101, 201, which include a first set of images and its associated first absolute pose set 103. The first sequence 101, 201 may include one or more image pose pairs 201, which include, but are not limited to, one or more images and the associated absolute pose set 103 (position and orientation of a camera / virtual camera within the scene) that generated the image. Each image pose pair 201 of the first sequence 101, 201 may include a single image and a single associated pose. The absolute pose 103 may refer to a pose known from the input image.

[0051] The one or more processors can also be configured to receive image pose pairs of a second sequence 101a, 201, which include a second set of images and an associated second set of absolute poses 103. The format and composition of the image pose pairs 201 of the second sequence 101a, 201 can be the same as those of the image pose pairs of the first sequence 101, 201; for example, each image pose pair can consist of an image and an absolute pose associated with that image.

[0052] To describe it another way, for a given scene with M acquired images and M corresponding absolute poses 103 (relative to global frames of the scene), the apparatus and method can be used to receive / select K sets of image sequences, each set containing N images. These may also include associated image poses. For each input image i during training, a first set of N images spatially and temporally close to the input image can be selected. The selection process can be based on the timestamps at the time of image acquisition. The selection process can be unrestricted by any order. Similarly, K–1 other sets (a second set or other sets) of N close images (the number of images can be the same as the number of images in the first set) can be randomly selected. The images in these K sequences do not share similar images. Each of these K sets can span a random point in the scene. Images in the same sequence may or may not share the same view. Images in the same sequence may not follow a temporal (time) order. Therefore, two or more sequences of image pose pairs 201 can be received or selected in the manner described above.

[0053] Upon receiving at least two sequences, one or more processors of the apparatus of the present invention can be used to: for one or more image pose pairs 201 in the first sequences 101, 201, iteratively determine one or more relative poses based on one or more first poses associated with a first image in the first sequences 101, 201, another pose associated with another image in the first sequences 101, 201, and poses associated with images in the second sequences 101a, 201. In other words, the apparatus and method of the present invention are used to calculate the relative pose between two absolute poses 103 associated with two input images, where the two absolute poses 103 are each part of an image pose pair. Figure 1 As can be seen, each absolute camera pose 101 from the image pose pairs of the first sequences 101, 201 can be iteratively compared with one or more other absolute camera poses from the image pose pairs within the first sequences 101, 201, and / or one or more other absolute camera poses from the image pose pairs of the second sequences 101a, 201. Thus, each absolute image pose from the first sequences 101, 201 can be compared with other absolute poses 103 in the same sequence that are spatially and temporally close, or with other absolute poses 103 from the second sequences 101a, 201 that are spatially and temporally distant from the iteration.

[0054] Therefore, the one or more processors can iteratively calculate the relative pose between two poses with indices i and j in the same sequence:

[0055] The pose T can be composed of a [3×3] rotation matrix R and a [3×1] translation vector t. In the definition below, the estimator is represented by a hat (^) symbol on top of the character (e.g.: The same character without the hat symbol is the ground truth (real) value.

[0056] The error in the relative pose of the rotation matrix and translation vector can be defined as follows:

[0057] Given the above, one definition of translation error can be:

[0058] Where ||……||2 represents l2 normalization, and r represents rotation. The rotation error can be defined as:

[0059] One or more processors and methods of the present invention can also create relative constraints (poses) between the image and the next image in the sequence (the image at the next index: i+1) during iterative determination of the one or more relative poses described above. Therefore, for a sequence of N images, N–1 relative constraints (poses) are created. Furthermore, iteratively determining the one or more relative poses can also involve applying the same process to the remaining K sequences (e.g., a second sequence or another sequence of image pose pairs). In mathematical notation, the error signal between the relative ground truth constraint and the estimated relative pose is defined as:

[0060] item Let represent the translation vector of the relative ground truth pose between consecutive cameras i and i+1 in sequence K. The same notation applies to the other terms in the equation. The terms with hats are the estimators discussed above.

[0061] Furthermore, as part of the iterative process for determining one or more relative poses discussed above, one or more processors can be used to: for an image at index i in any sequence (e.g., a first sequence, a second sequence, and / or another sequence), compute the relative pose of that image with respect to an image at index i in other sequences. This is possible because all sequences can have the same number of images as part of image pose pairs. Thus, as Figure 1 As shown, it can come from Figure 1 The images of the first sequence 101, 201 and from Figure 1 The images of the second sequences 101a and 201 are compared to compare absolute (ground truth) poses, thereby obtaining one or more relative poses as described above. When comparing image pose pairs across K sequences, the cross-sequence error can be defined using the following expression:

[0062] item Let represent the translation vector of the relative ground truth pose between two distant cameras with index i, one in sequence k and the other in sequence k+1. The same notation applies to the other terms in the equation.

[0063] In summary, at each iteration, a random image I is selected from the set of M images. Therefore, the image sequence can be created / received by one or more processors. Using this set of sequences, in the above example including the first and second sequences 101a and 201, the method is applied to create a set of relative poses from the available absolute (ground truth) poses of these images.

[0064] After one or more processors of the training apparatus 100 of the present invention have received two or more sequences and determined one or more relative poses, the one or more processors can be used to pass these sequences and relative poses to the weighted processing network 104. In this process, the one or more processors can be used to refine one or more parameters of the weighted processing network 104. This can be achieved by the one or more processors iteratively performing the following operations: calculating one or more estimated poses using the weighted processing network 104 based on one or more image pose pairs from first sequences 101, 201, one or more image pose pairs from second sequences 101a, 201, and the determined one or more relative poses.

[0065] In other words, the apparatus can be used to estimate the estimated pose for each image (with an associated absolute pose) by passing the created / received image sequences and the determined relative poses to a weighted processing network 104. Based on these estimated poses, the one or more processors can be used to create a set of estimated relative poses. This can be achieved by the one or more processors computing the estimated relative poses in the same manner as the relative poses discussed above, but the estimated pose information can be used as input instead of the absolute pose 103 for each image. This process can be performed iteratively for one or more images in each of two or more sequences (as part of an image pose pair).

[0066] After the weighted processing network 104 calculates the estimated relative pose based on the factors implemented on one or more processors of the apparatus of the present invention, the one or more processors can be used to: determine, for each calculated estimated pose, the error between one or more estimated poses and one or more relative poses; and update one or more parameters of the weighted processing network 104 based on the determined error. The error between one or more estimated poses of an image randomly selected from one of two or more sequences and the determined relative pose (geometric constraint) can be determined iteratively, such that the parameters of the weighted processing network 104 are iteratively updated and refined. In some cases, the error between poses can be based on the following exemplary expression:

[0067] The error signal determined between the one or more estimated poses and the relative pose can be calculated by the one or more processors. The determined error may include one or more of the following: a minimization error based on the difference between two or more image pose pairs in the first sequences 101, 201; and / or a minimization error based on the difference between one or more image pose pairs in the first sequences 101, 201 and their associated absolute and estimated values ​​and one or more image pose pairs in the second sequences 101a, 201.

[0068] Furthermore, as part of the iterative refinement of the parameters of the weighted processing network 104, the one or more processors can compute another error signal. This additional error signal can be the error signal between the estimated absolute pose of the input image I and its absolute ground truth pose. This additional error can also be used to update the parameters of the weighted processing network 104. These error signals are minimized in each iteration. In other words, the determined error can also be determined by: computing one or more errors between two or more image pose pairs in the first sequence 101, 201 and / or the second sequence 101a, 201 based on comparing the ground truth of each image pose pair with the estimated relative pose of each image pose pair; summing and minimizing the one or more computed errors. At the end of training (loss minimization), the network 104 learns geometric constraints and encodes these constraints in its parameters.

[0069] Figure 2 A general example process that can be implemented by one or more processors of the apparatus of the present invention is shown, wherein the one or more processors are used to fine-tune a trained weighted processing network 104 that forms part of a positioning pipeline 302. Specifically, the one or more processors can also be used to fine-tune the parameters of the weighted processing network 104. This can be achieved by inputting one or more image location pairs into the training apparatus 100, and in particular the trained weighted processing network 104. Each such image location pair 301 may include an image and its associated location information (e.g., GPS data recorded when the image was captured), which can be combined below. Figure 2 Image 301 is referred to as such. The one or more processors can utilize this data when computing the estimated pose, including location information, using the weighted processing network 104. The location information can be included as part of an image location pair. The one or more processors can then determine a fine-tuning error between the estimated pose 307 and the known image location pair, particularly the known location information. At the end of each iteration, one or more parameters of the weighted processing network 104 can be updated based on minimizing the determined fine-tuning error. These steps can be performed iteratively in a manner similar to the training described above, but with a lower learning rate, for example, smaller adjustments to the parameters of the weighted processing network 104 compared to the adjustments made during the training phase implemented by the one or more processors. The implemented fine-tuning can be performed in a small number of epochs (e.g., 10 to 15).

[0070] Furthermore, the localization pipeline that may include the weighted processing network 104 trained in the manner described above may also include the following as outputs of the weighted processing network 104: estimated 3D coordinates in the global frame, estimated 3D coordinates in the camera frame, and / or additional estimated weights associated with the 3D coordinate matrices of the coordinates in the global frame and the camera frame. These will be described below in conjunction with the inference device (image processing apparatus 400). However, it should be understood that these outputs of the weighted processing network 104 can form part of the training process, as these outputs of the weighted processing network 104 can be used to generate a final predicted (estimated) pose 107, 207 compared with the absolute pose of the input image, thereby training the weighted processing network to achieve a close relationship between the two quantities.

[0071] The training method described above, implemented in the apparatus of the present invention, particularly on one or more processors, provides the ability to train the weighted processing network 104 in a weakly supervised manner by using the relative poses between images as geometric constraints, without requiring additional user-input labels. Thus, the weighted processing network 104 of the present invention can optimize the localization pipeline using the relative poses between adjacent and distant cameras. These inputs for training the network 104 can be obtained only from the minimum available labels of the localization pipeline (e.g., absolute pose 103). Therefore, as described above, no 3D ground truth data is required to train the network 104. The above steps can be implemented by one or more processors of the training apparatus 100 of the present invention, or can be performed as a separate method.

[0072] This document also describes an image processing apparatus 400 for global camera repositioning. An example of such an image processing apparatus 400 is provided in… Figure 3 As shown in the image. Figure 3 As shown, the present invention includes an image processing method and apparatus, wherein the image processing apparatus 400 includes one or more processors, the one or more processors being configured to perform the following combinations Figures 3 to 7 The described function.

[0073] First, the one or more processors can be used to receive an input image 401 representing at least a portion of the scene. This image can be an image received from a camera or an image received electronically from another source. Then, the one or more processors are used to input the input image 401 into a trained weighted processing network 402. The trained weighted processing network 402 can be a weighted processing network 402 having the same properties as discussed above in conjunction with the training device 100 and can be trained in the same manner. Specifically, the trained weighted processing network can be trained using two sequences of input images discussed above in conjunction with the training device 100 and a determined relative pose of each input image. The trained weighted processing network may also include one or more parameters. The determined relative pose can be based on comparing one or more first poses associated with the first image in the first sequence of input image pose pairs 101, 201 described by the training device 100 with the following: another pose associated with another image in the first sequence 101, 201 and the pose associated with the image in the second sequence of input image pose pairs 101a, 201 described by the training device 100.

[0074] Although the image processing apparatus 400 described herein can also be used to perform the functions of the training apparatus 100 described above, this is not always the case, nor is it actually necessary. This is because in some cases, the image processing apparatus 400 can be used to utilize a trained weighted processing network that has already been trained through the process described above. In other words, such training can be performed by a separate device of the image processing apparatus 400, such as the training apparatus 100 described above.

[0075] Back Figure 3 The one or more processors can be used to: compute one or more of the following based on the input image using a trained weighted processing network: estimated image coordinates in a global frame of the scene, estimated image coordinates in a camera frame of the scene, and / or one or more weights associated with coordinates within the scene. This can be considered as an inference step of the trained weighted processing network implemented on the image processing device 400. The trained weighted processing network can be considered as a deep neural network for estimating three outputs, such as estimated 3D coordinates in a global frame, estimated 3D coordinates in a camera frame, and / or additional estimated weights associated with a 3D coordinate matrix of coordinates in the global frame and the camera frame.

[0076] These outputs, generated by one or more processors in a trained weighted processing network implemented on the image processing apparatus 400, can be used to estimate the pose of an input image for which no previously known pose information is available. The one or more processors are used to estimate the pose of the input image based on one or more of the following: estimated image coordinates in a global frame of the scene, estimated image coordinates in a camera frame of the scene, one or more weights and / or absolute image coordinates associated with coordinates within the scene. The image processing apparatus 400 of the present invention can use these variables to estimate the pose of the input image in various ways, as described below. Figures 4 to 7 Describe these methods. Figure 4 All potential methods for pose estimation are shown. Multiple of these methods can be used to estimate the pose of an input image, and the estimated poses can be compared to form a composite estimated pose.

[0077] In short, the process of estimating the pose of an input image is as follows. One technique is pose estimation via rigid alignment. In this case, a non-learned, parameter-free rigid alignment module can be used to estimate the pose in two ways. First, through a weighted rigid alignment method, such as... Figure 4 As shown, in addition to the weighting factors (weights) in the camera frames and global frames, the module (implemented on one or more processors) also acquires the 3D coordinates in both frames and estimates the pose. Secondly, an unweighted rigid alignment method is used, such as... Figure 6 As shown, the module (implemented on one or more processors) only takes the 3D coordinates in two frames and estimates the pose.

[0078] Alternative methods that can be implemented by one or more processors of the image processing device 400 may be as follows: Figure 7 The method shown is for pose estimation via perspective-n-point (PnP), where a module (implemented on one or more processors) acquires only the 3D coordinates and 2D pixels (easily obtained from the input image) in the global frame and estimates the pose via the perspective-n-point module in the random sample consensus (RANSAC) scheme.

[0079] The data volume in pose estimation described below can have the following characteristics: The input image can be a 1-channel (grayscale) or 3-channel (RGB) input. The input image can have the following form: 1×H×W (representing grayscale) or 3×H×W (representing RGB), where H and W define the height and width of the image, respectively. The 3D coordinates in the global frame can be a 3-channel output of size 3×H×W, the 3D coordinates in the camera frame can be a 3-channel output of size 3×H×W, the weights can be a 1-channel output of size 1×H×W, and the pose can be a 6-degrees-of-freedom (DoF) quantity. The 6 degrees of freedom can be represented as a 6×1 vector (3 for position, 3 for orientation) or a 7×1 vector (3 for position, 4 for orientation).

[0080] exist Figure 4 In this context, one or more processors are used to implement a trained weighted processing network (deep neural network) 402 to obtain the three quantities discussed above. In addition to the weighting factors in the two frames (camera and global), the weighted rigid alignment module 403 also obtains the 3D coordinates in the two frames and estimates the pose 407. In other words, one or more processors can be used to employ weighted alignment pose alignment 403 when estimating the pose 407 of the input image 401 by using one or more of the following: estimated image coordinates in the global frame of the scene, estimated image coordinates in the camera frame of the scene, and one or more weights associated with the coordinates within the scene.

[0081] Similarly, in some cases, the perspective n-point modules 404 and 704 in the RANSAC scheme can be obtained in combination (e.g.) Figure 4 and Figure 7 (As shown) 3D coordinates and 2D pixels 405, 705 in the global frame (easily obtained from the input image), and estimate poses 407, 707. In other words, one or more processors are also used to: when estimating the poses 407, 707 of the input images 401, 701, employ perceptual point alignment by estimating the poses 407, 707 of the input images 401, 701 using one or more of the following: estimated image coordinates in the global frame of the scene and 2D image coordinates 405, 705 of the input images 401, 701. Furthermore, in Figure 5 In some of the cases shown, one or more processors are also used to: use random sampling consistency techniques 404, 704 when estimating the poses 407, 707 of the input images 401, 701.

[0082] exist Figure 4In this process, weighted alignment pose estimation 403 can be combined with perceptual point alignment technology 404, allowing one or more processors to be used to generate pose 407 using each technology and combine poses 407 to form a final pose. Figure 7 In this approach, the pose 707 is determined using a unique perspective n-point module 704 from the RANSAC scheme, based on the 3D global coordinates output from the trained weighted processing network 702 when computing the input image 701. In this case, the weights and 3D coordinates from the camera frame are not used. The 3D coordinates in the global frame and the corresponding 2D pixels obtainable from the input image are used to estimate the pose 707.

[0083] exist Figure 5 In this process, one or more processors can be used to estimate the pose 507 solely through the weighted rigid alignment module 503. In other words, one or more processors can be used to employ weighted aligned pose estimation 503 by estimating the pose of the input image 501 using one or more of the following: estimated image coordinates in a global frame of the scene, estimated image coordinates in a camera frame of the scene, and one or more weights associated with the coordinates within the scene. Thus, the weights can measure how the 3D coordinates in the global frame and camera frame are applied on an RGB basis. Therefore, each pixel will have a weight associated with one of the pixel information from the global frame and camera frame, which can determine how the pixels in the pose 507 of the image were generated.

[0084] Figure 6 An example is shown in which one or more processors are used to: when estimating the pose of an input image 601, employ an unweighted aligned pose estimation 610 by estimating the pose of the input image 601 using one or more of the following: estimated image coordinates in a global frame of the scene and estimated image coordinates in a camera frame of the scene. This example does not consider weights that may be output from a trained weighted processing network 602, obtains 3D coordinates from the camera frame and the global frame, and combines the 3D coordinates to form a pose 607.

[0085] It should be understood that Figures 4 to 7 The input image, the trained weighted processing network, the 3D global coordinates and 3D camera coordinates, and the pose seen in the figures can be the same. The different reference numerals used for these different figures should not be construed as implying that these common aspects of the figures are different. Therefore, in Figures 4 to 7 In the figures, reference numerals ending with the same numbers are intended to indicate the same stage of processing implemented by one or more processors of the image processing apparatus 400 of the present invention. The differences between these figures lie in the techniques used to generate each corresponding pose.

[0086] In addition to the known advantages, the training apparatus 100, the corresponding method, the image processing apparatus 400, and the corresponding method described in this paper offer numerous advantages and utilize pose labels to create another set of supervisory signals (geometric constraints) without requiring additional user intervention. These labels are relative geometric constraints between adjacent and distant camera frames / images. These constraints described in this paper can be used to train the network to learn to estimate 3D coordinates and a set of weighting factors in global frames and camera frames, thus enabling pose estimation using geometric information instead of just regression. This improves the accuracy of the pose estimation.

[0087] Furthermore, this invention leverages the ease of data acquisition, readily obtaining pose information via inexpensive sensors (GPS, IMU, Wi-Fi signals) without the need to create 3D SFM models or use depth sensors such as LiDAR or laser scanners to capture 3D scenes. Since camera pose can be determined from a single image, the device is less expensive and has a wider range of applications.

[0088] Furthermore, this invention provides a method for fine-tuning a pre-trained network using only location labels. In some scenarios, only location information may be provided. This information can still be used to train the network to learn location and orientation, thereby saving costs.

[0089] Finally, this invention provides a technique for pose estimation using rigid alignment and a perspective n-point algorithm. Therefore, using the same output dataset, the method of this invention can estimate pose in three different ways using two algorithms. This improves the flexibility and accuracy of pose estimation. The 3D scene can also be represented using the weights / parameters of a weighted processing network (deep neural network). Compared to a 3D SFM model of the scene, the scene representation of this network is more compact than currently known scene representations, thus reducing the amount of memory used and consequently lowering memory requirements.

[0090] In this invention, only pose (position and orientation) is needed to train the model. Although the above method can refer to receiving image pose pairs, the image pose should be understood to include both the image and the associated pose, or, when the image pose pair is used to train a weighted processing network, only the pose. The pose can be obtained via GPS (position) and IMU (orientation). For indoor scenes, the position relative to a Wi-Fi router can also be obtained.

[0091] The apparatus and method of the present invention have many applications, some of which are shown below: Mobile phones / navigation / virtual reality; Crowdsourced data for a location / historical area / city obtained from a mobile phone (camera, GPS, IMU) (or from public image and metadata collections); The model is trained in the cloud based on this collected data, or further fine-tuned on new input data (which could also be just location labels). Location / navigation based on a single given image (phone); Navigate to a specific location in the city (e.g., a restaurant, a shop, or a museum). Navigate to a specific store within the shopping mall; Based on an image of a given location, list relevant information about that scene (augmented reality).

[0092] Another use case could be in the robotics / navigation domain, where image processing methods can be used as a global coarse localizer. One example of this is that if a mobile robot gets lost (deviates from its path), the proposed method can be used to quickly relocalize it based on a single image. Alternatively, the method could be used in the field of continuous learning: as the robot explores new territory, data collection can be initiated, and the data can then be sent to the cloud to train / fine-tune the localization pipeline.

[0093] The applicant hereby discloses each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, regardless of whether such features or combinations of features solve any problem disclosed herein; and without limiting the scope of the claims. The applicant notes that aspects of the invention may include any such individual feature or combination of features. In view of the foregoing description, those skilled in the art will appreciate that various modifications can be made within the scope of the invention.

Claims

1. An image processing apparatus for global camera repositioning, characterized in that, The image processing apparatus includes one or more processors, the one or more processors being used for: Receive an input image representing at least a portion of the scene; The input image is fed into a trained weighted processing network, wherein the trained weighted processing network is trained using two sequences of input images and a relative pose determined for each input image, and the trained weighted processing network includes one or more parameters. Based on the input image, one or more of the following are computed using the trained weighted processing network: Estimated image coordinates in the global frames of the scene; Estimated image coordinates in the camera frames of the scene; One or more weights associated with the coordinates within the scene; The pose of the input image is estimated based on one or more of the following: the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, and one or more weights and / or absolute image coordinates associated with the coordinates within the scene.

2. The image processing apparatus according to claim 1, characterized in that, The determined relative pose is based on comparing one or more first poses associated with the first image in the first sequence of image pose pairs with the following: another pose associated with another image in the first sequence and the pose associated with an image in the second sequence of image pose pairs.

3. The image processing apparatus according to claim 1 or 2, characterized in that, The one or more processors are configured to: when estimating the pose of the input image, employ weighted aligned pose estimation by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, and the one or more weights associated with the coordinates within the scene.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The one or more processors are configured to: when estimating the pose of the input image, employ unweighted aligned pose estimation by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene and the estimated image coordinates in the camera frame of the scene.

5. The image processing apparatus according to any one of claims 1 to 4, characterized in that, The one or more processors are further configured to: employ perceptual point alignment when estimating the pose of the input image by using one or more of the following to estimate the pose of the input image: the estimated image coordinates in the global frame of the scene and the 2D image coordinates of the input image.

6. The image processing apparatus according to claim 5, characterized in that, The one or more processors are further configured to: use random sampling consistency techniques when estimating the pose of the input image.

7. The image processing apparatus according to any one of claims 1 to 6, characterized in that, The one or more processors are configured to: when estimating the pose of the input image, Weighted aligned pose estimation is employed by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, and one or more weights associated with the coordinates within the scene; Perceptual point alignment is employed by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene and the 2D image coordinates of the input image.

8. The image processing apparatus according to any one of claims 1 to 7, characterized in that, The trained weighted processing network is trained in the following manner: Receive one or more image pose pairs of a first sequence and one or more image pose pairs of a second sequence; For one or more image pose pairs in the image pose pairs, one or more relative poses are determined based on the one or more image pose pairs in the first sequence and the one or more image pose pairs in the second sequence; Based on the determined one or more relative poses, a weighted processing network is used to iteratively calculate one or more estimated poses; For each calculated estimated pose, determine the error between the one or more estimated poses and the one or more relative poses; The weighted processing network is updated based on the determined error.

9. The image processing apparatus according to claim 8, characterized in that, The one or more processors are used to train the parameters of the trained weighted processing network in the following manner: Receive two or more sequences, wherein the two or more sequences include image pose pairs of a first sequence, the image pose pairs of the first sequence including a first set of images and its associated first absolute pose set, and the two or more sequences also include image pose pairs of a second sequence, the image pose pairs of the second sequence including a second set of images and its associated second absolute pose set; For the one or more image pose pairs in the first sequence, one or more relative poses are iteratively determined by comparing one or more first poses associated with the first image in the first sequence with the following: another pose associated with another image in the first sequence and a pose associated with an image in the second sequence; Refine one or more parameters of the weighted processing network by iteratively performing the following operations: Based on the one or more image pose pairs from the first sequence, the one or more image pose pairs from the second sequence, and the determined one or more relative poses, the weighted processing network is used to compute one or more estimated poses. For each calculated estimated pose, determine the error between the one or more estimated poses and the one or more relative poses; The weighted processing network is updated based on the determined error.

10. The image processing apparatus according to any one of claims 1 to 9, characterized in that, The parameters of the weighted processing network are fine-tuned iteratively by performing the following operations: One or more image location pairs are input into the image processing device; Calculate the estimated pose, including position information; Determine the fine-tuning error between the estimated pose and the known image position pair; The weighted processing network is updated based on the minimization of the fine-tuning error.

11. An image processing method for global camera repositioning, characterized in that, The method includes: Receive an input image representing at least a portion of the scene; The input image is fed into a trained weighted processing network, wherein the trained weighted processing network is trained using two sequences of input images and a relative pose determined for each input image, and the trained weighted processing network includes one or more parameters. Based on the input image, one or more of the following are computed using the trained weighted processing network: Estimated image coordinates in the global frames of the scene; Estimated image coordinates in the camera frames of the scene; One or more weights associated with the coordinates within the scene; The pose of the input image is estimated based on one or more of the following: the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, and one or more weights and / or absolute image coordinates associated with the coordinates within the scene.

12. The data processing method according to claim 11, characterized in that, The determined relative pose is based on comparing one or more first poses associated with the first image in the first sequence with the following: another pose associated with another image in the first sequence and the pose associated with an image in the second sequence.

13. The data processing method according to claim 11 or 12, characterized in that, Estimating the pose of the input image includes: Weighted aligned pose estimation is employed by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, and one or more weights associated with the coordinates within the scene.

14. The data processing method according to any one of claims 11 to 13, characterized in that, Estimating the pose of the input image includes: Unweighted aligned pose estimation is performed by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene and the estimated image coordinates in the camera frame of the scene.

15. The data processing method according to any one of claims 11 to 14, characterized in that, Estimating the pose of the input image includes: Perceptual point alignment is employed by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene and the 2D image coordinates of the input image.

16. The data processing method according to claim 15, characterized in that, Estimating the pose of the input image includes: Use random sampling consistency techniques.

17. The data processing method according to any one of claims 11 to 16, characterized in that, Estimating the pose of the input image includes: Weighted aligned pose estimation is employed by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene, the estimated image coordinates in the camera frame of the scene, and one or more weights associated with the coordinates within the scene; Perceptual point alignment is employed by estimating the pose of the input image using one or more of the following: the estimated image coordinates in the global frame of the scene and the 2D image coordinates of the input image.

18. The data processing method according to any one of claims 11 to 17, characterized in that, The trained weighted processing network is trained in the following manner: Receive two or more sequences, wherein the two or more sequences include one or more image pose pairs of a first sequence, and the two or more sequences also include one or more image pose pairs of a second sequence; For one or more image pose pairs in the image pose pairs, one or more relative poses are determined based on the one or more image pose pairs in the first sequence and the one or more image pose pairs in the second sequence; Based on the determined one or more relative poses, a weighted processing network is used to iteratively calculate one or more estimated poses; For each calculated estimated pose, determine the error between the one or more estimated poses and the one or more relative poses; The weighted processing network is updated based on the determined error.

19. The data processing method according to claim 18, characterized in that, The trained weighted processing network is trained in the following manner: Receive the image pose pair of the first sequence, wherein the image pose pair includes a first set of images and its associated first absolute pose set; Receive the image pose pair of the second sequence, wherein the image pose pair includes a second set of images and its associated second absolute pose set; For the one or more image pose pairs in the first sequence, the one or more relative poses are iteratively determined by comparing one or more first poses associated with the first image in the first sequence with the following: another pose associated with another image in the first sequence and a pose associated with an image in the second sequence; Refine one or more parameters of the weighted processing network by iteratively performing the following operations: Based on the one or more image pose pairs from the first sequence, the one or more image pose pairs from the second sequence, and the determined one or more relative poses, the weighted processing network is used to compute the one or more estimated poses; For each calculated estimated pose, determine the error between the one or more estimated poses and the one or more relative poses; The weighted processing network is updated based on the determined error.

20. The data processing method according to any one of claims 11 to 19, characterized in that, The method also includes fine-tuning the trained weighted processing network iteratively by performing the following operations: One or more image location pairs are input into the image processing device; Calculate the estimated pose, including position information; Determine the fine-tuning error between the estimated pose and the known image position pair; The weighted processing network is updated based on the minimization of the fine-tuning error.

21. A training apparatus for training a weighted processing network to process images to determine camera positioning, characterized in that, The device includes one or more processors, the one or more processors being used for: Receive two or more sequences, wherein the two or more sequences include image pose pairs of a first sequence, the image pose pairs of the first sequence including a first set of images and its associated first absolute pose set, and the two or more sequences also include image pose pairs of a second sequence, the image pose pairs of the second sequence including a second set of images and its associated second absolute pose set; For the one or more image pose pairs in the first sequence, one or more relative poses are iteratively determined based on one or more first poses associated with the first image in the first sequence, another pose associated with another image in the first sequence, and a pose associated with an image in the second sequence. Refine one or more parameters of the weighted processing network by iteratively performing the following operations: Based on the one or more image pose pairs from the first sequence, the one or more image pose pairs from the second sequence, and the determined one or more relative poses, the weighted processing network is used to compute one or more estimated poses. For each calculated estimated pose, determine the error between the one or more estimated poses and the one or more relative poses; The weighted processing network is updated based on the determined error.

22. The training device according to claim 21, characterized in that, The determined error includes one or more of the following: Based on minimizing the error of the difference between the two or more image pose pairs in the first sequence; or The error is minimized based on the difference between one or more image pose pairs in the first sequence and their associated absolute and estimated values ​​and one or more image pose pairs in the second sequence.

23. The training device according to claim 22, characterized in that, The determined error is further determined in the following way: One or more errors between two or more image pose pairs in the first sequence and / or the second sequence are calculated by comparing the ground truth of each image pose pair with the estimated relative pose of each image pose pair. Summate and minimize the one or more calculation errors.

24. The training apparatus according to any one of claims 21 to 23, characterized in that, The one or more image pose pairs of the first sequence are spatially and temporally distant from the one or more image pose pairs of the second sequence.

25. The training apparatus according to any one of claims 21 to 24, characterized in that, The one or more processors are configured to: determine the one or more relative poses of the image pose pairs by comparing the pose of the image pose pair in the first sequence and / or the second sequence with the other pose of one or more other image pose pairs.

26. The training apparatus according to any one of claims 21 to 25, characterized in that, The one or more processors are also configured to fine-tune the parameters of the weighted processing network iteratively by performing the following operations: One or more image location pairs are input into the image processing device; Calculate the estimated pose, including position information; Determine the fine-tuning error between the estimated pose and the known image position pair; The weighted processing network is updated based on the minimization of the fine-tuning error.

27. A method for training a weighted processing network to process images to determine camera location, characterized in that, The method includes: Receive two or more sequences, wherein the two or more sequences include image pose pairs of a first sequence, the image pose pairs of the first sequence including a first set of images and its associated first absolute pose set, and the two or more sequences also include image pose pairs of a second sequence, the image pose pairs of the second sequence including a second set of images and its associated second absolute pose set; For the one or more image pose pairs in the first sequence, one or more relative poses are iteratively determined based on one or more first poses associated with the first image in the first sequence, another pose associated with another image in the first sequence, and a pose associated with an image in the second sequence. Refine one or more parameters of the weighted processing network by iteratively performing the following operations: Based on the one or more image pose pairs from the first sequence, the one or more image pose pairs from the second sequence, and the determined one or more relative poses, the weighted processing network is used to compute one or more estimated poses. For each calculated estimated pose, determine the error between the one or more estimated poses and the one or more relative poses; The weighted processing network is updated based on the determined error.

28. The method according to claim 27, characterized in that, The determined error includes one or more of the following: Based on minimizing the error of the difference between the two or more image pose pairs in the first sequence; or The error is minimized based on the difference between one or more image pose pairs in the first sequence and their associated absolute and estimated values ​​and one or more image pose pairs in the second sequence.

29. The method according to claim 28, characterized in that, The determined error is further determined in the following way: One or more errors between two or more image pose pairs in the first sequence and / or the second sequence are calculated by comparing the ground truth of each image pose pair with the estimated relative pose of each image pose pair. Summate and minimize the one or more calculation errors.

30. The method according to any one of claims 27 to 29, characterized in that, The one or more image pose pairs of the first sequence are spatially and temporally distant from the one or more image pose pairs of the second sequence.

31. The method according to any one of claims 27 to 30, characterized in that, The one or more processors are configured to: determine the one or more relative poses of the image pose pairs by comparing the pose of the image pose pair in the first sequence and / or the second sequence with the other pose of one or more other image pose pairs.

32. The method according to any one of claims 27 to 31, characterized in that, The method further includes fine-tuning the parameters of the weighted processing network by iteratively performing the following operations: One or more image location pairs are input into the image processing device; Calculate the estimated pose, including position information; Determine the fine-tuning error between the estimated pose and the known image position pair; The weighted processing network is updated based on the minimization of the fine-tuning error.

Images