Six-DOF camera pose estimation system based on learning scene general features and specific features

Abstract

A six-degree-of-freedom camera pose estimation system based on learning scene common features and specific features belongs to the field of visual positioning technology. The present invention addresses the problems of mutual interference between scene shared parameters and scene specific parameters in existing visual positioning and gradient conflicts during training. It includes: a basic network module, which uses the shared parameters and specific parameters obtained through training to estimate the nth task scene image I captured by the six-degree-of-freedom camera. n Perform feature extraction to obtain image extraction feature F n ; shared parameters φ and specific parameters φ n Calculated using the average gradient algorithm; regression layer: extract feature F from the image n Make a prediction to get the 3D scene coordinate D n and 1D uncertainty U n ; Posture estimation module: using PnP algorithm, based on 3D scene coordinates D n and 1D uncertainty U n Calculate the rotation matrix R of the six-degree-of-freedom camera n and the translation vector t n , thereby determining the camera pose T in the nth task scene n The present invention is used for camera pose estimation.

Description

Technical Field

[0001] The present invention relates to a six-degree-of-freedom camera pose estimation system based on general features and specific features of a learning scene, and belongs to the technical field of visual positioning. Background Art

[0002] Given an input image and visual localization, inferring the 6DOF camera pose is a key component of computer vision tasks such as simultaneous localization and mapping, virtual reality, and structure from motion.

[0003] There are two main mainstream approaches to visual localization: structure-based methods and deep learning methods. The classic structure-based visual localization framework builds an association between 2D keypoints and 3D scene coordinates by matching local descriptors, and then uses a RANSAC-based pass-and-pass (PnP) algorithm to retrieve the 6DOF camera pose. Advances in deep learning have led to three main categories in visual localization: absolute pose regression (APR), relative pose regression (RPR), and scene coordinate regression (SCoRe). SCoRe stands out for its accuracy and concise model representation. It uses a convolutional neural network (CNN) to regress 3D scene coordinates and a pass-and-pass (PnP) algorithm for camera pose estimation. Although these methods are effective, they face parameter count challenges. Specifically, the scene-specific nature of SCoRe requires separate training for each new scene, resulting in a model size that increases proportionally with the number of scenes, which may hinder the network's efficiency in a variety of scenes.

[0004] Most existing multi-task learning visual positioning technologies use an adaptive shared basis and multiple specific regression layers to achieve unified positioning for multiple separate scenes. To achieve flexible parameter sharing, the existing technology proposes a layer adaptive sharing strategy that relaxes the combination problem into a learnable problem and automatically determines whether each layer in the active layer is shared for different scenes. In addition, to solve the gradient conflict problem, the existing technology proposes a gradient normalization algorithm that makes the gradient amplitude of the scene shared parameters uniform so that all tasks converge at the same speed. Although the multi-task learning technology described above can be deployed into the visual positioning framework to work effectively, it still has the following inherent problems:

[0005] (1) The key to the existing technology is to identify scene-shared parameters that encode common scene properties across scenes while maintaining storage efficiency, and to learn scene-related features and determine scene-specific parameters to maintain positioning performance. In theory, scene-shared parameters represent a synthesis of all scenes and are affected by all scenes; while scene-specific parameters extract unique features related to each scene and are therefore only affected by their respective scenes. Therefore, optimizing scene-shared parameters and scene-specific parameters in a single stage inevitably leads to interference between them during the optimization process. This means that when optimizing scene-shared parameters, scene-specific parameters will also be inadvertently adjusted, preventing the network from achieving the optimal configuration of both parameters simultaneously.

[0006] (2) For a given active layer in all scenes, existing techniques introduce a learnable score to determine the shared pattern of multiple corresponding active layers. While this approach is effective, it assigns the same score to active layers at the same location, which significantly limits the flexibility of the shared pattern. Since specific parameter values ​​are customized to extract the unique features of each scene, the active layers at the corresponding locations are processed using the same scene-specific parameters, ignoring the different requirements of each individual scene.

[0007] Structure-based visual positioning:

[0008] Structure-based methods determine the correspondence between 2D pixel locations and 3D scene coordinates given an image query, often accompanied by a PnP algorithm to estimate the 6-DOF pose. Recent advances in this field have introduced image retrieval to reduce the search space. State-of-the-art feature matching methods, such as SuperGlue, have been further employed to establish reliable 2D-3D matches, resulting in accurate 2D-3D correspondences. HLoc utilizes multi-task distillation to extract global descriptors and dense local descriptors, enabling global retrieval and local localization at a manageable computational cost. Inloc utilizes NetVLAD for global retrieval and multi-scale features extracted by a CNN for local feature matching in a coarse-to-fine manner. Scenesqueezer reduces the number of point cloud and feature dimensions to compress the scene map, significantly reducing the memory required to store the map. The integration of efficient image retrieval and feature matching techniques significantly improves the applicability and robustness of structured methods for large scenes. Experiments demonstrate that structure-based methods achieve excellent localization performance in large scenes and demonstrate impressive generalization to unseen scenes. However, they consume a lot of memory space to store pre-built 3D scene models and corresponding descriptors, making them costly to deploy on resource-constrained devices.

[0009] Learning-based visual localization:

[0010] Learning-based visual localization methods can generally be categorized into relative pose regression (RPR), absolute pose regression (APR), and scene coordinate regression (SCoRe) methods. RPR methods determine the camera's 3D pose by learning the similarity between a query image and a database of images, which is time-consuming. APR methods offer a simple yet effective end-to-end pipeline for directly predicting the camera pose, however, their localization accuracy is poor. SCoRe methods directly regress the 2D-3D correspondences between pixels in an image and then compute the 3D camera pose using a PnP algorithm. Previous work has used random forests to regress pixel-level 3D coordinates. While these methods have achieved promising results, they have proven to be highly image-dependent and difficult to adapt to complex environments. The mainstream SCoRe method uses CNNs for more accurate coordinate regression. KFNet utilizes a Kalman filter to integrate temporal information, achieving accurate localization. HSCNet designs a hierarchical network structure to regress scene coordinates. DSAC* utilizes a neural network to establish dense correspondences between images and 3D coordinates and applies the DSAC algorithm for robust model fitting. VSNet regresses the pixel coordinates of selected landmarks to eliminate the influence of poor-quality scene coordinates. EAAINet simultaneously extracts absolute and relative features of the image to mitigate ambiguity caused by similar image patches. While these methods can be optimized end-to-end and achieve impressive results, they also suffer from several drawbacks. Both pose regression and scene coordinate regression are scene-specific and must be retrained for new scenes, resulting in a linear increase in total model size as the number of scenes increases.

[0011] Gradient Normalization in Multi-Task Learning:

[0012] During the training process of multi-task learning (MTL), the gradient magnitudes and directions of different tasks interact with each other in a complex manner through back-propagation. This phenomenon is called task interference. Previous methods simplified the problem into two types of gradient differences (i.e., the magnitude and direction of the task gradient), and proposed various techniques to reconcile this difference. For the gradient magnitude, there have been endless approaches in recent years, such as describing multi-task learning as multi-objective optimization and providing an upper limit for multi-objective loss. Or the gradient magnitude is uniformed by normalization and scaling to ensure training convergence. For the gradient direction, it is currently more common to make the task gradients statistically indistinguishable through adversarial training. According to PCGrad, if there is a conflict in the task gradients, the projection of one task gradient on the other task gradient is abandoned. Summary of the Invention

[0013] To address the problems of mutual interference between scene-shared parameters and scene-specific parameters in existing visual positioning and gradient conflicts during training, the present invention provides a six-degree-of-freedom camera pose estimation system based on learning scene common features and specific features.

[0014] The present invention provides a six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features, comprising:

[0015] Basic network module, based on the shared parameters φ and specific parameters φ obtained through training n The nth task scene image I captured by the six-degree-of-freedom camera n Perform feature extraction to obtain image extraction feature F n ; The shared parameter φ and the specific parameter φ n Calculated using the average gradient algorithm; n = 1, 2, ..., N;

[0016] Regression layer: extract features F from the image n Make a prediction to get the 3D scene coordinate D n and 1D uncertainty U n ;

[0017] Posture estimation module: using PnP algorithm, based on 3D scene coordinates D n and 1D uncertainty U n Calculate the rotation matrix R of the six-degree-of-freedom camera n and the translation vector t n , thereby determining the camera pose T in the nth task scene n ;

[0018] After obtaining the image feature F n The process includes the task scene image I n The general feature extraction stage and the specific feature extraction stage are as follows: in the general feature extraction stage, all currently active layers in the basic network module share all shared parameters φ, and the gradient normalization algorithm is used to optimize the shared parameters φ; in the specific feature extraction stage, the optimized shared parameters φ are fixed, and the specific parameters φ of all currently active layers in the basic network module are optimized. n Optimize;

[0019] During the training process of the pose estimation system, based on the 3D scene coordinates D n and 1D uncertainty U n The loss function adjusts the shared parameters φ and the specific parameters φ n ;

[0020] In the shared parameter φ and the specific parameter φ nDuring the calculation process, the binary score of the corresponding parameter is determined based on the learnable score; the learnable score is optimized based on the gradient.

[0021] According to the six-degree-of-freedom camera pose estimation system based on the general and specific features of the learning scene of the present invention, the basic network module includes a preprocessing layer with a step size of 2 and four residual layers; the shared parameter φ spans N task scenes; the specific parameter φ n Corresponding to the nth task scenario;

[0022] Image extraction feature F n for:

[0023]

[0024] Where f represents the basic network module, is a set of real numbers, C o F n The channel dimension;

[0025] H o =H / 8,W o =W / 8,

[0026] Where H is the length of the task scene image, and W is the width of the task scene image;

[0027] φ∈X sh ,φ n ∈χ sp ,

[0028] χ sh is the set of shared parameters of N task scenarios obtained through iteration, χ sp is a set of N task scenario specific parameters obtained through iteration.

[0029] According to the six-degree-of-freedom camera pose estimation system based on the general and specific features of the learning scene of the present invention, it is assumed that the regression layer of the nth task scene is h, and its parameter θ n ∈X sp ,but:

[0030] D n ,U n =h(F n θ n ),

[0031] D n ={d n,i =(x n,i ,y n,i ,z n,i )|i=1,2,3,…,P},

[0032] U n ={un,i |i=1,2,3,…,P},

[0033] Where d n,i is the 3D scene coordinate D n The i-th value of (x n,i ,y n,i ,z n,i ) is d n,i The three-dimensional coordinates of u n,i is the 1D uncertainty U n The i-th value of ; P is the number of 3D scene coordinates.

[0034] According to the six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features of the present invention, the camera pose T in the nth task scene n for:

[0035]

[0036] Where p represents the pose estimation module.

[0037] According to the six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features of the present invention, the shared parameter φ and the specific parameter φ n The calculation method is:

[0038]

[0039]

[0040]

[0041] In the formula Shared parameters for the nth task scenario, Specific parameters for the nth task scenario; is the I-th iteration result of the n-th task scenario specific parameters, v is the learning rate, L n,I is the loss of the nth task scenario in the Ith iteration; Share parameters for the nth task scenario.

[0042] According to the six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features of the present invention, the optimization method of the shared parameter φ is:

[0043] For the nth task scenario, assume that the eth convolutional layer in the module consisting of the basic network module and the regression layer is the active layer, and set the initial universal weight of the eth convolutional layer Among them C out is the number of output channels, C in is the number of input channels, k is the convolution kernel size, and is shared in the common feature extraction stage:

[0044] In the general feature extraction stage:

[0045] T n =p(h(f(I n ;φ,φ n );θ n )),

[0046]

[0047] Where M is the number of active layers, L is the number of inactive layers, is the universal weight of the lth inactive layer.

[0048] According to the six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features of the present invention, in the specific feature extraction stage:

[0049] Assume that the specific weight of the e-th convolutional layer in the module consisting of the base network module and the regression layer is Learnable fractions In the forward propagation, define the indicator function To determine whether to add scene-specific weights to the current convolutional layer in the current iteration:

[0050]

[0051] Where λ is the preset threshold;

[0052] If the indicator function Then add scene-specific weights to the current convolutional layer Get the adaptive weight of the task in the current iteration

[0053]

[0054] Otherwise, only the general weights are considered

[0055] In the specific feature extraction stage:

[0056]

[0057] According to the six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features of the present invention, the loss function is expressed as L n :

[0058]

[0059] In the formula For the corresponding d n,i The true value of .

[0060] Beneficial Effects of the Invention: The system of the present invention provides a unified visual localization framework for scene-general and scene-specific features. It can decompose localization for multiple scenes into two stages, maximizing the effectiveness of scene-shared parameters and scene-specific parameters, thereby achieving high-precision and efficient visual localization across multiple scenes. In the general feature extraction stage, parameters from all active layers in the backbone are shared to effectively extract comprehensive scene-general features. In the specific feature optimization stage, scene-specific parameters are dynamically expanded based on the optimized shared weights from the general feature extraction stage, providing the model with greater flexibility and richer scene-related information. In addition, the present invention also improves the scoring mechanism, meticulously evaluating the unique scene-specific parameter requirements of each individual scene, enabling comprehensive learning of scene-related features. The clear separation in optimizing scene-shared and scene-specific parameters ensures that the scene-shared and scene-specific parameters do not interfere with each other. In addition, to mitigate gradient conflicts, a gradient normalization algorithm is adopted to uniformly adjust the gradient amplitudes of scene-shared parameters across scenes, ensuring that all tasks converge at similar but optimal speeds. The two-stage optimization strategy maximizes the encoding effect of scene-general and scene-specific attributes while resolving the problem of parameter interleaving.

[0061] This paper considers visual localization of multiple scenes as a multi-task learning task, and optimizes multiple scenes simultaneously through multi-task learning. By sharing some parameters, the parameter utilization rate is significantly improved, thus achieving efficient localization.

[0062] The present invention adopts a gradient normalization algorithm to reweight the task gradients at each learning step to eliminate the difference in gradient magnitude, thereby ensuring that all tasks converge at the same speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 1 is a block diagram of a six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to the present invention;

[0064] Figure 2 It is a comparison diagram of the training process of shared parameters and non-shared parameters (specific parameters) of the existing method and the method of the present invention. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0066] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0067] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0068] Specific implementation method 1. Combination Figure 1 and Figure 2 As shown, the present invention provides a six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features, including:

[0069] Basic network module, based on the shared parameters φ and specific parameters φ obtained through training n The nth task scene image I captured by the six-degree-of-freedom camera n Perform feature extraction to obtain image extraction feature F n ; The shared parameter φ and the specific parameter φ n Calculated using the average gradient algorithm; n = 1, 2, ..., N;

[0070] Regression layer: extract features F from the image n Make a prediction to get the 3D scene coordinate D n and 1D uncertainty U n ;

[0071] Posture estimation module: adopts the PnP algorithm based on RANSAC, based on the 3D scene coordinates D n and 1D uncertainty U n Calculate the rotation matrix R of the six-degree-of-freedom camera n and the translation vector t n , thereby determining the camera pose T in the nth task scene n ;

[0072] After obtaining the image feature F n The process includes the task scene image I n The general feature extraction stage and the specific feature extraction stage are as follows: in the general feature extraction stage, all currently active layers in the basic network module share all shared parameters φ, and the gradient normalization algorithm is used to optimize the shared parameters φ; in the specific feature extraction stage, the optimized shared parameters φ are fixed, and the specific parameters c of all currently active layers in the basic network module are optimized. n Optimize;

[0073] During the training process of the pose estimation system, based on the 3D scene coordinates D n and 1D uncertainty U n The loss function adjusts the shared parameters φ and the specific parameters φ n ;

[0074] In the shared parameter φ and the specific parameter φ n During the calculation process, the binary score of the corresponding parameter is determined based on the learnable score; the learnable score is optimized based on the gradient.

[0075] In this embodiment, the task scene image I n For RGB images, the visual localization task aims to estimate the rigid transformation from the camera coordinate system to the world coordinate system. Such a transformation consists of a 3D rotation matrix R and a translation vector t.

[0076] Furthermore, the basic network module includes a preprocessing layer with a step size of 2 and four residual layers; the shared parameter φ spans N task scenarios; the specific parameter φ n Corresponding to the nth task scenario;

[0077] Image extraction feature F n for:

[0078]

[0079] Where f represents the basic network module, is a set of real numbers, C o F n The channel dimension;

[0080] H o =H / 8,W o =W / 8,

[0081] Where H is the length of the task scene image, and W is the width of the task scene image;

[0082] φ∈χ sh ,φ n ∈χ sp

[0083] χ sh is the set of shared parameters of N task scenarios obtained through iteration, χ sp is a set of N task scenario specific parameters obtained through iteration.

[0084] In this implementation, the base network first maps the input image to a higher dimension and lower resolution using a preprocessing layer with a stride of 2, and then uses four residual layers ResBlocks (with strides of 1, 1, 2, 2) to extract features.

[0085] It is assumed that the layers with scene-shared parameters in the base network are active layers, such as convolution and normalization layers, while other layers, such as 1×1 convolution layers, ReLU layers, and Sigmoid layers, are considered inactive layers.

[0086] In this embodiment, it is assumed that the regression layer of the nth task scene is h, and its parameter θ n ∈χ sp , which is F n As input, and predict the 3D scene coordinates D n and the uncertainty U of 1D n ,but:

[0087] D n ,U n =h(F n θ n ),

[0088] D n ={d n,i =(x n,i ,y n,i ,z n,i )|i=1,2,3,…,P},

[0089] U n ={u n,i |i=1,2,3,…,P},

[0090] Where d n,i is the 3D scene coordinate D n The i-th value of (x n,i ,y n,i ,z n,i ) is d n,i The three-dimensional coordinates of u n,i is the 1D uncertainty U n The i-th value of ; P is the number of 3D scene coordinates.

[0091] Considering that there are no learnable parameters involved in pose estimation, the camera pose T in the nth task scene is n for:

[0092]

[0093] Where p represents the posture estimation module. n ∈r 3×3 , t n ∈R 3 .

[0094] Furthermore, existing multi-task learning visual localization technology uses scene-shared parameters for general feature extraction and scene-specific parameters for scene-related feature derivation. Although it has good performance, it optimizes both scene-shared parameters and scene-specific parameters at the same time, which inevitably leads to the mutual influence of the two parameters. This embodiment replaces the gradient normalization algorithm with the average gradient. The shared parameter φ and the specific parameter φ n The calculation method is:

[0095]

[0096]

[0097]

[0098] In the formula Shared parameters for the nth task scenario, Specific parameters for the nth task scenario; is the I-th iteration result of the n-th task scenario specific parameters, v is the learning rate, L n,I is the loss of the nth task scenario in the Ith iteration; The present invention uses X to share parameters for the nth task scenario. sh and X sp Represents the network parameters. The network parameters determined after training are shared parameters φ and specific parameters φ n .

[0099] As shown in the above formula, the shared parameters By all N individual scene-specific parameters Coupling training within one stage is bound to lead to entanglement of scene-shared parameters and scene-specific parameters, so this implementation adopts a separate two-stage framework to maximize the optimization of scene-shared and scene-specific parameters, thereby improving localization performance.

[0100] Two-stage optimization strategy:

[0101] Existing multi-task learning visual localization techniques optimize both scene-shared and scene-specific parameters in a single phase, leading to severe parameter aliasing and ultimately suboptimal results. This implementation breaks down the visual localization task into two phases: a general feature extraction phase and a specific feature optimization phase. By optimizing scene-shared parameters in the general feature extraction phase and then independently supplementing and optimizing scene-specific parameters in the specific feature optimization phase, both phases achieve optimal task-shared and scene-specific features.

[0102] Scene-shared parameters represent common information across scenes while reducing the number of parameters. This is particularly true for datasets rich in task-wide information, such as LIVL. Considering the mutual influence between scene-shared and scene-specific parameters, the first stage shares parameters in all active layers and optimizes them using a gradient normalization algorithm. Parameters in inactive and regression layers, on the other hand, are specific.

[0103] The optimization method of the shared parameter φ is:

[0104] For the nth task scenario, I n It is processed by the basic network and regression layer in turn. Assuming that the basic network module and

[0105] The e-th convolutional layer in the module composed of regression layers is the active layer, and the initial universal weight of the e-th convolutional layer is set Among them C out is the number of output channels, C in is the number of input channels, k is the convolution kernel size, and is shared in the common feature extraction stage to maximize the common feature extraction of the task; technically, the gradient normalization algorithm is used to ensure the consistency of the shared parameters of the scene:

[0106] During the general feature extraction phase, all scene-specific parameters in the base network come from inactive layers:

[0107] T n =p(h(f(I n ;φ,φ n );θ n )),

[0108]

[0109] Where M is the number of active layers, L is the number of inactive layers, is the universal weight of the lth inactive layer.

[0110] Going further, in the specific feature extraction stage:

[0111] Scene-specific parameters can learn features unique to the corresponding scene, thereby improving positioning performance. In order to derive scene-specific features, in the specific feature extraction stage, we focus on supplementing and optimizing scene-specific parameters for the model.

[0112] After parameter optimization in the general feature extraction stage, the optimized scene-shared parameters in the basic network across N scenes are obtained. Based on the optimized scene-shared parameters, this implementation proposes a dedicated feature-specific optimization phase for φ, automatically extending and optimizing the scene-specific parameters. This provides the model with greater flexibility and richer scene-related information with minimal parameter increase. Unlike existing multi-task learning visual localization techniques, this implementation fixes the scene-shared parameters optimized in the general feature extraction phase to avoid interference with the scene-shared parameters and maximize the learning of scene-specific features.

[0113] For the nth scene, taking the eth convolutional layer as an example, in addition to the fixed scene sharing weights in the active layer In addition, two additional parameters are added: the specific weight of the e-th convolutional layer in the module consisting of the basic network module and the regression layer Learnable fractions In the forward propagation, define the indicator function To determine whether to add scene-specific weights to the current convolutional layer in the current iteration:

[0114]

[0115] Where λ is the preset threshold;

[0116] If the indicator function Then add scene-specific weights to the current convolutional layer Get the adaptive weight of the task in the current iteration

[0117]

[0118] Otherwise, only the general weights are considered

[0119] Apply the above process to all active layers to achieve explicit scene-specific feature extraction. In the specific feature extraction stage:

[0120]

[0121] In the second stage, instead of optimizing any scene-shared parameters, all scene-specific parameters are optimized, e.g., φ n ,θ n .

[0122] In this implementation, the loss function proposed by KFNet is used to maximize the log-likelihood of the predicted scene coordinate probability density function. The loss function is expressed as L n :

[0123]

[0124] In the formula For the corresponding d n,i The true value of .

[0125] Gradient Optimization:

[0126] Since each task has its own dataset domain, multiple GPUs are needed to optimize these tasks. For the convenience of description, it is assumed that a single GPU is used to train the corresponding scene. In addition, the scene-shared and scene-specific parameters of the nth scene are denoted as χ sh and

[0127] Will Gradient Defined as:

[0128]

[0129] The scene-specific parameters on each GPU will be calculated based on When using multiple GPUs to optimize a scene, the gradients on each GPU will be averaged and the parameters will be updated accordingly.

[0130] Will Gradient Defined as:

[0131]

[0132] The optimization method for shared parameter gradients is:

[0133] For scene-shared parameters, a gradient normalization algorithm is used to make the gradient magnitudes of scene-shared parameters of all scenes consistent, so that all tasks converge at the same speed and reduce gradient conflicts. Considering that the magnitude and rate of change of the gradient reflect the reliability of the optimization direction in the current iteration, D is defined as a linear combination of the gradient magnitudes between tasks:

[0134] Define D as the common scale of the linear combination of gradient magnitudes across task scenarios:

[0135]

[0136] Where W n,I is the relative convergence of the nth task scenario in the Ith iteration:

[0137]

[0138] According to the public scale D, the optimization gradient of the nth task scene is obtained

[0139]

[0140] Then all Average, ensure that the gradients of the scene shared parameters of all scenes are the same, and obtain the gradient after the shared parameters are optimized

[0141]

[0142] Combine Figure 2 As shown, the optimization method for the learnable score is:

[0143] The gradient of the learnable score is expressed as

[0144]

[0145] Since the indicator function The derivative is zero at all points except λ, where it is undefined, so it makes no sense to involve it in backpropagation. To solve this problem, remove the terms involving the partial derivatives of the indicator function of the learnable score. By The gradient of The gradient of . Transforming the above formula, we get:

[0146]

[0147] In the formula is the indicator function gradient.

[0148] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. A six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features, characterized by: include: Basic network module, based on the shared parameters φ and specific parameters c obtained through training n The nth task scene image I captured by the six-degree-of-freedom camera n Perform feature extraction to obtain image extraction feature F n ; The shared parameter φ and the specific parameter φ n Calculated using the average gradient algorithm; n = 1, 2, ..., N; Regression layer: extract features F from the image n Make a prediction to get the 3D scene coordinate D n and 1D uncertainty U n ; Posture estimation module: using PnP algorithm, based on 3D scene coordinates D n and 1D uncertainty U n Calculate the rotation matrix R of the six-degree-of-freedom camera n and the translation vector t n , thereby determining the camera pose T in the nth task scene n ; After obtaining the image feature F n The process includes the task scene image I n The general feature extraction stage and the specific feature extraction stage; In the general feature extraction stage, all currently active layers in the basic network module share all shared parameters φ, and the gradient normalization algorithm is used to optimize the shared parameters φ; in the specific feature extraction stage, the optimized shared parameters φ are fixed, and the specific parameters φ of all currently active layers in the basic network module are optimized. n Optimize During the training process of the pose estimation system, based on the 3D scene coordinates D n and 1D uncertainty U n The loss function adjusts the shared parameters φ and the specific parameters φ n ; In the shared parameter φ and the specific parameter φ n During the calculation of , the binary score of the corresponding parameter is determined based on the learnable score; Gradient-based optimization of the learnable fraction.

2. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 1, characterized in that The basic network module includes a preprocessing layer with a step size of 2 and four residual layers; the shared parameter φ spans N task scenarios; the specific parameter φ n Corresponding to the nth task scenario; Image extraction feature F n for: Where f represents the basic network module, is a set of real numbers, C o F n The channel dimension; H o =H / 8,W o =W / 8, Where H is the length of the task scene image, and W is the width of the task scene image; φ∈χ sh ,f n ∈x sp χ sh is the set of shared parameters of N task scenarios obtained through iteration, χ sp is a set of N task scenario specific parameters obtained through iteration.

3. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 2, characterized in that Assume that the regression layer of the nth task scenario is h, and its parameter θ n ∈χ sp ,but: D n ,U n =h(F n ;θ n ), D n ={d n,i =(x n,i ,y n,i ,z n,i )∣i=1,2,3,…,P}, IN n ={in n,i ∣i=1,2,3,…,P}, Where d n,i is the 3D scene coordinate D n The i-th value of (x n,i ,y n,i ,z n,i ) is d n,i The three-dimensional coordinates of u n,i is the 1D uncertainty U n The i-th value of ; P is the number of 3D scene coordinates.

4. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 3, characterized in that The camera pose T in the nth task scene n for: Where p represents the pose estimation module.

5. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 4, characterized in that Shared parameter φ and specific parameter φ n The calculation method is: In the formula Shared parameters for the nth task scenario, Specific parameters for the nth task scenario; is the I-th iteration result of the n-th task scenario specific parameters, v is the learning rate, L n,I is the loss of the nth task scenario in the Ith iteration; Share parameters for the nth task scenario.

6. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 5, characterized in that The optimization method of the shared parameter φ is: For the nth task scenario, assume that the eth convolutional layer in the module consisting of the basic network module and the regression layer is the active layer, and set the initial universal weight of the eth convolutional layer Among them C out is the number of output channels, C in is the number of input channels, k is the convolution kernel size, and is shared in the common feature extraction stage: In the general feature extraction stage: T n =p(h(f(I n ;f,f n );i n )), Where M is the number of active layers, L is the number of inactive layers, is the universal weight of the lth inactive layer.

7. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 6, characterized in that In the specific feature extraction stage: Assume that the specific weight of the e-th convolutional layer in the module consisting of the base network module and the regression layer is Learnable fractions In the forward propagation, define the indicator function To determine whether to add scene-specific weights to the current convolutional layer in the current iteration: Where λ is the preset threshold; If the indicator function Then add scene-specific weights to the current convolutional layer Get the adaptive weight of the task in the current iteration Otherwise, only the general weights are considered In the specific feature extraction stage:

8. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 7, characterized in that: The loss function is expressed as L n : In the formula For the corresponding d n,i The true value of .

9. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 8, characterized in that Will Gradient Defined as: Will Gradient Defined as: The optimization method for shared parameter gradients is: Define D as the common scale of the linear combination of gradient magnitudes across task scenarios: Where W n,I is the relative convergence of the nth task scenario in the Ith iteration: According to the public scale D, the optimization gradient of the nth task scene is obtained Then all Average to get the gradient after shared parameter optimization 10. The six-degree-of-freedom camera pose estimation system based on learning scene general features and specific features according to claim 9, characterized in that: The optimization method for the learnable score is: The gradient of the learnable score is expressed as Transforming the above formula, we get: In the formula is the indicator function gradient.

Images