Camera pose estimation method, device, computer equipment and storage medium
By constructing a camera pose estimation model for feature extraction and cross attention calculation at a large-scale forest fire site, the problems of difficulty in image feature extraction and low matching accuracy are solved, and fast and accurate camera pose estimation is achieved, which improves robustness and portability for drone deployment and real-time operation.
Patent Information
- Application Number
- CN202410696041.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-31
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-05-31
AI Technical Summary
At large-scale forest fire scenes, the problem of difficult image feature extraction and low matching accuracy leads to inaccurate camera position estimation, making it difficult to real-time acquisition and analysis of three-dimensional multi-source information at the fire scene.
By using a camera pose estimation method, a camera pose estimation model including feature extraction, position encoding, cross attention calculation and feature matching modules is constructed, combining depth regulator and confidence classifier. Predict the network skeleton, perform feature extraction, position encoding, cross attention calculation and feature matching, and achieve fast and accurate estimation of camera poses.
It greatly improves the anti-interference ability to low-texture noise, improves the robustness and portability of camera pose estimation, and can achieve accurate camera pose estimation in high dynamic large-scale forest fires, suitable for embedded drones Deploy and run in real time.
Smart Images

Figure CN118537405B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and more specifically, to a camera pose estimation method, device, computer equipment and storage medium. Background Art
[0002] With the development of modern science and technology, with the help of Beidou satellite navigation system, watchtower monitoring, big data and Internet of Things, fire monitoring and visual information collection under two-dimensional remote sensing maps can be realized, which has improved the efficiency of fire fighting to a certain extent. However, in practical applications, there is still a lack of effective technical means to realize the real-time collection and analysis of three-dimensional spatial multi-source information at the scene of large-scale forest fires. The vegetation characteristics in large-scale forest fire scenes are of low significance, the fire burning range is large, the light intensity changes violently, and the noise disturbance is strong, resulting in the image collected by the camera with low texture and high dynamic characteristics, which makes it difficult to locate the three-dimensional spatial pose of the camera, and thus affects the subsequent intelligent forest fire prevention and control. For this reason, it is urgent to study the camera spatial pose estimation method in forest fire scenes to solve the problem of low feature matching accuracy and inaccurate camera freedom pose estimation caused by low texture and high dynamic of forest fire images, and provide a reliable basis for the dynamic reconstruction of forest fire scenes.
[0003] At present, the most advanced method in the research direction of dynamic scene camera pose estimation is to use deep learning as a means, and adopt a variety of innovative methods to improve the accuracy and robustness of pose estimation. One part uses spatial convolution to model the relationship between adjacent image frames, extracts features in dynamic scenes, and then uses them for pose estimation. For example, the Chinese patent with application number 202211704585.6 discloses a dynamic visual odometer method based on Canny edge, which solves the problem of image feature point extraction and matching through spatial convolution modeling. Another part of the method is to simplify the system design through end-to-end learning algorithms, and directly map the input image to the camera's pose, which is suitable for real-time pose estimation tasks. For example, the Chinese patent with application number 202211269544.9 discloses an end-to-end visual odometer method and device, which obtains the camera's pose estimation information by fusing image features according to the jump-fusion-FCNN method. Most of the existing methods use deep neural networks such as CNN and LSTM to directly estimate the camera pose, which has limited robustness to images with high dynamic targets and low texture backgrounds, and is not suitable for application in complex large-scale forest fire environments. Summary of the invention
[0004] 1. Technical issues to be solved
[0005] In view of the problem in the prior art that the static vegetation and dynamic flames at the forest fire scene make the collected images have the characteristics of low texture and high dynamics, which leads to difficulty in extracting image features and low matching accuracy during the camera pose estimation process, the present invention provides a camera pose estimation method, device, computer equipment and storage medium, which can quickly and accurately estimate the real-time three-dimensional pose of the fire, intelligently monitor the scene of large-scale forest fires, improve the multivariate analysis capability of the fire scene, and provide strong theoretical and technical support for the intelligent emergency management of forest fires.
[0006] 2. Technical solution
[0007] The purpose of the present invention is achieved through the following technical solutions.
[0008] A camera pose estimation method comprises the following steps:
[0009] Establish a fire image sequence dataset and obtain fire images;
[0010] Constructing a camera pose estimation model and a prediction network skeleton; the camera pose estimation model includes a feature extraction and position encoding module, a cross attention calculation module and a feature matching module, and the prediction network skeleton includes a confidence classifier and a depth regulator;
[0011] Input the fire image into the feature extraction and position encoding module to perform feature extraction and feature position fusion, and generate a feature representation with spatial position information; input the feature representation with spatial position information into the cross attention calculation module to calculate the dependency relationship between different feature representations with spatial position information, and obtain a feature representation with fused dependency information; input the feature representation with fused dependency information into the feature matching module to obtain a feature matching relationship;
[0012] The confidence classifier predicts the distribution of feature matches, and the depth regulator regulates the distribution of feature matches;
[0013] Supervise the training of the camera pose estimation model, calculate the camera pose, and obtain a trained camera pose estimation model;
[0014] Input the fire image sequence dataset into the trained camera pose estimation model to perform camera pose estimation and obtain the estimation result.
[0015] Furthermore, in the feature extraction and position encoding module, the spatial position information is added to the feature by rotational position encoding to generate a feature representation with spatial position information. The rotational position encoding is represented as:
[0016]
[0017] Where i represents the i-th position, j represents the j-th position, n represents the total number of feature vectors, R represents the rotation position encoding, and R i Represents the eigenvector rotation position encoding value of the i-th position, R j represents the rotation position encoding value of the feature vector at the jth position, q represents the query vector, and q i represents the query vector of the feature vector at the i-th position, k represents the key vector, k j represents the key vector of the feature vector at the jth position, v represents the value vector, and v j represents the value vector of the feature vector at the jth position, represents the ReLU activation function, T represents the transpose operation, Q represents the attention query vector matrix, K represents the attention key vector matrix, V represents the attention value vector matrix, Attention(Q,K,V) i represents the attention matrix.
[0018] Furthermore, the cross-attention calculation module includes channel cross-attention and spatial cross-attention; the channel cross-attention enhances the feature representation with spatial position information by calculating the correlation between channels, and the spatial cross-attention enhances the representation ability of the camera pose estimation model by capturing the correlation of spatial positions.
[0019] Furthermore, in the cross-attention calculation module, the dependency between different feature representations with spatial position information is calculated by bidirectional similarity, and the calculation formula is:
[0020]
[0021] Among them, A represents a fire image, B represents another fire image, represents the attention value from the feature vector at position i of fire image A to the feature vector at position j of fire image B, represents the attention value from the feature vector at position j of the fire image B to the feature vector at position i of the fire image A, The key vector representing the feature vector at position i of the fire image A, The key vector representing the feature vector of the fire image B at position j.
[0022] Furthermore, in the feature matching module, the similarity and matching between the feature representations of the fused dependency information are calculated through the sub-pixel interpolation matching algorithm to predict the feature matching relationship;
[0023] The calculation formula for the similarity between feature representations that incorporate dependency information is:
[0024]
[0025] Among them, Si j represents the similarity score matrix, x represents the feature vector, xi represents the feature vector at the i-th position, x j express
[0026] The eigenvector at the jth position, Represents the feature vector of the i-th position of the fire image A, represents the feature vector of the jth position of the fire image B, and Linear represents the fully connected linear layer;
[0027] The calculation formula for the compatibility between feature representations that fuse dependent information is:
[0028] σi=Sigmoid(Linear(x,))∈[0,1]
[0029] Among them, σ represents the matching score, σi represents the matching score of the feature vector at the i-th position, and Sigmoid represents the Sigmoid activation function;
[0030] The similarity and matchability between the feature representations that fuse the dependent information are coupled to optimize the feature matching relationship. The coupling formula is:
[0031]
[0032] Where Pij represents the coupling calculation soft bias matching matrix, represents the matchability score of the feature vector at the ith position of the fire image A, represents the matchability score of the j-th position feature vector of the fire image B, Skj represents the similarity score matrix of the feature vector of the fire image A, Si k Represents the similarity score matrix of the feature vectors of the fire image B.
[0033] Furthermore, the confidence classifier is used to predict the distribution of feature matching, and the prediction formula is:
[0034] ci=Sigmoid(MLP(x,))∈[0,1]
[0035] Where c represents the prediction distribution confidence, c i represents the feature vector x i Prediction distribution confidence, MLP represents multi-layer perceptron model;
[0036] The deep regulator adjusts the distribution of feature matching by estimating the stopping criterion of feature complexity. The adjustment formula is:
[0037]
[0038] Among them, exit means stopping the layer, N means the image length, M means the image width, l means the lth layer of the network, and L is the maximum number of layers of the network. represents the prediction distribution confidence of the l-th layer network, λ l represents the attenuation factor of the lth layer, and α represents the stopping threshold.
[0039] Furthermore, the camera pose is calculated using the PnP algorithm, and the calculation formula is:
[0040]
[0041] Among them, s i represents the scaling factor, u i Indicates the horizontal coordinate value in the pixel coordinate system, v i represents the ordinate value in the pixel coordinate system, K represents the camera intrinsic parameter matrix, R represents the rotation matrix, t represents the translation matrix, Xi represents the abscissa value in the world coordinate system, Y i Indicates the vertical coordinate value in the world coordinate system, Z i Indicates the depth value in the world coordinate system.
[0042] A camera pose estimation device, comprising:
[0043] Input module, establishes fire image sequence data set and obtains fire images;
[0044] A construction module is used to construct a camera pose estimation model and a prediction network skeleton, wherein the camera pose estimation model includes a feature extraction and position encoding module, a cross attention calculation module and a feature matching module, and the prediction network skeleton includes a confidence classifier and a depth regulator;
[0045] Training module, input fire image into feature extraction and position encoding module to perform feature extraction and feature position fusion, generate feature representation with spatial position information, input feature representation with spatial position information into cross attention calculation module to calculate dependency relationship between feature representations with different spatial position information, obtain feature representation with fused dependency information, input feature representation with fused dependency information into feature matching module to obtain feature matching relationship, confidence classifier predicts feature matching distribution, depth regulator regulates feature matching distribution, supervises camera pose estimation model training, calculates camera pose, and obtains trained camera pose estimation model;
[0046] The estimation module inputs the fire image sequence dataset into the trained camera pose estimation model to perform camera pose estimation and obtain the estimation result.
[0047] A computer device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned method when executing the computer program.
[0048] A computer-readable storage medium stores a computer program, and the computer program executes the above method when executed by a processor.
[0049] 3. Beneficial effects
[0050] Compared with the prior art, the advantages of the present invention are:
[0051] (1) A camera pose estimation method, apparatus, computer device and storage medium of the present invention build a self-attention unit and a cross-attention unit with targeted position encoding to enhance the dynamic feature extraction capability of the network; formulate a feature matching relationship prediction strategy that couples similarity and matchability, and captures the spatial relationship of inter-frame features in sub-pixels, greatly improving the ability to resist low texture noise interference.
[0052] (2) A camera pose estimation method, apparatus, computer equipment and storage medium of the present invention, combined with a trained feature matching camera pose estimation model, builds a deep adaptive prediction network skeleton, which can be flexibly adjusted according to the difficulty of the fire image pair. By predicting the corresponding feature matching relationship and supervising the camera pose estimation model training, it can achieve faster reasoning speed on easy-to-match fire image pairs, maintain accuracy on fire image pairs of high-dynamic large-scale forest fires, and improve the robustness and portability of the camera pose estimation algorithm.
[0053] (3) The camera pose estimation method, device, computer equipment and storage medium of the present invention can realize the embedded deployment and real-time operation of UAVs, are suitable for delay-sensitive downstream tasks, effectively break through the key problem of high-precision positioning of real-time camera spatial coordinates and observation angles, and have strong usability and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 This is a flow chart of a camera pose estimation method according to an embodiment of the present invention;
[0055] Figure 2 This is a structural diagram of a camera pose estimation model according to an embodiment of the present invention;
[0056] Figure 3 This is a flow chart of a camera pose estimation model training method according to an embodiment of the present invention;
[0057] Figure 4 This is a structural diagram of feature extraction according to an embodiment of the present invention;
[0058] Figure 5 The figure is a flow chart of a feature extraction method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0059] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0060] Example
[0061] like Figure 1 As shown, a camera pose estimation method is provided for this embodiment. Due to the static vegetation and dynamic flames at the forest fire scene, the images captured by the camera have the characteristics of low texture and high dynamics, which makes it difficult to extract image features and the matching accuracy is low during the camera pose estimation process. Most of the prior art enhances image matching features and realizes efficient image feature matching by dynamically sensing targets through estimating optical flow information. However, the fire scene in a large-scale forest fire scene has a large burning range, drastic changes in light intensity, and strong noise disturbance, which leads to a large dynamic mask range and difficulty in extracting static features, resulting in poor image matching effect and low camera pose estimation accuracy. Therefore, this embodiment provides a camera pose estimation method, which can realize fast and accurate image feature extraction in a large-scale forest fire scene, and effectively improve the robustness and portability of the camera pose estimation algorithm.
[0062] Specifically in this embodiment, first, a fire image sequence data set is established to obtain fire images. In this embodiment, a large-scale forest fire situation is photographed by an unmanned aerial vehicle to construct a large-scale forest fire image sequence data set, with the timestamp as the image sequence number, which contains the internal parameter matrix of the camera. In addition, each fire image sequence includes T frames, and the height of each fire image frame is H and the width is W. Then, the fire image is obtained in the fire image sequence data set, and the fire image is preprocessed, and the preprocessing of the fire image includes a saliency map algorithm and a Laplace filter. The fusion weight map of the fire image is calculated by the saliency map algorithm and the Laplace filter, and then the fusion weight map is used for multi-scale fusion to obtain the preprocessed fire image. It should be noted that in this embodiment, the saliency map algorithm and the Laplace filter used are prior art. Therefore, by preprocessing the fire image, the fog of the fire image can be reduced, the contrast of the fire image can be increased, and the characteristics of the fire image can be enhanced.
[0063] Further, a camera pose estimation model and a prediction network skeleton are constructed. Figure 2 As shown in the figure, the camera pose estimation model includes a feature extraction and position encoding module, a cross-attention calculation module and a feature matching module.
[0064] Specifically, the feature extraction and position encoding module is used to extract features and fuse feature positions of fire images. Figure 4As shown, a multi-layer convolutional neural network combined with residual connection is used as a feature extraction encoder. In this embodiment, the feature extraction encoder is composed of a convolution layer, a maximum pooling layer and a nonlinear activation layer. The size of the fire image can be reduced by the feature extraction encoder, and the feature points and descriptors in the fire image can be extracted at the same time. In this embodiment, the size of the fire image is reduced to 1 / 8 of the output image by three maximum pooling layers, and the feature map of the fire image with a dimension of (W / 8, H / 8, 128) is processed into a feature map with a dimension of (W / 8, H / 8, 65). When there are no feature points in the local area, the features are output through the Dustbin channel. Combined with the Softmax operation and the Reshape operation, the features are restored to the size of the original fire image, thereby completing the construction of the image feature encoder, and then extracting the feature points of the fire image through the image feature encoder. In this embodiment, the spatial relationship between the two frames of the fire image is estimated by extracting the feature points of the fire image, and the six-degree-of-freedom pose value of the camera is regressed. In this embodiment, a residual connection descriptor detection encoder is also constructed, and the feature description branch is improved by introducing a residual connection. The multi-scale feature extraction technology of the feature pyramid enables the feature pyramid to capture fire image information at different scales, thereby generating a more abstract feature representation. The residual connection establishes a direct connection between different layers of the encoder so that information can be effectively transmitted and integrated, while retaining and enhancing shallow features, thereby improving the expressiveness and robustness of the features. In this embodiment, the descriptor detection encoder not only utilizes the complementary information of the underlying features and high-level features, but also alleviates the gradient vanishing problem, improves training efficiency and performance, thereby more accurately detecting and encoding key features in the fire image, and providing reliable descriptors for subsequent tasks. For feature position fusion, in this embodiment, in the feature position fusion process, each feature point is required to pay attention to all feature points in the same fire image. For each feature point at position i, in order to calculate the attention value of the current feature point i, the current feature vector x i Decompose into key vector k through linear transformation i and the query vector q i , use the rotation code R to define the attention score a between the feature point at position i and the feature point at position j ij , to capture the relative position of the self-attention features. The closer the feature points are, the more relevant they may be. Figure 5 As shown, the feature points in the fire image are extracted by the feature extraction encoder, and the descriptors in the fire image are extracted by the descriptor detection encoder. Then the feature point scores are calculated, and the camera pose estimation model parameters are optimized using the feature point correlation loss and the feature vector correlation loss to obtain high-scoring feature points and descriptors in the fire image.
[0065] In this embodiment, the cross-attention calculation module includes channel cross-attention and spatial cross-attention. Channel cross-attention enhances the feature representation with spatial position information by calculating the correlation between channels, and spatial cross-attention enhances the representation ability of the camera pose estimation model by capturing the correlation of spatial positions. Specifically, channel cross-attention focuses on the information interaction and dependency between different channels in the feature map, and enhances the feature representation by calculating the correlation between channels. Spatial cross-attention focuses on the information interaction between different spatial positions in the feature map, and enhances the representation ability of the camera pose estimation model by capturing the correlation of spatial positions. It is worth noting that in this embodiment, the cross-attention calculation module considers the information between channels and spatial positions at the same time, effectively captures long-term series dependencies, and improves the long-term fire image matching performance. Therefore, the matching specificity of the features is enhanced by the cross-attention calculation module, and the dynamic feature extraction capability of the network is enhanced, and then the similarity of the specific feature points between different fire images is compared to obtain a soft bias matching matrix, so as to facilitate more accurate matching estimation.
[0066] In this embodiment, the feature matching module includes a feature extractor and a feature memory. At the same time, a sub-pixel interpolation matching algorithm is used in the feature matching module to predict the matching status of feature points at any level according to the updated state, and the feature matching relationship is predicted by coupling similarity and matchability. The sub-pixel captures the spatial relationship of inter-frame features, thereby effectively improving the anti-interference ability against high dynamic and low texture noise and improving the camera pose estimation accuracy.
[0067] In this embodiment, the prediction skeleton network is a prediction network skeleton with depth adaptive changes. The prediction network skeleton includes a confidence classifier and a depth regulator. The confidence classifier is a lightweight confidence classifier that can predict the distribution of feature matching at any level update state. The depth regulator is used to balance the inference time and the depth conflict of the camera pose estimation model, thereby adjusting the distribution of feature matching.
[0068] Therefore, if Figure 3As shown, the constructed camera pose estimation model is trained. Fire images are obtained from the established fire image sequence dataset. After preprocessing the fire images, the fire images are input into the feature extraction and position encoding module for feature extraction and feature position fusion, thereby generating a feature representation with spatial position information. It should be noted that after extracting the features, the spatial position information is added to the features through the rotation position encoding of the self-attention module to generate a feature representation with spatial position information. Since the self-attention mechanism algorithm is unordered, in order to enable the camera pose estimation model to understand and utilize the sequential information of the input sequence, the position information is added to the input vector, i.e., the feature, through position rotation encoding, so that the camera pose estimation model can distinguish different positions in the sequence, thereby better capturing the patterns and dependencies in the sequence. The rotation position encoding encodes the feature vector in a complex form, and its formula is:
[0069]
[0070] Among them, q represents the feature vector of the descriptor to be encoded, m represents the position sequence of the feature vector, f(q,m) represents the encoding operation, R represents the rotation position encoding, and R f (q,m) indicates that the feature vector q of the descriptor to be encoded is rotated and encoded at the position sequence m of the feature vector, u indicates an imaginary number, θ indicates the position arithmetic term, Θ indicates the angular value of the feature vector, and e indicates the natural exponent.
[0071] It should be noted that the rotation position coding is a relative position coding that can be used for linear attention. Linear attention is based on the inner product, and the rotation position coding, as a multiplicative coding, can also be inserted into the inner product. Therefore, in this embodiment, the rotation position coding is expressed as:
[0072]
[0073] Where i represents the i-th position, j represents the j-th position, n represents the total number of feature vectors, R represents the rotation position encoding, and R i Represents the eigenvector rotation position encoding value of the i-th position, R j represents the rotation position encoding value of the feature vector at the jth position, q represents the query vector, and q i represents the query vector of the feature vector at the i-th position, k represents the key vector, k j represents the key vector of the feature vector at the jth position, v represents the value vector, and v j represents the value vector of the feature vector at the jth position, represents the ReLU activation function, T represents the transpose operation, Q represents the attention query vector matrix, K represents the attention key vector matrix, V represents the attention value vector matrix, Attention(Q,K,V) i represents the attention matrix.
[0074] Therefore, in this embodiment, after obtaining the feature representation with spatial position information, the feature representation with spatial position information is input into the cross attention calculation module to calculate the dependency relationship between different feature representations with spatial position information, and obtain the feature representation with fused dependency information. Specifically, in this embodiment, each feature point in the fire image I will pay attention to all feature points in another fire image S, that is, each feature point in the fire image S will be queried, so the key vector k of each feature point is calculated. i , but there is no need to calculate the query vector q for each feature point i , thereby improving the efficiency and flexibility of matching, optimizing the utilization of computing resources by fixing and centralizing feature representations, and ensuring the comprehensiveness and accuracy of the matching process. In this embodiment, the dependency between feature representations with different spatial location information is calculated by bidirectional similarity, and the calculation formula is:
[0075]
[0076] Among them, A represents a fire image, B represents another fire image, represents the attention value from the feature vector at position i of fire image A to the feature vector at position j of fire image B, represents the attention value from the feature vector at position j of the fire image B to the feature vector at position i of the fire image A, The key vector representing the feature vector at position i of the fire image A, The key vector representing the feature vector of the fire image B at position j.
[0077] It should be noted that since the relative position information between fire images has no application significance, there is no need to fuse the position vector. Then, the attention weighted mean calculation formula of the key vector of all feature vectors of the fire image A is calculated as follows:
[0078]
[0079] Among them, m represents the attention calculation method between fire images, represents cross attention, W represents the weight matrix, The feature vector representing the location j of the fire image B.
[0080] Furthermore, the feature representation of the fused dependent information is input into the feature matching module to obtain the feature matching relationship. In this embodiment. The accuracy and robustness of feature matching are improved by coupling the prediction strategy of the feature matching relationship. At the same time, the sub-pixel accuracy ensures more precise feature positioning, reduces error accumulation, and adapts to the resolution and complex motion of different fire images. By comprehensively considering similarity and matchability, the matching relationship prediction is optimized, the probability of false matching is reduced, and the stability and accuracy of the matching algorithm in different scenarios are enhanced. In the feature matching module of this embodiment, the similarity and matchability between the feature representations of the fused dependent information are calculated by the sub-pixel interpolation matching algorithm, that is, the pairwise similarity score matrix S∈R between the feature points in different fire images is calculated. M*N , predict feature matching relationship;
[0081] The calculation formula for the similarity between feature representations that incorporate dependency information is:
[0082]
[0083] Among them, S ij represents the similarity score matrix, x represents the feature vector, x i represents the eigenvector at the ith position, x j represents the feature vector at the jth position, Represents the feature vector of the i-th position of the fire image A, Represents the feature vector of the j-th position of the fire image B, and Linear represents the fully connected linear layer.
[0084] The calculation formula for the compatibility between feature representations that fuse dependent information is:
[0085] σi=Sigmoid(Linear(x,i))∈[0,1]
[0086] Among them, σ represents the matching score, σi represents the matching score of the feature vector at the i-th position, and Sigmoid represents the Sigmoid activation function;
[0087] The similarity and matchability between the feature representations of fused dependency information are coupled to calculate the soft bias matching matrix and optimize the feature matching relationship. The coupling formula is:
[0088]
[0089] Where Pij represents the coupling calculation soft bias matching matrix, represents the matchability score of the feature vector at the ith position of the fire image A, represents the matchability score of the j-th position feature vector of the fire image B, Skj represents the similarity score matrix of the feature vector of the fire image A, Si k Represents the similarity score matrix of the feature vectors of the fire image B.
[0090] Thus, if and only if both feature points are predicted to be matched and the similarity between the two feature points is higher than any other feature points in the fire image, the point pair (i, j) will have a corresponding relationship, set the threshold τ, and select Pi j A pair of feature points that is greater than the threshold τ and greater than any other element in its row and column.
[0091] Thus, after obtaining the feature matching relationship, the confidence classifier in the prediction network skeleton with deep adaptive changes is used to predict the distribution of feature matching, and the depth regulator in the prediction network skeleton with deep adaptive changes is used to adjust the distribution of feature matching. Specifically, a lightweight confidence classifier is used to predict the distribution of feature matching at any level of update state. The confidence classifier helps decide whether to stop the reasoning process. If a few feature points have high confidence, the reasoning process continues to the next layer and cuts off unmatched feature points with low confidence. In this embodiment, the confidence classifier is used to predict the distribution of feature matching, and the prediction formula is:
[0092] c,=Sigmoid(MLP(x,))∈[0,1]
[0093] Where c represents the prediction distribution confidence, c i represents the feature vector x i Prediction distribution confidence, MLP represents multi-layer perceptron model;
[0094] In order to balance the inference time and the depth conflict of the camera pose estimation model, it is necessary to formulate a stopping criterion for feature complexity estimation during depth adjustment. That is, for a given layer l, if c i >λl, the feature point is considered to be credible. If the feature points reaching the proportion α are credible, the reasoning is stopped. In this embodiment, the distribution of feature matching is adjusted by estimating the stopping criterion through feature complexity. The adjustment formula is:
[0095]
[0096] Among them, exit means stopping the layer, N means the image length, M means the image width, l means the lth layer of the network, and L is the maximum number of layers of the network. represents the prediction distribution confidence of the l-th layer network, λ l represents the attenuation factor of the lth layer, and α represents the stopping threshold.
[0097] It is worth noting that in this embodiment, when training the camera pose estimation model, it is necessary to supervise the camera pose estimation model training. Therefore, in this embodiment, the camera pose estimation model training is supervised by the homography transformation matrix. Specifically, the ground truth label obtained by the homography transformation between two frames of images is used to supervise the soft bias matching matrix P, and the loss function is expressed as follows by minimizing the log-likelihood of the distribution predicted by each layer l:
[0098]
[0099] Among them, loss represents the loss function, L represents the total number of network layers, and M represents the matching true value. represents the true depth value of image A, represents the true depth value of image B, Represents the matching score of the feature vector at position A of image A, Represents the matchability score of the feature vector at position B in image B.
[0100] Then, the camera pose is regressed and calculated through the PnP algorithm to obtain the trained camera pose estimation model, whose calculation formula is:
[0101]
[0102] Among them, s i represents the scaling factor, u i Indicates the horizontal coordinate value in the pixel coordinate system, v i represents the vertical coordinate value in the pixel coordinate system, K represents the camera intrinsic parameter matrix, R represents the rotation matrix, t represents the translation matrix, X i Indicates the horizontal coordinate value in the world coordinate system, Y i Indicates the vertical coordinate value in the world coordinate system, Z i Indicates the depth value in the world coordinate system.
[0103] Finally, after obtaining the trained camera pose estimation model, the fire image sequence dataset is input into the trained camera pose estimation model to perform camera pose estimation and obtain the estimation result.
[0104] Therefore, the camera pose estimation method provided in this embodiment uses the camera pose estimation model training method to train the camera pose estimation model, so that the spatial six-degree-of-freedom pose of the camera can be estimated with high precision. At the same time, the camera pose estimation method provided by the present invention builds a self-attention unit and a cross-attention unit with targeted position encoding to enhance the dynamic feature extraction capability of the network; formulates a feature matching relationship prediction strategy that couples similarity and matchability, and sub-pixel captures the spatial relationship of inter-frame features, greatly improving the ability to resist low texture noise. In addition, combined with the trained feature matching relationship, a deep adaptive prediction network skeleton is built, which can be flexibly adjusted according to the difficulty of the fire image pair. By predicting the feature matching relationship, a faster reasoning speed can be achieved on the fire image pairs that are easy to match, and the accuracy is maintained on the fire image pairs of high-dynamic large-scale forest fires, while improving the robustness and portability of the camera pose estimation algorithm. In addition, a camera pose estimation method provided in this embodiment can realize the embedded deployment and real-time operation of UAVs, is suitable for delay-sensitive downstream tasks, and effectively breaks through the key problem of high-precision positioning of real-time camera spatial coordinates and observation angles. It provides strong support for the real-time collection and analysis of three-dimensional space multi-source information at large-scale forest fire scenes, promotes the digitalization and intelligent improvement of forest fire prevention, and provides strong theoretical and technical support for the intelligent emergency management of forest fires. It has strong practicality and wide applicability.
[0105] It should be noted that this embodiment also provides a camera pose estimation system, which uses the camera pose estimation method proposed in this embodiment for local deployment and hardware implementation. Specifically, the camera pose estimation system uses the HUAWEI Ascend series (HUAWEI Ascend) AI processor as the main control computing unit to build a forestry intelligent fire prevention computing solution, and onboard color cameras, near-infrared cameras, infrared cameras, GPS, IMU and other high-precision sensors to provide high-quality data support for multi-modal fusion forest fire scene real-time prevention and control; at the same time, the camera image capture GUI interface is built, and the synchronous hardware trigger function of multi-source data acquisition is built-in. The camera's image capture speed is as high as 30 frames / second, supporting the deployment of high-precision dynamic forest fire scene camera spatial pose estimation models.
[0106] This embodiment also provides a camera pose estimation device, including an input module, a construction module, a training module and an estimation module. In the input module, a fire image sequence data set is established to obtain a fire image. In the construction module, a camera pose estimation model and a prediction network skeleton are constructed, and the camera pose estimation model includes a feature extraction and position encoding module, a cross-attention calculation module and a feature matching module, and the prediction network skeleton includes a confidence classifier and a depth regulator. In the training module, the fire image is input into the feature extraction and position encoding module for feature extraction and feature position fusion, and a feature representation with spatial position information is generated. The feature representation with spatial position information is input into the cross-attention calculation module to calculate the dependency relationship between different feature representations with spatial position information, and the feature representation of fused dependency information is obtained. The feature representation of fused dependency information is input into the feature matching module to obtain a feature matching relationship, the confidence classifier predicts the distribution of feature matching, the depth regulator adjusts the distribution of feature matching, supervises the training of the camera pose estimation model, calculates the camera pose, and obtains a trained camera pose estimation model. In the estimation module, the fire image sequence data set is input into the trained camera pose estimation model to perform camera pose estimation and obtain an estimation result. The camera pose estimation device provided in this embodiment can implement any of the camera pose estimation methods, and the specific working process of a camera pose estimation device can refer to the corresponding process in the camera pose estimation method embodiment. The method and device provided in this embodiment can be implemented in other ways. For example, the device embodiment described above is only schematic; for example, the division of a module is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual connection or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.
[0107] This embodiment also provides a computer device. A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the camera pose estimation method when executing the computer program.
[0108] This embodiment also provides a computer-readable storage medium. A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a camera pose estimation method described in this embodiment is executed. The computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0109] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited thereto, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all belong to the protection scope of the present invention. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate the name, and do not indicate any specific order.
Claims
1. A camera pose estimation method, comprising the following steps: Establish a fire image sequence dataset and obtain fire images; Constructing a camera pose estimation model and a prediction network skeleton; the camera pose estimation model includes a feature extraction and position encoding module, a cross attention calculation module and a feature matching module, and the prediction network skeleton includes a confidence classifier and a depth regulator; Input the fire image into the feature extraction and position encoding module to perform feature extraction and feature position fusion, and generate a feature representation with spatial position information; input the feature representation with spatial position information into the cross attention calculation module to calculate the dependency relationship between different feature representations with spatial position information, and obtain a feature representation with fused dependency information; input the feature representation with fused dependency information into the feature matching module to obtain a feature matching relationship; The confidence classifier predicts the distribution of feature matching, and the depth regulator adjusts the distribution of feature matching. The confidence classifier predicts the distribution of feature matching, and the prediction formula is: Where c represents the prediction distribution confidence, c i represents the feature vector x i Prediction distribution confidence, MLP represents multi-layer perceptron model; The deep regulator adjusts the distribution of feature matching by estimating the stopping criterion of feature complexity. The adjustment formula is: Among them, exit means stopping the layer, N means the image length, M means the image width, l means the lth layer of the network, and L is the maximum number of layers of the network. represents the prediction distribution confidence of the l-th layer network, λ l represents the attenuation factor of the lth layer, and α represents the stopping threshold; Supervise the training of the camera pose estimation model, calculate the camera pose, and obtain a trained camera pose estimation model; Input the fire image sequence dataset into the trained camera pose estimation model to perform camera pose estimation and obtain the estimation result.
2. A camera pose estimation method according to claim 1, characterized in that: In the feature extraction and position encoding module, the spatial position information is added to the feature by rotating the position encoding to generate a feature representation with spatial position information. The rotation position encoding is represented as: Where i represents the i-th position, j represents the j-th position, n represents the total number of feature vectors, R represents the rotation position encoding, and R i Represents the eigenvector rotation position encoding value of the i-th position, R j represents the rotation position encoding value of the feature vector at the jth position, q represents the query vector, and q i represents the query vector of the feature vector at the i-th position, k represents the key vector, k j represents the key vector of the feature vector at the jth position, v represents the value vector, and v j represents the value vector of the feature vector at the jth position, represents the ReLU activation function, T represents the transpose operation, Q represents the attention query vector matrix, K represents the attention key vector matrix, V represents the attention value vector matrix, Attention(Q,K,V) i represents the attention matrix.
3. A camera pose estimation method according to claim 2, characterized in that: The cross-attention calculation module includes channel cross-attention and spatial cross-attention; the channel cross-attention enhances the feature representation with spatial position information by calculating the correlation between channels, and the spatial cross-attention enhances the representation ability of the camera pose estimation model by capturing the correlation of spatial positions.
4. A camera pose estimation method according to claim 3, characterized in that: In the cross-attention calculation module, the dependency between different feature representations with spatial position information is calculated through bidirectional similarity. The calculation formula is: Among them, A represents a fire image, B represents another fire image, represents the attention value from the feature vector at position i of fire image A to the feature vector at position j of fire image B, represents the attention value from the feature vector at position j of the fire image B to the feature vector at position i of the fire image A, The key vector representing the feature vector at position i of the fire image A, The key vector representing the feature vector of the fire image B at position j.
5. A camera pose estimation method according to claim 4, characterized in that: In the feature matching module, the similarity and matching between the feature representations of the fused dependent information are calculated through the sub-pixel interpolation matching algorithm to predict the feature matching relationship; The calculation formula for the similarity between feature representations that incorporate dependency information is: Among them, S ij represents the similarity score matrix, x represents the feature vector, x i represents the eigenvector at the ith position, x j represents the feature vector at the jth position, Represents the feature vector of the i-th position of the fire image A, represents the feature vector of the jth position of the fire image B, and Linear represents the fully connected linear layer; The calculation formula for the compatibility between feature representations that fuse dependent information is: Among them, σ represents the matchability score, σ i represents the matchability score of the feature vector at the i-th position, and Sigmoid represents the Sigmoid activation function; The similarity and matchability between the feature representations that fuse the dependent information are coupled to optimize the feature matching relationship. The coupling formula is: Among them, P ij represents the coupled calculation soft bias matching matrix, represents the matchability score of the feature vector at the ith position of the fire image A, represents the matchability score of the j-th position feature vector of the fire image B, S kj Represents the similarity score matrix of the feature vector of the fire image A, S ik Represents the similarity score matrix of the feature vectors of the fire image B.
6. A camera pose estimation method according to claim 5, characterized in that: The camera pose is calculated using the PnP algorithm, and the calculation formula is: Among them, s i represents the scaling factor, u i Indicates the horizontal coordinate value in the pixel coordinate system, v i represents the vertical coordinate value in the pixel coordinate system, K represents the camera intrinsic parameter matrix, R represents the rotation matrix, t represents the translation matrix, X i Indicates the horizontal coordinate value in the world coordinate system, Y i Indicates the vertical coordinate value in the world coordinate system, Z i Indicates the depth value in the world coordinate system.
7. A camera pose estimation device, characterized in that: include: Input module, establishes fire image sequence data set and obtains fire images; A construction module is used to construct a camera pose estimation model and a prediction network skeleton, wherein the camera pose estimation model includes a feature extraction and position encoding module, a cross attention calculation module and a feature matching module, and the prediction network skeleton includes a confidence classifier and a depth regulator; In the training module, the fire image is input into the feature extraction and position encoding module for feature extraction and feature position fusion, and a feature representation with spatial position information is generated. The feature representation with spatial position information is input into the cross attention calculation module to calculate the dependency relationship between different feature representations with spatial position information, and the feature representation with fused dependency information is obtained. The feature representation with fused dependency information is input into the feature matching module to obtain the feature matching relationship. The confidence classifier predicts the distribution of feature matching, and the depth regulator adjusts the distribution of feature matching. The confidence classifier predicts the distribution of feature matching, and the prediction formula is: Where c represents the prediction distribution confidence, c i represents the feature vector x i Prediction distribution confidence, MLP represents multi-layer perceptron model; The deep regulator adjusts the distribution of feature matching by estimating the stopping criterion of feature complexity. The adjustment formula is: Among them, exit means stopping the layer, N means the image length, M means the image width, l means the lth layer of the network, and L is the maximum number of layers of the network. represents the prediction distribution confidence of the l-th layer network, λ l represents the attenuation factor of the lth layer, and α represents the stopping threshold; supervise the training of the camera pose estimation model, calculate the camera pose, and obtain the trained camera pose estimation model; The estimation module inputs the fire image sequence dataset into the trained camera pose estimation model to perform camera pose estimation and obtain the estimation result.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is executed.
Citation Information
Patent Citations
End-to-end visual odometer method and device
CN115358962A
Dynamic visual odometer method based on Canny edge
CN116245899A
Image frame depth estimation method and device, electronic equipment and storage medium
CN111340867A
Three-dimensional reconstruction and camera pose estimation method based on multi-task learning algorithm
CN113160375A
Positioning method and device and method and device for training a feature matching networ
CN113592015A
