Low-light feature detection and description method based on multi-task automatic coding transformation
Through the low-light feature detection and description method of multi-task automatic encoding transformation, the problem of key point detection and matching in extreme low-light environments is solved, robust feature extraction and efficient matching are achieved, and the accuracy of pose estimation is improved.
Patent Information
- Application Number
- CN202311208627.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-19
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art is difficult to effectively detect and match key points in extremely low-light environments, traditional methods perform poorly under light changes and noise conditions, and the image enhancement process is prone to artifacts.
Using the low-light feature detection and description method of multi-task automatic coding transformation, generate robust feature maps through network training, predict low-light degradation transformation, calculate global and local descriptor similarity, establish multi-task orthogonal constraints, and utilize shared weights Encoder, degradation parameter prediction decoder, global descriptor sub-decoder and feature detection decoder to realize feature detection and matching.
Extract robust feature information in extremely low-light environments, adapt to light changes and noise changes caused by camera sensors, resist the influence of noise artifacts, obtain reliable key points and high-resolution descriptors, and improve the accuracy of pose estimation.
Smart Images

Figure CN119090970A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a feature detection and matching method in the field of robot and autonomous driving posture estimation. Background Art
[0002] With the widespread application of simultaneous localization and mapping technology and visual localization technology in robots and autonomous driving, key point detection and matching technology has become a research hotspot in computer vision. Since autonomous equipment operating around the clock needs to work in extreme scene tasks such as nighttime autonomous driving and robotic mine exploration, low-light feature detection and matching are becoming increasingly important. Therefore, effective key point detection and description under low light and noise conditions remains a challenge. Early studies such as SIFT and ORB methods mainly rely on pixel-level changes such as local pixel gradients and color space to extract key points. However, these traditional methods have difficulty detecting effective features in extreme low light environments. With the introduction of deep learning, which has the ability to explore image feature changes, traditional key point detection and description methods have gradually been replaced by convolutional neural networks (CNNs) and become mainstream. Therefore, the key point detection and description methods based on deep learning have evolved from "detection first, then description" to "joint detection and description" and then to "description first, then detection". In addition, some researchers have proposed a method that abandons the feature detection process and directly matches features. Although these methods can stably extract matching key points in difficult environments such as illumination changes, large perspective changes, weak textures, and motion blur, it is still difficult to effectively match key points in extreme environments with natural noise and low-light degradation factors.
[0003] In order to solve the key point detection and descriptor matching problems in low-light environments, some researchers proposed the "enhancement first, then detection" approach. First, low-light enhancement methods such as histogram equalization, Retinex decomposition, and neural networks are used to increase the brightness and reduce noise for low-light images, and then the enhanced images are input into the feature detection network to extract key points and descriptors. However, this method is prone to artifacts and incorrect matching key points during the image enhancement process. Later, He et al. proposed the DarkFeat network to eliminate the influence of noise in low-light images to improve the key point detection and description capabilities. However, this method ignores the illumination changes introduced by sensor parameters such as camera aperture, shutter speed, and ISO in order to adapt to low-light environments, which makes it difficult to effectively match key points under real low-light illumination changes. Therefore, it is necessary to propose an effective key point detection and description method to adapt to different illumination changes in extreme low-light environments. Summary of the invention
[0004] In order to solve the problem that the key point detection method cannot effectively adapt to the illumination change and artifacts in extreme low-light environment, this paper proposes a low-light feature detection and description method with multi-task automatic encoding transformation. The proposed low-light feature detection and description method can realize feature detection and matching in extreme dark light environment. Figure 1 As shown, the technical solution for implementing the present invention includes two processes: network training and reasoning.
[0005] The network training process includes the following steps:
[0006] Step 1: Generate a corresponding dark light image pair (i.e., t(I) and t(I')) by transforming the normal light image pair (i.e., I and I') with a rigid body transformation relationship captured by the camera through a low light degradation transformation (i.e., t(·));
[0007] Step 2: Input the normal light image pair and the dark light image pair into the shared weight encoder (i.e., E[·]) to generate the corresponding robust feature maps (i.e., E[I, I'] and E[t(I), t(I')]);
[0008] Step 3: Input the robust feature map into the degradation parameter prediction decoder to predict the low-light degradation transformation to form an automatic encoding transformation mechanism in a low-light environment;
[0009] Step 4: Input the robust feature map into the global descriptor decoder D g Generate the corresponding image global descriptor in , and calculate the similarity of the global descriptors of the normal light image pair and the dark light image pair;
[0010] Step 5: Input the robust feature maps of the dark-light image pair into the feature detection decoder D d and calculate the loss of local descriptors and key points;
[0011] Step 6: Establish a multi-task orthogonalization constraint based on automatic encoding transformation, that is, maximize the orthogonal direction of different decoder task outputs along the tangent, so that the downstream tasks of the encoder will not interfere with each other and affect the training accuracy;
[0012] Step 7: Use the normal light image dataset and the synthetic dark light image dataset to train the encoder, degradation parameter prediction decoder, global descriptor decoder, and feature detection decoder, and use the Adam optimizer to update the network weight gradient.
[0013] Furthermore, the process of predicting the low light degradation transformation in step 6 is expressed as: pre =D p {E[I,I'],E[t(I),t(I')]}, where t pre represents the predicted low-light degradation transform, D p{·} represents the degradation parameter prediction decoder, and the degradation parameter prediction loss function is: L deg = l p (P Deg ,T Deg ), where l p (·) represents the mean square error, P Deg and T Deg represent the predicted low-light degradation parameters and the actual low-light degradation parameters, respectively.
[0014] Furthermore, the loss function of the global descriptor similarity in step 4 is expressed as: L sim =1-0.5·cosine{f g [I],f g [t(I)]}-0.5·cosine{f g [I'],f g [t(I')]}, where f g [·] represents the global descriptor, and cosine{·} is the cosine similarity.
[0015] Furthermore, the specific process of the feature detection decoder in step 5 is:
[0016] a. First, the robust feature map is input into the local descriptor network. The loss calculation process of the local descriptor corresponding to each key point is:
[0017] 1) The size of each window on a single image of the dark-light image pair is g d ×g d Select the same random offset in the grid as the position of the initial key point;
[0018] 2) For each key point x in image t(I) i Select N points evenly on the polar line corresponding to t(I') to form a corresponding key point set Y Eline , and x i The matching probability is expressed as: P[y i j |f(x i ),f w (Y Eline )]=SoftMax[f(x i ) T f w (y i j )], where y i j Represents Y Eline Points on, f(·) and f w(·) represents the local description subgraph of t(I) and t(I'), and SoftMax represents the activation function;
[0019] 3) Calculate each key point x in the image t(I) i Corresponding to Y Eline The maximum matching probability of y is used to obtain a rough matching point y i max , and in y i max Select matching point y in the local window i center , the process is expressed as: i center =y i max +0.5 W patch ·u, where W patch is the size of the local window, u is the random offset;
[0020] 4) Calculate each key point x in t(I) i Corresponding to y i center The point with the maximum matching probability is used as a more accurate matching point y i maxp , and the key point x i and the corresponding matching point y i maxp The epipolar distance is expressed as: Desc(x i ,y i )=||y i maxp ·F matrix ·x i ||1, where F matrix is the fundamental matrix between t(I) and t(I'), and ||·||1 is the L1 norm;
[0021] b. Then, the depth projection error of each matching key point is obtained based on the camera internal and external parameters between t(I) and t(I'), and its expression is: H(x i )=|Warp[x i ,T c ,d(x i )]-d(y i )|, where |·| is the absolute value symbol, Warp[·] represents the function of calculating the projection depth using the camera pose transformation, T c is the intrinsic and extrinsic parameters of the camera, d(x i ) and d(y i) represent the true depth value of the matching key point of t(I) and the true depth value of the corresponding matching key point of t(I'); then, the depth projection error is used as the matching key point x in t(I). i Constraint M i d , when the depth error threshold is less than ε d When M i d is 1, otherwise it is 0; at the same time, x i The matching epipolar line corresponding to t(I') is restricted to the image boundary and serves as another key point constraint M i c , then the local descriptor loss function of t(I) and t(I') can be expressed as: L desc =∑ i W i ·||y i maxp ·F matrix ·x i ||1 / ∑ i W i , where W i It is M i d and M i c The intersection of i maxp The mask of the variance of ;
[0022] c. Finally, the local description subgraphs of t(I) and t(I') are input into the key point network to obtain the corresponding key point score graph. k ×g k The point with the highest score is selected from each grid as the candidate key point set Q1 and Q2, and the score expression is: kp (x) = Sigmoid(f) · SoftMax[f G(x) ], where Sigmoid(·) is the activation function, f represents the local descriptor corresponding to the candidate key point, and f G(x) represents the local descriptor corresponding to the grid where the candidate key point is located; then the key point loss function can be expressed as: L kp ={∑ i l rew (x i ,y j )+λ reg [∑ i log P kp (x i )+∑ i log P kp (y j)]} / (|Q1|+|Q2|), where λ reg is the regularization penalty coefficient, l rew (x i ,y j ) is expressed as: l rew (x i ,y j )=P m (x i ,y j )R(x i ,y j )·log(P kp (x i ) kp (y j )), where P m (x i ,y j ) is the approximate matching probability of the similarity matrix of Q1 and Q2, and is truncated to 0 when it is less than 0.9. i ,y j ) is the geometric score of Q1 and Q2, when Desc(x i ,y j ) is less than or equal to the epipolar distance threshold ε, R(x i ,y j ) is λ pos , otherwise λ neg .
[0023] Furthermore, the expression of the multi-task orthogonalization constraint in step 6 is: Where α is the penalty weight, E', D' p , D' g and D' d They are the encoder outputs from I' and t(I'), the degradation parameter prediction decoder, the global descriptor decoder and the feature detection decoder.
[0024] The network inference process includes the following steps: first, input the dark-light image pair into the trained encoder and detection decoder, and output the local descriptor map and the key point score map; second, use the non-maximum suppression, boundary culling and score threshold restriction methods to obtain the key points, and obtain the corresponding descriptors through the key point positions; finally, calculate the cosine distance of the descriptors extracted from the two dark-light images with a rigid body transformation relationship, and the key points with a distance less than a certain threshold are used as matching key points.
[0025] The shared weight encoder E is composed of the first three convolutional layers of ResNet-50. The normal light image or the dark light image is input into the encoder, and the robust feature map resolution of the encoder output is 1 / 16 of the input image.
[0026] The degradation parameter prediction decoder D p It consists of a global pooling layer and three fully connected layers, which input the robust feature maps of normal light images and dark light images into D p In, the predicted low-light degradation transform is output, i.e., five low-light degradation transform parameters;
[0027] The global descriptor decoder D g It consists of a convolutional layer and an L2-Norm operation, which inputs the robust feature maps of normal light images and dark light images into D g In , the resolution of the output global descriptor feature map is 1 / 16 of the input image, and the channel dimension is 128;
[0028] The feature detection decoder D d It is composed of a descriptor network and a key point network. The specific process is as follows: the descriptor network consists of two upsampling convolutional layers, and jumps to the feature information from the encoder. The dark-light image is input into the descriptor network, and the resolution of the output local description subgraph is 1 / 4 of the input image, and the channel dimension is 128; the key point network follows the output of the descriptor network, which consists of two ordinary convolutional layers and bidirectional linear interpolation, and jumps to the feature information from the encoder and the image. The local description subgraph is input into the key point network, and the resolution of the output key point score map is the same as the input image.
[0029] The low-light degradation transformation is a method for converting a normal-light image into a dark-light image, which is divided into three steps: RGB-to-RAW, low-light degradation, and RAW-to-RGB: the RGB-to-RAW method consists of inverse tone mapping, inverse gamma correction, sRGB-to-cRGB color space transformation, and inverse white balance gain operation; the low-light degradation method in the RAW domain consists of random dark light intensity and shot and read noise processing of the camera sensor; the RAW-to-RGB method is an ISP method, specifically consisting of quantization noise, white balance gain, cRGB-to-sRGB color space transformation, and gamma correction; in addition, the low-light degradation transformation parameters consist of dark light intensity, inverse gamma correction factor, inverse white balance gain (i.e., red channel and blue channel gain), and quantization noise factor.
[0030] The beneficial effect of this method is that the proposed low-light feature detection and description method based on multi-task automatic encoding transformation tightly couples image enhancement and feature extraction, and enhances the features of low-light images from the perspective of machine vision perception, so that the extracted robust feature information can adapt to different illumination changes and noise changes caused by camera sensors in low-light environments, and can resist the influence of noise artifacts, thereby obtaining reliable key points and highly discriminative descriptors in low-light environments. Therefore, the proposed feature detection and description method can obtain effective matching key points in extreme low-light environments, so that subsequent pose estimation can obtain more accurate results. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Attached Figure 1 Schematic diagram of training and reasoning of the multi-task automatic encoding transformation low-light feature detection and description method of the present invention.
[0032] Attached Figure 2 The correct matching rate curves of different low-light feature detection algorithms with different pose error thresholds on the dark light dataset.
[0033] Attached Figure 3 The top-50 key point matching results of the proposed low-light feature detection algorithm for real indoor dark-light images.
[0034] Attached Figure 4 Top-50 key point matching results of the proposed low-light feature detection algorithm for real outdoor low-light images. DETAILED DESCRIPTION
[0035] A typical embodiment of a low-light feature detection and description method based on multi-task automatic encoding conversion of the present invention is described in detail below, and the method is further described in detail. It is necessary to point out here that the following embodiments are only used to further illustrate the method and cannot be understood as limiting the scope of protection of the method. Personnel skilled in the art in the field may make some non-essential improvements and adjustments to the method based on the above content of the method, which still fall within the scope of protection of the present invention.
[0036] The present invention proposes a low-light feature detection and description method based on multi-task automatic encoding transformation, such as Figure 1 As shown, the method includes two processes: network training and reasoning.
[0037] The network training process includes the following steps:
[0038] Step 1: Generate a corresponding dark light image pair (i.e., t(I) and t(I')) by transforming the normal light image pair (i.e., I and I') with a rigid body transformation relationship captured by the camera through a low light degradation transformation (i.e., t(·));
[0039] Step 2: Input the normal light image pair and the dark light image pair into the shared weight encoder (i.e., E[·]) to generate the corresponding robust feature maps (i.e., E[I, I'] and E[t(I), t(I')]);
[0040] Step 3: Input the robust feature map into the degradation parameter prediction decoder to predict the low-light degradation transformation to form an automatic encoding transformation mechanism in a low-light environment;
[0041] Step 4: Input the robust feature map into the global descriptor decoder D g Generate the corresponding image global descriptor in , and calculate the similarity of the global descriptors of the normal light image pair and the dark light image pair;
[0042] Step 5: Input the robust feature map of the dark-light image pair into the feature detection decoder D d and calculate the loss of local descriptors and key points;
[0043] Step 6: Establish a multi-task orthogonalization constraint based on automatic encoding transformation, that is, maximize the orthogonal direction of different decoder task outputs along the tangent, so that the downstream tasks of the encoder will not interfere with each other and affect the training accuracy;
[0044] Step seven, use the normal light image dataset and the synthetic dark light image dataset to train the encoder, degradation parameter prediction decoder, global descriptor decoder and feature detection decoder.
[0045] Furthermore, the process of predicting the low light degradation transformation in step 6 is expressed as: pre =D p {E[I,I'],E[t(I),t(I')]}, where t pre represents the predicted low-light degradation transform, D p {·} represents the degradation parameter prediction decoder, and the degradation parameter prediction loss function is: L deg = l p (P Deg ,T Deg ), where l p (·) represents the mean square error, P Deg and T Deg represent the predicted low-light degradation parameters and the actual low-light degradation parameters, respectively.
[0046] Furthermore, the loss function of the global descriptor similarity in step 4 is expressed as: L sim =1-0.5·cosine{f g [I],f g [t(I)]}-0.5·cosine{f g [I'],f g[t(I')]}, where f g [·] represents the global descriptor, and cosine{·} is the cosine similarity.
[0047] Furthermore, the specific process of the feature detection decoder in step 5 is:
[0048] a. First, the robust feature map is input into the local descriptor network. The local descriptor loss calculation process corresponding to each key point is:
[0049] 1) The size of each window on a single image of the dark-light image pair is g d ×g d Select the same random offset in the grid as the position of the initial key point;
[0050] 2) For each key point x in image t(I) i Select N points evenly on the polar line corresponding to t(I') to form a corresponding key point set Y Eline , and x i The matching probability is expressed as: P[y i j |f(x i ),f w (Y Eline )]=SoftMax[f(x i ) T f w (y i j )], where y i j Represents Y Eline Points on, f(·) and f w (·) represents the local description subgraph of t(I) and t(I'), and SoftMax represents the activation function;
[0051] 3) Calculate each key point x in the image t(I) i Corresponding to Y Eline The maximum matching probability of y is used to obtain a rough matching point y i max , and in y i max Select matching point y in the local window i center , the process is expressed as: i center =y i max +0.5 W patch ·u, where W patch is the size of the local window, u is the random offset;
[0052] 4) Calculate each key point x in t(I) i Corresponding to y i center The point with the maximum matching probability is used as a more accurate matching point y i maxp , and the key point x i and the corresponding matching point y i maxp The epipolar distance is expressed as: Desc(x i ,y i )=||y i maxp ·F matrix ·x i ||1, where F matrix is the fundamental matrix between t(I) and t(I'), and ||·||1 is the L1 norm;
[0053] b. Then, the depth projection error of each matching key point is obtained based on the camera internal and external parameters between t(I) and t(I'), and its expression is: H(x i )=|Warp[x i ,T c ,d(x i )]-d(y i )|, where |·| is the absolute value symbol, Warp[·] represents the function of calculating the projection depth using the camera pose transformation, T c is the intrinsic and extrinsic parameters of the camera, d(x i ) and d(y i ) represent the true depth value of the matching key point of t(I) and the true depth value of the corresponding matching key point of t(I'); then, the depth projection error is used as the matching key point x in t(I). i Constraint M i d , when the depth error threshold is less than ε d When M i d is 1, otherwise it is 0; at the same time, x i The matching epipolar line corresponding to t(I') is restricted to the image boundary and serves as another key point constraint M i c , then the local descriptor loss function of t(I) and t(I') can be expressed as: L desc =∑ i W i ·||y i maxp ·F matrix ·x i ||1 / ∑ i W i , where W iIt is M i d and M i c The intersection of i maxp The mask of the variance of ;
[0054] c. Finally, the local description subgraphs of t(I) and t(I') are input into the key point network to obtain the corresponding key point score graph. k ×g k The point with the highest score is selected from each grid as the candidate key point set Q1 and Q2, and the score expression is: kp (x) = Sigmoid(f) · SoftMax[f G(x) ], where Sigmoid(·) is the activation function, f represents the local descriptor corresponding to the candidate key point, and f G(x) represents the local descriptor corresponding to the grid where the candidate key point is located; then the key point loss function can be expressed as: L kp ={∑ i l rew (x i ,y j )+λ reg [∑ i log P kp (x i )+∑ i log P kp (y j )]} / (|Q1|+|Q2|), where λ reg is the regularization penalty coefficient, l rew (x i ,y j ) is expressed as: l rew (x i ,y j )=P m (x i ,y j )R(x i ,y j )·log(P kp (x i ) kp (y j )), where P m (x i ,y j ) is the approximate matching probability of the similarity matrix of Q1 and Q2, and is truncated to 0 when it is less than 0.9. i ,y j ) is the geometric score of Q1 and Q2, when Desc(x i ,y j) is less than or equal to the epipolar distance threshold ε, R(x i ,y j ) is λ pos , otherwise λ neg .
[0055] Furthermore, the expression of the multi-task orthogonalization constraint in step 6 is: Where α is the penalty weight, E', D' p , D' g and D' d They are the encoder outputs from I' and t(I'), the degradation parameter prediction decoder, the global descriptor decoder and the feature detection decoder.
[0056] The network inference process includes the following steps: first, input the dark-light image pair into the trained encoder and detection decoder, and output the local descriptor map and the key point score map; second, use the non-maximum suppression, boundary culling and score threshold restriction methods to obtain the key points, and obtain the corresponding descriptors through the key point positions; finally, calculate the cosine distance of the descriptors extracted from the two dark-light images with a rigid body transformation relationship, and the key points with a distance less than a certain threshold are used as matching key points.
[0057] The shared weight encoder E is composed of the first three convolutional layers of ResNet-50. The normal light image or the dark light image is input into the encoder, and the robust feature map resolution of the encoder output is 1 / 16 of the input image.
[0058] The degradation parameter prediction decoder D p It consists of a global pooling layer and three fully connected layers, which input the robust feature maps of normal light images and dark light images into D p In, the predicted low-light degradation transform is output, i.e., five low-light degradation transform parameters;
[0059] The global descriptor decoder D g It consists of a convolutional layer and an L2-Norm operation, which inputs the robust feature maps of normal light images and dark light images into D g In , the resolution of the output global descriptor feature map is 1 / 16 of the input image, and the channel dimension is 128;
[0060] The feature detection decoder D dIt is composed of a descriptor network and a key point network. The specific process is as follows: the descriptor network consists of two upsampling convolutional layers, and jumps to the feature information from the encoder. The low-light image is input into the descriptor network, and the resolution of the output local description subgraph is 1 / 4 of the input image, and the channel dimension is 128; the key point network follows the output of the descriptor network, which consists of two ordinary convolutional layers and bidirectional linear interpolation, and jumps to the feature information from the encoder and the image. The local description subgraph is input into the key point network, and the resolution of the output key point score map is the same as the input image.
[0061] The low-light degradation transformation is a method for converting a normal-light image into a dark-light image, which is divided into three steps: RGB-to-RAW, low-light degradation, and RAW-to-RGB: the RGB-to-RAW method consists of inverse tone mapping, inverse gamma correction, sRGB-to-cRGB color space transformation, and inverse white balance gain operation; the low-light degradation method in the RAW domain consists of random dark light intensity and shot and read noise processing of the camera sensor; the RAW-to-RGB method is an ISP method, specifically consisting of quantization noise, white balance gain, cRGB-to-sRGB color space transformation, and gamma correction; in addition, the low-light degradation transformation parameters consist of dark light intensity, inverse gamma correction factor, inverse white balance gain (i.e., red channel and blue channel gain), and quantization noise factor.
[0062] In the example of the present invention, the normal light images used in network training are from the MegaDepth dataset and the GL3D dataset, while the synthesized dark light images are generated by the low-light degradation transformation. The real dark light images used in network inference are from the MID dataset. The range of the degradation parameters described in the low-light degradation transformation needs to be set first, specifically described as: the dark light intensity is set to a real number between [0.01, 1.0], the inverse gamma correction factor is set to a real number between [0.5, 3.5], the inverse white balance gain (ie, the red channel and blue channel gain) are set to real numbers between [1.9, 2.4] and [1.5, 1.9] respectively, and the quantization noise factor is set to one of 12, 14 and 16. In the setting of hyperparameters, N on the polar line is set to 100, and the size ratio W of the local window is patch is set to 0.1, and the window size g of the key points is randomly selected d and g k are 16 and 8 respectively, and the depth error threshold ε d is 0.05, the epipolar distance threshold ε is 2, and the regularization penalty coefficient λ reg is -0.001, λ pos With λ negare 1 and -0.25 respectively, and the penalty weight α is 0.5. The training process of the proposed network is divided into three stages: the first stage uses the normal light images and synthetic dark light images of the MegaDepth dataset to train the encoder and descriptor network as well as the degradation parameter prediction decoder and the global descriptor decoder. The loss functions involved are L desc , L deg , L sim and L ort , and use the Adam optimizer with an initial learning rate of 0.001 to update the corresponding weight gradients; the second stage uses the synthetic dark light images of the GL3D dataset to train the encoder and descriptor network, and the loss functions involved are L desc , and use the Adam optimizer with an initial learning rate of 0.0001 to update the corresponding weight gradients; the third stage uses the dark light images synthesized by the GL3D dataset to train only the key point network and freeze the weights of the encoder and descriptor network. The loss functions involved are L kp , and the corresponding weight gradients are updated using the SGD optimizer with an initial learning rate of 0.001. The training environment of the proposed method is the Pytorch framework of NvidiaRTX 3090GPU and Intel Core(TM)i7-7820X CPU@3.60Hz.
[0063] Table 1 Correct matching rates of different low-light feature detection algorithms in real low-light indoor and low-light outdoor environments
[0064]
[0065] In order to verify the effectiveness and generalization ability of the proposed method, the present invention uses the trained network to test the performance of feature detection and matching in a real low-light environment. The proposed method uses the MID low-light dataset as a test evaluation dataset. Table 1 is the feature matching rate of different low-light feature detection algorithms in indoor and outdoor scenes of the MID dataset. The proposed evaluation method sets the error thresholds of the estimated pose and the actual pose to 5 and 10, and the matching points less than the error threshold are regarded as correct matching points, so the evaluation index is the correct feature matching rate. At the same time, the proposed evaluation method combines image enhancement methods such as SID, HistEQ (histogram equalization) and ElightenGAN with feature detection and matching algorithms such as SP+SG (SuperPint+SuperGlue), R2D2+NN, LoFTR, and DarkFeat+NN to perform feature matching in low-light environments, where NN represents the nearest neighbor search matching method. According to the results in Table 1, the proposed method has a higher correct feature matching rate and more accurate pose estimation in low-light environments than other methods. As shown in Table 1, the proposed method has a higher correct feature matching rate and more accurate pose estimation in low-light environments. Figure 2As shown in the figure, the proposed evaluation method also visualizes the matching rate curves of different low-light feature detection and matching methods with a pose error threshold of 1 to 10, among which ALIKE+NN and PoSFeat are the mainstream feature detection and matching algorithms. Figure 3 The proposed method is used to compare key point matching in an indoor environment, where the ISO is 400 and 100, the SS (Shut Speed) is 0.01 and 0.025, and the average rotation and translation errors compared with the true value are 1.09 degrees and 1.449 degrees respectively. The figure selects the top 50 matching key points based on the key point matching scores. Similarly, Figure 4 The proposed method performs key point matching comparison in an outdoor environment, and the average rotation and translation errors compared with the true value are 0.30 degrees and 6.03 degrees respectively. According to the visualization results, the proposed low-light feature detection and description method is stable and effective under different exposure conditions.
Claims
1. A low-light feature detection and description method based on multi-task automatic encoding transformation, characterized in that: It includes two processes: network training and reasoning: The network training process is specifically described as follows: Step 1: The normal light image pair (i.e., I and I') captured by the camera and having a rigid body transformation relationship is transformed through low light degradation transformation (i.e., t(·)) to generate a corresponding dark light image pair (i.e., t(I) and t(I')); Step 2: Input the normal light image pair and the dark light image pair into the shared weight encoder (i.e., E[·]) to generate the corresponding robust feature maps (i.e., E[I, I'] and E[t(I), t(I')]); Step 3: Input the robust feature map into the degradation parameter prediction decoder to predict the low-light degradation transformation to form an automatic encoding transformation mechanism in low-light environment. The process is described as follows: pre =D p {E[I,I'],E[t(I),t(I')]}, where t pre represents the predicted low-light degradation transform, D p {·} represents the degradation parameter prediction decoder, and the degradation parameter prediction loss function is: L deg = l p (P Deg ,T Deg ), where l p (·) represents the mean square error, P Deg and T Deg represent the predicted low-light degradation parameters and the real low-light degradation parameters respectively; Step 4: Input the robust feature map into the global descriptor decoder D g Generate the corresponding image global descriptor in , and calculate the global descriptor similarity of the normal light image pair and the dark light image pair. The global descriptor similarity loss function is expressed as: L sim =1-0.5·cosine{f g [I],f g [t(I)]}-0.5·cosine{f g [I'],f g [t(I')]}, where f g [·] represents the global descriptor, cosine{·} is the cosine similarity; Step 5: Input the robust feature maps of the dark-light image pair into the feature detection decoder D d The specific process is: a. First, the robust feature map is input into the local descriptor network. The loss calculation process of the local descriptor corresponding to each key point is: 1) The size of each window on a single image of the dark-light image pair is g d ×g d Select the same random offset in the grid as the position of the initial key point; 2) For each key point x in image t(I) i Select N points evenly on the polar line corresponding to t(I') to form a corresponding key point set Y Eline , and x i The matching probability is expressed as: P[y i j |f(x i ),f w (Y Eline )]=SoftMax[f(x i ) T f w (y i j )], where y i j Represents Y Eline Points on, f(·) and f w (·) represents the local description subgraph of t(I) and t(I'), and SoftMax represents the activation function; 3) Calculate each key point x in the image t(I) i Corresponding to Y Eline The maximum matching probability of y is used to obtain a rough matching point y i max , and in y i max Select matching point y in the local window i center , the process is expressed as: i center =y i max +0.5 W patch ·u, where W patch is the size of the local window, u is the random offset; 4) Calculate each key point x in t(I) i Corresponding to y i center The point with the maximum matching probability is used as a more accurate matching point y i maxp , and the key point x i and the corresponding matching point y i maxp The epipolar distance is expressed as: Desc(x i ,y i )=||y i maxp ·F matrix ·x i ||1, where F matrix is the fundamental matrix between t(I) and t(I'), and ||·||1 is the L1 norm; b. Then, the depth projection error of each matching key point is obtained based on the camera internal and external parameters between t(I) and t(I'), and its expression is: H(x i )=|Warp[x i ,T c ,d(x i )]-d(y i )|, where |·| is the absolute value symbol, Warp[·] represents the function of calculating the projection depth using the camera pose transformation, T c is the intrinsic and extrinsic parameters of the camera, d(x i ) and d(y i ) represent the true depth value of the matching key point of t(I) and the true depth value of the corresponding matching key point of t(I'); then, the depth projection error is used as the matching key point x in t(I). i Constraint M i d , when the depth error threshold is less than ε d When M i d is 1, otherwise it is 0; at the same time, x i The matching epipolar line corresponding to t(I') is restricted to the image boundary and serves as another key point constraint M i c , then the local descriptor loss function of t(I) and t(I') can be expressed as: L desc =∑ i W i ·||y i maxp ·F matrix ·x i ||1 / ∑ i W i , where W i It is M i d and M i c The intersection of i maxp The mask of the variance of ; c. Finally, the local description subgraphs of t(I) and t(I') are input into the key point network to obtain the corresponding key point score graph. k ×g k The point with the highest score is selected from each grid as the candidate key point set Q1 and Q2, and the score expression is: kp (x) = Sigmoid(f) · SoftMax[f G(x) ], where Sigmoid(·) is the activation function, f represents the local descriptor corresponding to the candidate key point, and f G(x) represents the local descriptor corresponding to the grid where the candidate key point is located; then the key point loss function can be expressed as: L kp ={∑ i l rew (x i ,y j )+λ reg [∑ i log P kp (x i )+∑ i log P kp (y j )]} / (|Q1|+|Q2|), where λ reg is the regularization penalty coefficient, l rew (x i ,y j ) is expressed as: l rew (x i ,y j )=P m (x i ,y j )R(x i ,y j )·log(P kp (x i ) kp (y j )), where P m (x i ,y j ) is the approximate matching probability of the similarity matrix of Q1 and Q2, and is truncated to 0 when it is less than 0.
9. i ,y j ) is the geometric score of Q1 and Q2, when Desc(x i ,y j ) is less than or equal to the epipolar distance threshold ε, R(x i ,y j ) is λ pos , otherwise λ neg ; Step 6: Establish a multi-task orthogonalization constraint based on automatic coding transformation, that is, maximize the orthogonal direction of different decoder task outputs along the tangent, expressed as: Where α is the penalty weight, E', D' p , D' g and D' d are the encoder outputs from I' and t(I'), the degradation parameter prediction decoder, the global descriptor decoder, and the detection decoder; Step 7, using the normal light image dataset and the synthetic dark light image dataset to train the encoder, degradation parameter prediction decoder, global descriptor decoder and feature detection decoder; The network reasoning process is specifically described as follows: Step 1: Input the dark-light image pair into the trained encoder and feature detection decoder, and output the local description sub-graph and key point score graph; Step 2: Use non-maximum suppression, boundary elimination and score threshold limiting methods to obtain key points, and obtain corresponding descriptors according to the key point positions; Step 3: Calculate the cosine distance of the descriptors extracted from the two dark-light images with a rigid transformation relationship, and use the key points whose distance is less than a certain threshold as matching key points.
2. The low-light feature detection and description method based on multi-task automatic encoding transformation according to claim 1 is characterized in that: The shared weight encoder E in step 2 of the network training process is composed of the first three convolutional layers of ResNet-50. The normal light image or the dark light image is input into the encoder, and the robust feature map resolution output by the encoder is 1 / 16 of the input image. The degradation parameter prediction decoder D in step 3 of the network training process p It consists of a global pooling layer and three fully connected layers, which input the robust feature maps of normal light images and dark light images into D p In the network training process, the predicted low-light degradation transform is output, that is, five low-light degradation transform parameters; the global descriptor decoder D in step 4 of the network training process g It consists of a convolutional layer and an L2-Norm operation, which inputs the robust feature maps of normal light images and dark light images into D g In the above, the resolution of the output global descriptor feature map is 1 / 16 of the input image, and the channel dimension is 128; the feature detection decoder D in step 5 of the network training process d It is composed of a descriptor network and a key point network. The specific process is as follows: the descriptor network consists of two upsampling convolutional layers, and jumps to the feature information from the encoder. The dark-light image is input into the descriptor network, and the resolution of the output local description subgraph is 1 / 4 of the input image, and the channel dimension is 128; the key point network follows the output of the descriptor network, which consists of two ordinary convolutional layers and bidirectional linear interpolation, and jumps to the feature information from the encoder and the image. The local description subgraph is input into the key point network, and the resolution of the output key point score map is the same as the input image.
3. The low-light feature detection and description method based on multi-task automatic encoding transformation according to claim 2 is characterized in that: The low-light degradation transformation is a method for converting a normal light image into a dark light image, which is divided into three steps: RGB-to-RAW, low-light degradation, and RAW-to-RGB. The specific process is as follows: (1) The RGB-to-RAW method consists of inverse tone mapping, inverse gamma correction, sRGB-to-cRGB color space conversion, and inverse white balance gain operation; (2) The low-light degradation method in the RAW domain consists of random dark light intensity and the shot and read noise processing of the camera sensor; (3) The RAW-to-RGB method is an ISP method, which is specifically composed of quantization noise, white balance gain, cRGB-to-sRGB color space transformation, and gamma correction; in addition, the low-light degradation transformation parameters are composed of dark light intensity, inverse gamma correction factor, inverse white balance gain (i.e., red channel and blue channel gain), and quantization noise factor.
Citation Information
Patent Citations
Extremely dark light object detection method based on RAW image
CN114022732A
Deep neural network pose estimation system
US20210350566A1
