Visual feature detection and matching method fusing orb and sift

Abstract

The application discloses a visual feature detection and matching method combining ORB and SIFT, comprising the following steps: initializing system parameters; detecting DoG extreme points in a scale space by using the response significance of a FAST operator; combining a hybrid descriptor of ORB and SIFT to generate a matching strategy; finally, outputting matching point pairs containing key points in left and right images and corresponding three-dimensional point information, and buffering the key points of the current frame and the ORB / SIFT descriptors thereof as initial values for the next frame matching; the application combines the response significance of the FAST operator to guide the DoG extreme point detection in the scale space, uses a fusion mechanism to make the descriptor have stronger scale adaptability and matching stability, dynamically adjusts the proportion of the SIFT and ORB descriptors by using a smooth weight function, and retains the robustness of the SIFT in a large scale area and plays the efficiency of the ORB in a small scale area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, in particular to a visual feature detection and matching method fusing ORB and SIFT. BACKGROUND

[0002] In the field of machine vision, in order to identify objects, features need to be extracted from image pixels. With the rapid development of computer vision and artificial intelligence technology, image feature extraction and matching has become the basis for many visual tasks, especially playing a key role in tasks such as three-dimensional reconstruction, target recognition, stereo matching, and visual navigation. Currently widely used image feature detection and description algorithms mainly include SIFT, SURF, ORB, BRIEF, AKAZE, etc. Among them, SIFT is considered a high-precision solution in image matching due to its good scale invariance, rotation invariance, and robustness to light changes. However, this algorithm requires Gaussian pyramid construction, gradient direction calculation, and descriptor generation in the key point detection and description process, with high overall computational complexity, making it unsuitable for embedded or mobile platforms with high real-time requirements.

[0003] In contrast, the ORB algorithm uses an improved version of the FAST detector and BRIEF descriptor to achieve fast and effective feature point extraction and matching. It is highly efficient and suitable for mobile devices and real-time systems. However, the ORB algorithm may experience a decline in matching accuracy when dealing with large scale changes, significant light changes, or severe image distortion. In binocular vision systems, due to the existence of parallax, contrast differences, and certain image distortion between left and right views, traditional single feature matching algorithms cannot balance matching speed and accuracy under all conditions. Therefore, how to improve the stability and accuracy of matching while ensuring operational efficiency has become an important challenge in current research and application. SUMMARY

[0004] The purpose of the present application is to provide a visual feature detection and matching method fusing ORB and SIFT, which combines the response significance of the FAST operator to guide the detection of DoG extreme points in the scale space, and uses a smoothly changing weight function to dynamically adjust the proportion of SIFT and ORB descriptors according to the scale of the feature points, retaining the robustness of SIFT in large scale areas and the efficiency of ORB in small scale areas, making the descriptor have stronger scale adaptability and matching stability, to solve the problems raised in the background technology.

[0005] The specific technical solution provided by the present application is as follows: a visual feature detection and matching method fusing ORB and SIFT, comprising the following operation steps:

[0006] Step 1: initialize system parameters, set the number of feature points, scale level number, and related parameters including descriptor dimension in ORB and SIFT algorithms;

[0007] Preferably, the internal parameters of the binocular camera are configured , wherein the parameters represent the focal length of the camera in and directions, and the image center is calibrated in the pixel coordinate system. The image center is used for subsequent accurate geometric transformation and depth calculation operations. By centering the origin of the image coordinate system and moving the origin to the image center, the internal parameter matrix model is used to transform the world point to the pixel coordinate . The radial distortion parameters of the left and right cameras can be obtained based on the internal parameter matrix of the left camera and the internal parameter matrix of the right camera.

[0008] Step 2: detect DoG extreme points in the scale space using the FAST operator.

[0009] Preferably, the DoG extreme point detection in the scale space specifically includes:

[0010] Step 201: input the binocular image pair , perform preprocessing such as illumination correction and image alignment;

[0011] Step 202: construct a joint multi-scale space including SIFT and ORB pyramids;

[0012] Step 203: detect DoG extreme points. By only calculating the mask area, the constructed FAST response value responds to the mask;

[0013] Step 204: use Taylor expansion to refine the candidate key point set. Perform quadratic function fitting at the current position of the key point to calculate the extreme point offset;

[0014] Step 205: fuse the key point directions calculated by SIFT and ORB to obtain the final key point direction. The key point set containing key point position, scale, and direction information is output.

[0015] Step 3: fuse the hybrid descriptor of ORB and SIFT to generate a matching strategy.

[0016] Preferably, the matching strategy generation specifically includes:

[0017] Step 301: dynamically adjust the fusion weight of SIFT and ORB descriptors using scale space information to generate a hybrid descriptor.

[0018] Step 302: adaptively adjusting the matching threshold according to the local texture complexity of the region around the feature point;

[0019] Step 303: removing false matches by using the geometric constraint between consecutive frames;

[0020] Step 304: optimizing the key point trajectory in the sliding window.

[0021] Step 4: output the matching point pairs according to the matching strategy, and cache the key points and their ORB / SIFT descriptors of the current frame as the initial values for the next frame matching.

[0022] Preferably, the final matching point pairs containing left and right image coordinates and three-dimensional space coordinate information are output, the key points and their ORB / SIFT descriptors of the current frame are cached to provide initial values for the next frame matching, and the cycle fusion detection and matching are realized.

[0023] Compared with the prior art, the present application has the following beneficial effects:

[0024] (1) The present application guides the DoG extreme point detection in the scale space by combining the response significance of the FAST operator, uses a dynamic threshold to construct a response mask, and only performs multi-scale difference operation in the significant area, effectively avoiding large-area invalid calculation and improving the detection efficiency. The response-guided detection is completed in the scale space, which not only takes into account the calculation efficiency and feature stability, but also has high practical value and engineering popularization potential.

[0025] (2) The present application uses a smooth weight function according to the scale of the feature point to dynamically adjust the proportion of SIFT and ORB descriptors, retains the robustness of SIFT in large-scale areas, and plays the efficiency of ORB in small-scale areas. The fusion mechanism makes the descriptor have stronger scale adaptability and matching stability, which is a more innovative improvement to the traditional multi-descriptor fusion idea. In addition, the present application further introduces local entropy as a measure of image texture richness to construct an adaptive matching threshold mapping function in view of the problem that the threshold setting in the matching strategy cannot adapt to different image complexities. By linking the Hamming distance and the proportion test threshold, the matching strategy can automatically adjust the matching strength according to the regional complexity, significantly improving the matching accuracy and robustness.

[0026] (3) This invention combines the verification methods of reprojection error and optical flow consistency into a joint model and introduces geometric consistency verification in the front-end matching stage, which effectively improves the spatiotemporal consistency of the matching points. At the same time, combined with the sliding window filter, the motion trajectory of key points is smoothly optimized without increasing the system delay, which further improves the stability of front-end feature tracking. In addition, the sliding window filtering process considers rigid body motion constraints, making the optimization results physically interpretable and applicable to various dynamic scenarios. Attached Figure Description

[0027] Figure 1 This is a flowchart of the steps of the visual feature detection and matching method that integrates ORB and SIFT provided in the embodiments of the present invention;

[0028] Figure 2 This is a schematic diagram of the execution logic for detecting DoG extrema points in scale space provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram illustrating the calculation of only the mask area when detecting DoG extreme points in an embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram of a fusion decision tree that integrates ORB and SIFT provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0032] Example 1:

[0033] like Figures 1-4 As shown in the figure, the visual feature detection and matching method fusing ORB and SIFT described in this embodiment is mainly executed by a robot dog and a bipedal robot equipped with LiDAR and optical cameras including binocular cameras or depth cameras. The binocular cameras of the robot dog are located at the head position, facing directly forward; the camera of the bipedal robot is located at a position slightly below directly forward, facing the ground at a downward angle of 45 degrees. The specific operation steps of the method are as follows:

[0034] Step 1: Initialize system parameters, and set relevant parameters for ORB and SIFT algorithms, including the number of feature points, the number of scale levels, and the dimension of descriptors;

[0035] In the embodiment, since the existing laser radar precision cannot support the analysis requirement, in order to improve the navigation and obstacle avoidance precision of the robot dog and the robot, in addition to the laser radar, an optical camera including a binocular camera or a depth camera is carried on the robot dog and the biped robot, that is, positioning is performed by using the high-pixel recognition of the optical camera to recognize more things around, so as to improve the positioning precision, therefore, the binocular camera parameters including focal length, baseline, intrinsic matrix, distortion parameter and the like are configured according to the application scene; an image preprocessing module, a feature detection module and a matching module are initialized, and the related parameters including the number of feature points, the number of scale levels and the dimension of the descriptor in the ORB and SIFT algorithms are set,

[0036] Exemplarily, the binocular camera intrinsic parameters are configured , wherein the parameters represent the focal length of the camera in the and directions, the image center in the pixel coordinate system is calibrated, and is used for subsequent accurate geometric transformation and depth calculation and the like of the image, by centering the origin of the image coordinate system, the origin is moved to the image center, and the world point is transformed into the pixel coordinate by using the intrinsic matrix model, the intrinsic matrix model is:

[0037]

[0038] , wherein , respectively represent the original pixel coordinates , , the rotation matrix and the translation vector, the point under the camera coordinate system is projected to the intrinsic matrix of the image coordinate system based on the intrinsic matrix model, that is, the intrinsic matrix of the left camera is represented as:

[0039]

[0040] the intrinsic matrix of the right camera is represented as:

[0041]

[0042] Based on the intrinsic matrix of the left camera and the intrinsic matrix of the right camera, the radial distortion parameters of the left and right cameras can be obtained, the distortion coefficients include , wherein represents the radial distortion parameter, For tangential distortion coefficient, for correcting radial distortion and tangential distortion generated in the camera imaging process, making the image restore to more accurate geometric shape, in this embodiment, the distortion correction is carried out through Brown-Conrady model, and the correction formula is:

[0043]

[0044] wherein, represents the corrected normalized image coordinates, represents the normalized image coordinates after removing the influence of internal parameters, all represent radial distortion coefficients, represent tangential distortion coefficients, represents the normalized radius, represents the radial distortion correction, represents the tangential distortion correction, represents the tangential distortion cross term, and respectively represent the tangential distortion correction term for x coordinate and y coordinate, and the look-up table is constructed by iterative optimization of inverse mapping to realize real-time correction.

[0045] Further, the baseline is configured simultaneously, and an adaptive strategy based on close-range mode and long-range mode is set by using a depth calculation model, wherein the depth calculation model is: wherein, is a depth value, representing the vertical distance from the target point to the camera plane, represents the focal length of the camera, which is defined as the focal length of the two cameras in binocular vision, represents the baseline length, that is, the distance between the optical centers of the two cameras, represents the parallax, that is, the difference between the pixel coordinates of the same space point projected on the left and right camera images, and the specific calculation is: wherein is the horizontal pixel coordinate of the target point on the left camera image, is the horizontal pixel coordinate of the target point on the right camera image; the baseline determines the accuracy and range of depth calculation.

[0046] ​​​​In this embodiment, when constructing the Gaussian pyramid, the scale space is usually divided into several groups, and each group is divided into several layers. By initializing the feature detection parameters including the number of pyramid groups, the number of layers in each group, and the reference scale, the number of pyramid groups determines the number of layers of the constructed image pyramid. Different levels can detect features of different scales. Increasing the number of groups can expand the scale range of feature detection. The number of layers in each group further refines the image representation at each scale level, enabling the capture of more detailed features within the same scale range. More layers help to more accurately locate and describe feature points at different scales. The reference scale is the basic scale for constructing the scale space. Subsequent scale changes are calculated based on this reference scale. It determines the scale size of the initial feature detection. A suitable reference scale can ensure that representative features are detected in the initial stage.

[0047] Further, the initialization includes the ORB key parameters of the FAST corner threshold , the BRIEF descriptor dimension , and the non-maximum suppression radius , and the IFT key parameters including the extreme value detection threshold and the edge response threshold, fuse the heterogeneous features, and generate the ORB-SIFT hybrid strategy: generate candidate pairs through the ORB fast initial matching, use SIFT to verify the candidate pairs, fuse the descriptors , , and use the weighted voting mechanism for matching; through the parameter dynamic optimization model and the heterogeneous feature fusion to improve the system adaptability, combine the epipolar constraint to accelerate and the distortion real-time correction to correctly preserve the real-time and accuracy, and suitable for the binocular vision system in complex scenes.

[0048] Step 2: Use the response significance of the FAST operator to detect the DoG extreme points in the scale space.

[0049] Step 201: Take the binocular image pair as input, and perform pre-processing of light correction and image alignment;

[0050] In this embodiment, the binocular image pair is the basis data for subsequent feature detection and matching. The left and right images contain information obtained from different perspectives of the same scene. The light correction model is used to eliminate the light difference between the left and right images. The light correction model is:

[0051]

[0052] wherein, represents the original image pixel, represents the local mean of the image, represents the local standard deviation of the image, denotes target standard deviation, denotes target mean, by block calculation of local light statistics, using bilinear interpolation to generate global light field, realizing pixel-level normalization mapping, after pre-processing of photometric correction of the two images, using histogram to match feature points, by stereo correction to constrain matching points to the same horizontal line to perform epipolar alignment, making .

[0053] Step 202: Construct a joint multi-scale space including SIFT and ORB pyramids:

[0054] In the embodiment, a Gaussian pyramid is constructed, and the Gaussian kernel variance :

[0055]

[0056] wherein, denotes pyramid index, denotes layer index, denotes number of layers in each group, denotes reference scale, in the embodiment, is set to , , , to compensate for blur caused by sampling, further applying Gaussian blur to the pre-processed image, to generate scale space image in the group in the layer of the pyramid

[0057]

[0058] wherein, is a two-dimensional Gaussian kernel function, is a pre-processed input image, by using Gaussian kernel convolution, in the direction, the image is convolved with a one-dimensional Gaussian kernel, in the direction, the result of the previous step is convolved with a one-dimensional Gaussian kernel, to reduce the computational complexity, in each group, by successive Gaussian blur and down-sampling, when all layers of a group are completed, the last layer is down-sampled and used as the initial image of the next group, by down-sampling the image to different degrees to construct multiple layers, the scale space is obtained by convolving the image with Gaussian kernels of different scales, the formula is used to determine the Gaussian kernel scale in different layers and groups, Gaussian kernels of different scales can smooth the image to different degrees, so that features can be detected at different scales, by constructing a joint scale space in this way, unified feature analysis can be performed on binocular images at multiple scales.

[0059] Further, the FAST response mask is constructed, and the FAST response value of the whole image is calculated and the dynamic threshold , the quantile of the response value distribution is extracted:

[0060]

[0061] wherein, is a binary mask matrix, for identifying the potential feature region, is an indicator function, outputting 1 when the condition is true, otherwise outputting 0, is the FAST response value, representing the corner intensity of the pixel point , and is the dynamic threshold, which is an adaptive threshold that changes with the scale, wherein the response value is calculated as follows:

[0062]

[0063] wherein, is the gray value of the center pixel, is the set pixel value, is the basic threshold, is the indicator function;

[0064] the dynamic threshold is calculated as follows:

[0065]

[0066] wherein, represents the 70% quantile, which is the quantile point of the FAST response value of the whole image, is a natural constant, is the current scale, is the maximum scale; by collecting all values of the whole image, arranging them in ascending order to form a sequence , and calculating the position index corresponding to the value, the FAST is a fast feature point detection algorithm, which judges whether it is a feature point by calculating the FAST response value of the pixel point, and the response mask screens the potential feature point region according to the FAST response value and the dynamic threshold The calculation of the dynamic threshold combines the 70% quantile of the FAST response value and the relationship between the current scale and the maximum scale. This way can adaptively determine the feature point detection region according to the image content and the scale, and reduce invalid calculation.

[0067] Step 203: detecting the DoG extreme point, by only calculating the mask region, using the FAST response value Response to the mask;

[0068] In this embodiment, the difference between the Gaussian blurred images of two adjacent scales and is calculated to obtain the DoG response map, the mask is 0, the DoG response is directly set to 0, and the image set obtained by Gaussian blurring the original image at multiple scales is fused with the binary image of the same size as the original image to mark the region that may contain feature points, 1 represents the region of interest, and 0 represents the background or non-feature region. The difference image of each pair of adjacent scales is calculated, and the above difference image is multiplied pixel by pixel with the mask , that is, only the mask region is calculated, and other positions are assigned 0 to obtain the response map under the scale pair, and the response map is output , the non-zero value of which only appears in the region marked 1 by the mask. A candidate key point set is generated by detecting local extreme values in a set neighborhood.

[0069] For example, the calculation formula for calculating only the mask region using the DoG calculation model is:

[0070]

[0071] wherein, is the DoG response value, is the scale space image, is the current scale, is the next scale parameter, and the scale parameters and are two adjacent scales, and the ratio between them is fixed, is the response mask, is the pixel coordinate, DoG is a commonly used feature point detection method, which highlights potential feature points by calculating the difference between Gaussian blurred images of different scales. Here, only the mask region is calculated, that is, the region where the mask is 1, and the use of the previously constructed FAST response mask reduces a large amount of invalid calculation, greatly improving the detection efficiency, and concentrating the calculation of the potential feature point region improves the accuracy of feature point detection.

[0072] Step 204: Refining the generated candidate key point set using Taylor expansion, fitting a quadratic function at the current position of the key point, and calculating the extreme point offset;

[0073] In this embodiment, the position of each candidate key point is refined iteratively using the central difference method to calculate the gradient and Hessian matrix and the offset of the key point position The specific calculation formula is as follows:

[0074]

[0075]

[0076]

[0077] wherein, is the gradient vector of the DoG function at the key point, is the Hessian matrix of the DoG function at the key point, is the partial derivative, is the inverse Hessian matrix, and represent the first-order partial derivatives of the DoG function in the direction and the direction;

[0078] Further, the position is updated and iterative repositioning is used, and a convergence condition that the offset is smaller than a set pixel threshold or reaches a maximum number of iterations is set. According to the size of any component of the calculated offset and the set pixel threshold, iteration is performed. When any component of the offset is smaller than the set pixel threshold, the current key point position is output, and iteration is restarted. When the maximum number of iterations is exceeded or the position after the offset exceeds the neighborhood, the point is discarded. According to the iteration result, a refined final key point position is obtained , wherein, is the key point position of the discrete pixel coordinate of the current iteration. If the key point is offset to the adjacent pixel, the Hessian and the gradient of the adjacent position are recalculated, and adaptive threshold adjustment is performed The adaptive threshold formula is as follows: , is the basic threshold, is the adjustment coefficient, is the image noise level estimation, and the set threshold is dynamically adjusted according to the image noise level. When the noise is large, the set threshold is increased, and vice versa. The noise level is estimated through the statistical characteristics of the image gradient amplitude, and the eigenvalue verification of the Hessian matrix is performed synchronously to distinguish the corner point and the edge. The verification formula is as follows: wherein and are the eigenvalues of the Hessian matrix, is the larger eigenvalue, is the smaller eigenvalue,

[0079] Exemplarily, based on the second derivative matrix and the first derivative of the DoG function, the key point position is fine-tuned to locate the feature position in the image, and the Hessian matrix is used to describe the second derivative information of the function at a point, and by checking the proportional relationship of the eigenvalues, some unstable key points caused by image noise or non-feature structure can be removed. In the embodiment, the condition is set as As a judgment threshold, only the key points meeting the condition are retained, further improving the quality of the key points.

[0080] Step 205: Fuse the key point directions calculated by SIFT and ORB to obtain the final key point direction, and output the key points containing the key point position, scale and direction information to form a key point set;

[0081] In the embodiment, the SIFT direction and ORB direction are calculated by using the direction calculation model, the SIFT direction calculation and the ORB direction calculation are as follows:

[0082]

[0083] wherein, is the candidate direction, is the Gaussian weight, is the gradient direction histogram, is the histogram bucket offset, and are the direction matrices of and respectively;

[0084] The direction difference between the SIFT direction calculation and the ORB direction calculation is calculated by using the fusion strategy, and the calculation formula is:

[0085]

[0086] A dynamic tolerance threshold is set, when the SIFT and ORB direction difference is less than or equal to the set threshold, it is not directly rejected or directly adopted to a certain direction, but the direction weighted fusion is performed, the confidence weighted average of the two direction values is adopted, and the calculation formula of the weighted fusion direction is:

[0087]

[0088] wherein, is the weight assigned to the SIFT direction, For the weight assigned to the direction of ORB, if the main peak of the direction histogram of SIFT is prominent, it means that the direction definition of the key point is clear, and a higher weight is given to SIFT. At the same time, if the gradient amplitude at the key point is large, the weight of SIFT is also increased, because SIFT is based on gradient, and the larger the gradient, the more reliable the direction estimation. For the direction of ORB, if the local contrast is high, it means that the gray scale of the region changes obviously, and the direction estimation of the centroid method of ORB may be more accurate, so a higher weight is given. At the same time, since the direction of ORB may be more stable when the scale is large, and the normalized contrast is performed, the final direction is obtained by weighted averaging of the two directions according to the weight. This way of fusing the directions of the two algorithms combines the advantages of the two algorithms in direction calculation, and improves the accuracy of the direction assignment of the key points.

[0089] Step 3: fuse the hybrid descriptor of ORB and SIFT to generate a matching strategy;

[0090] Step 301: dynamically adjust the fusion weight of SIFT and ORB descriptor by using scale space information to generate a hybrid descriptor;

[0091] In this embodiment, the SIFT descriptor is normalized, the ORB descriptor is converted into a floating point vector and then normalized, the weight of SIFT and the weight of ORB are calculated according to the scale of the key point, and the normalized descriptor is weighted and fused. Then the fused descriptor is normalized again. The calculation formula of the weighted fusion of the descriptor is:

[0092]

[0093]

[0094] wherein, is the fused descriptor vector, is a scale-aware weight function, is the normalized SIFT descriptor, is the ORB descriptor converted into a floating point and normalized, is the binary to floating point, is a parameter for controlling the rate of weight change, is the scale at which the key point is located based on the scale space, is the median scale of weight switching. SIFT descriptor has good scale invariance and rotation invariance, but the calculation amount is large; ORB descriptor has high calculation efficiency but the performance is not as good as SIFT in some aspects. In this embodiment, a weight function that changes with the scale is used to dynamically adjust the weight of SIFT and ORB according to the scale of the key point, so that the advantages of SIFT and ORB are combined to improve the accuracy of the key point direction assignment. The SIFT and ORB descriptors are fused, the contribution ratio of both is dynamically adjusted at different scales, and the generated scale-aware descriptor can combine the advantages of both and has good description ability at different scales.

[0095] Step 302: adaptively adjusting a matching threshold according to the local texture complexity of the region around the feature point;

[0096] In the embodiment, for each query descriptor, two candidate descriptors with the minimum Hamming distance are searched in the target image and If <Adaptive Hamming distance threshold and <Adaptive scale test threshold of feature point position , the matching is accepted, the Hamming distance threshold is relaxed in the region with high local entropy, and the scale test is tightened, and the simple region is opposite, so that dynamic adjustment is realized.

[0097] For example, the dynamic threshold calculation formula is:

[0098]

[0099] Among them, is the adaptive Hamming distance threshold of the feature point position, is the adaptive scale test threshold of the feature point position, is the set basic threshold, and is the set adjustment coefficient, is the image entropy of the local region where the feature point is located, and the calculation formula is:

[0100]

[0101] Among them, is the probability of the gray value appearing in the circular region centered on the feature point, in the feature matching process, the selection of the matching threshold is crucial to the accuracy and efficiency of the matching result, the traditional method usually uses a fixed threshold, in the present application, the Hamming distance threshold and the scale test threshold are dynamically calculated according to the local entropy, the local entropy reflects the information entropy of the local region of the image, that is, the complexity of the region, and the matching threshold is adaptively adjusted according to the different characteristics of the local region, which can improve the accuracy of the matching and adapt to the needs of different scenes.

[0102] Step 303: removing false matches by using geometric constraints between consecutive frames;

[0103] In the embodiment, the spatio-temporal consistency check is performed, the geometric constraint of the matching points is verified by calculating the optical flow error and the re-projection error, the optical flow error measures the motion consistency between the feature points between adjacent frames, the re-projection error considers the error of the three-dimensional point projected to the image plane, the two are combined by weighted summation, and the weight is determined by the variance of the re-projection error. When the geometric constraint is greater than the set geometric error threshold, it is judged as an error matching point, and the matching is removed. The spatio-temporal consistency check can remove the error matching point and improve the reliability of the matching.

[0104] Step 304: optimizing the key point trajectory in the sliding window;

[0105] In the embodiment, the sliding window filtering is adopted to optimize the trajectory of the key points. The sliding window filtering performs weighted average on the key points in a certain time window, solves the translation speed and the rotation angular speed of the optimal reference point on the same object, and makes the residual error of the observation speed of all key points and the predicted speed of the rigid body motion model minimum. The weight exponentially decays with the increase of the time distance, so that the trajectory of the key points can be smoothed, and the influence of the noise and the abnormal points can be removed. At the same time, the rigid body motion constraint is met, which ensures that the motion of the key points conforms to the rigid body motion model, that is, the motion of the key points can be decomposed into the motion caused by the translation speed and the rotation angular speed, and the trajectory of the key points is further optimized, so that it is more consistent with the motion law of the object in the actual scene.

[0106] Step 4: outputting the matching point pair containing the key points in the left and right images and the corresponding three-dimensional point information, constructing the key point motion trajectory library, and buffering the key points and the ORB / SIFT descriptors of the current frame as the initial value of the next frame matching;

[0107] In the embodiment, the final matching point pair containing the left and right image coordinates and the three-dimensional space coordinate information is outputted, the key points and the ORB / SIFT descriptors of the current frame are buffered, and the initial value is provided for the next frame matching, so as to improve the matching efficiency between the time sequence frames in cooperation with the optical flow.

[0108] Exemplarily, the final matching point pair contains corresponding key points in left and right images and their corresponding three-dimensional point information, which is the final result of visual feature detection and matching, and can be used for further depth estimation, three-dimensional reconstruction and other tasks. The key point motion trajectory library records and tracks the position information of the key points at different times. By tracking the motion trajectory of the key points in multiple frames of images, the motion state and behavior of the object can be further analyzed, and it is also helpful to find abnormal key point motion and improve the stability and accuracy of the system. Finally, through the cross-frame prediction model, the position of the key point at the next time is predicted based on the rigid motion model using the position, translation speed and rotation angular velocity of the current key point, the position of the key point in the next frame is estimated in advance, the search range is reduced in the feature matching of the subsequent frame, the matching efficiency is improved, and the matching result can be verified and corrected, further improving the performance of visual feature detection and matching.

[0109] It should be noted that the relationship terms such as first and second in the present application are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0110] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and does not limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features, as long as they are within the spirit and principles of the present application. Any modification, equivalent replacement, improvement, etc. made within the scope of the present application should be included in the protection scope of the present application.

Claims

1. A method for visual feature detection and matching fusing ORB and SIFT, characterized in that: The method comprises the following operation steps: Step 1: initializing system parameters, setting ORB and SIFT algorithms including the number of feature points, scale level number, and the related parameters of descriptor dimension; Step 2: detecting DoG extreme points in the scale space by using a FAST operator; The step 2 specifically comprises: Step 201: taking a binocular image pair as input, performing pre-processing of light correction and image alignment; Step 202: constructing a joint multi-scale space including SIFT and ORB pyramids; Step 203: Detecting DoG extreme points, by only calculating the mask region, using the constructed FAST response value Response to the mask; Step 204: refining key points by using Taylor expansion on the generated candidate key point set, fitting a quadratic function at the current position of the key point, and calculating the extreme point offset; Step 205: fusing the directions of the key points calculated by SIFT and ORB, obtaining the final key point direction, and outputting the key point set containing the key point position, scale and direction information; calculating SIFT and ORB directions by using a direction calculation model; calculating the direction difference of SIFT and ORB by using a fusion strategy; setting a dynamic tolerance threshold, when the direction difference of SIFT and ORB is less than or equal to the set threshold, performing direction weighted fusion, and obtaining the final direction by weighted averaging the two directions according to the confidence weighted average; Step 3: fusing the hybrid descriptor of ORB and SIFT to generate a matching strategy; Step 4: outputting the matching point pair according to the matching strategy, and buffering the key points and ORB / SIFT descriptors of the current frame as the initial value for the next frame matching, and performing cyclic fusion detection and matching. 2.The method of claim 1, wherein: The initialization system parameters specifically comprise: configuring the internal parameters of a binocular camera; centralizing the origin of the image coordinate system, moving the origin to the image center, and transforming the world point to the pixel coordinate by using an internal parameter matrix model; calculating the internal parameter matrix of the point projection from the camera coordinate system to the image coordinate system, to obtain the radial distortion parameters of the left and right cameras based on the internal parameter matrix model; configuring the baseline length at the same time, and setting an adaptive strategy based on the close-range mode and the long-range mode by using a depth calculation model; Finally, the ORB key parameters including the FAST corner threshold, the BRIEF descriptor dimension and the non-maximum suppression radius are initialized, and the key parameters including the extreme detection threshold and the edge response threshold are initialized. The key parameters including the extreme detection threshold and the edge response threshold are initialized, and the ORB-SIFT hybrid strategy is generated by fusing the heterogeneous features. 3.The method of claim 2, wherein: constructing a joint multi-scale space comprises: constructing a Gaussian pyramid and calculating the Gaussian kernel variance; applying Gaussian blur to the pre-processed image to generate a scale space image at each layer of each group in the pyramid; in each group, constructing by means of continuous Gaussian blur and down-sampling, when all layers of a group are completed, down-sampling the last layer and taking it as the initial image of the next group, and constructing multiple levels by down-sampling the image to different degrees; finally, constructing a FAST response mask, calculating the FAST response value and the dynamic threshold of the whole image, and extracting the quantile of the response value distribution.

4. The method of claim 3, wherein the method further comprises: The calculation is only performed on the mask area, which comprises: calculating only the mask area by using a DoG calculation model.

5. The method of claim 4, wherein: The key point refinement comprises: performing position refinement iteration on each candidate key point, using central difference method, calculating the gradient and Hessian matrix of the current position by using the DoG image, and calculating the key point position offset; The position is updated and the iterative repositioning is used to set the convergence condition when the offset is less than a set pixel threshold or reaches a maximum iteration number; According to the size of any component of the calculated offset and the set pixel threshold, iteration is performed, and when any component of the offset is less than the set pixel threshold, the current key point position is outputted, and iteration is re-performed; When the maximum iteration number is exceeded or the offset position exceeds the neighborhood, the point is discarded, the refined final key point position is obtained according to the iteration result, and if the key point is offset to the adjacent pixel, the Hessian and gradient of the adjacent position are re-calculated, and adaptive threshold adjustment is performed; The set threshold is dynamically adjusted according to the image noise level, the set threshold is increased when the noise is large, and vice versa, the noise level is estimated through the statistical characteristics of the image gradient amplitude, the eigenvalue verification of the Hessian matrix is simultaneously performed, and the corner point and the edge are distinguished.

6. The fusion of ORB and SIFT visual feature detection and matching method according to claim 5, characterized in that: The generating matching strategy specifically comprises: Step 301: dynamically adjusting the fusion weight of the SIFT and ORB descriptors by using the scale space information to generate a hybrid descriptor; Step 302: adaptively adjusting the matching threshold according to the local texture complexity of the region around the feature point; Step 303: using the geometric constraint between consecutive frames to eliminate false matches; Step 304: optimizing and outputting the key point trajectory in the sliding window.

7. The fusion of ORB and SIFT visual feature detection and matching method according to claim 6, characterized in that: The generating hybrid descriptor comprises: normalizing the SIFT descriptor, converting the ORB descriptor into a floating-point vector and then normalizing it, calculating the weight of the SIFT and the weight of the ORB according to the scale of the key point, weighting and fusing the normalized descriptors, and outputting the fused descriptors after re-normalization. 8.The method of claim 7, wherein: The step 4 specifically comprises: outputting the final matching point pair containing the left and right image coordinates and three-dimensional space coordinate information, buffering the key points and the ORB / SIFT descriptors of the current frame to provide initial values for the next frame matching, and realizing cyclic fusion detection and matching.

Images