A method and system for dynamically adjusting the threshold of feature point extraction of an image

Abstract

The embodiment of the present invention provides a method and system for dynamically adjusting the feature point extraction threshold of an image, wherein feature extraction and feature tracking are regarded as two interdependent processes. The quality of the feature points extracted in the feature extraction stage will affect the result of feature tracking, and the result of feature tracking will be fed back to the feature extraction stage to adjust the key parameters required for feature extraction; the feature point extraction threshold is dynamically adjusted according to the current environmental conditions, so that the traditional feature extraction method can effectively extract feature points even in extreme lighting environments.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a method and system for dynamically adjusting a threshold value for extracting feature points of an image. Background Art

[0002] Direct visual SLAM (Simultaneous Localization And Mapping) can utilize a large amount of information on the image to perform registration between images. More information can usually provide more constraints, which is more conducive to solving the problem. However, when there is a large amount of noise in the image data or the image data is distorted, if the image is not properly processed, such as filtering, blurring, and dedistortion, a large amount of information used will be invalid or even wrong information, which will destroy the robustness of the system, and will cause wrong camera pose estimation for direct visual SLAM. On the other hand, the design of direct visual SLAM is based on the assumption that the luminosity of the front and back images is consistent, which puts high demands on the data itself, and in real life, the environment is often changeable and complex, which requires a large amount of pre-processing of the data, such as the adjustment of the image luminosity with reference to a reference image proposed in the embodiment of the present invention. In addition to direct visual SLAM, another very popular visual SLAM framework is feature point method visual SLAM. Feature point method visual SLAM uses a specially designed image feature extraction method to extract features from the image, and then uses the features extracted from the previous and next two frames of the image for feature matching. Finally, the obtained feature correspondence and multi-view geometric constraints are used to construct an overdetermined equation to solve the camera's position and posture. Compared with direct method visual SLAM, feature point method visual SLAM discards a large amount of redundant information on the image and only selects a small amount of the most representative information on the image for image registration, which to a certain extent reduces the interference of invalid image information on image registration. At the same time, the features extracted from the same image can describe the content of the image to a certain extent, so the features extracted from each image can also be saved and reused at a certain moment in the future to match a frame of image that is very far away on the time axis, which is often used for visual relocation. Feature point method visual SLAM is widely used because of its advantages such as being less affected by the environment and naturally supporting visual relocation technology. Therefore, it is also very necessary to discuss how its performance will be affected by extreme lighting conditions and how to operate more robustly under these extreme lighting conditions.

[0003] Feature point method visual SLAM uses artificially designed features to extract pixels with local distinguishability on the image. Commonly used image feature points include Fast feature, Harris feature, ORB feature, SIFT feature, and SUFT feature. Among them, Fast feature and ORB feature calculate the difference between each pixel and the surrounding circle of pixels. If there are enough continuous points with large differences from it, it is regarded as an image feature point. The difference size is judged here by calculating the difference size of the pixel value between two pixels. When the difference in pixel value is greater than a certain set threshold, it is considered that the two pixels are very different. Harris feature, SIFT feature, and SUFT feature first use a pre-designed template to perform convolution operation on the image to obtain the image response map for the template, and then find the local maximum response point on the response map. Finally, the pixel points corresponding to these local maximum response points on the image are used as the features of the image. The selection of the local maximum response point often sets a minimum response threshold to prevent the extraction of wrong feature points in weak texture areas. After feature extraction, it is necessary to find the corresponding relationship between the features extracted from the two frames of images. For ORB features, SIFT features, and SUFT features, they will additionally calculate feature descriptors for each feature point during the feature extraction stage. ORB features use the BRIEF algorithm to calculate feature descriptors. It selects two pixels in a fixed area around the feature point for comparison according to certain rules, and records the comparison results in sequence in the form of 0 or 1, eventually forming a binary sequence as the feature descriptor. SIFT features and SURF features divide a fixed area around the feature point into several sub-areas, calculate the gradient direction for each pixel in each sub-area and generate a gradient direction histogram, and then combine the gradient direction histograms obtained from each sub-area as the feature descriptor of the feature point. With feature descriptors, features can be matched directly using feature descriptors. The simplest method is to use a suitable distance metric to compare the descriptor of a feature on one image with the descriptor of each feature on the other image, and use the feature with the smallest distance as the matching feature. Harris features and SIFT features themselves do not calculate the feature descriptors of each feature additionally. For such features, the optical flow method can be used to track the features based on the pixel values ​​of the feature points themselves and the gradient distribution around the feature points to find the corresponding feature on another image.Usually, there will be some wrong matches among all the feature correspondences found. In order to filter out these wrong matching results, the RANSAC (Random Sample Consensus) algorithm is used to find a group of correspondences that account for the majority of all feature correspondences. Specifically, a model is first determined, which contains the conversion relationship between the corresponding pixels in the two frames of images. After having this model, several groups of all feature correspondences are randomly selected to solve this model, and then the number of all the remaining feature correspondences that support this model is counted. This process is repeated, and finally a model that can satisfy the most feature correspondences is found. At the same time, all feature correspondences that support this model are considered to be correct matches, while the remaining small part of feature correspondences that do not support this model are considered to be wrong matches and are eventually discarded. With the feature correspondence between the two frames of images, overdetermined equations can be constructed through the constraints provided by multi-view geometry. It can be used when the camera shoots the same target from different positions. Figure 1 To indicate. l and O r represents the imaging center of the two cameras, point P represents the object being photographed in the three-dimensional space, and the three-dimensional point P and the camera center O l The connection P l , that is, the projection dimension point P of the target point projected onto the camera during the imaging process and the camera center O l The connection P l , that is, the projection line of the target point projected onto the camera during the imaging process, mapping a point p on the image l ,Similarly, the three-dimensional point P and the camera center O r The connection P r Map a point p on the image r . We can always find a set of spatial rotations and translations to transform P l Convert to P r , expressed as:

[0004] P r =R·(P t -T) (1)

[0005] (P l -T=R -1 P r =R T P r (2)

[0006] Among them, R and T represent the rotation and translation in space respectively. l , P r The three vectors T and T are on the same plane, so there are the following constraints:

[0007] (Pl -T) T ·T×P l =0 (3)

[0008] Right now:

[0009] (R T P r ) T ·T×P l =0 (4)

[0010] in:

[0011]

[0012] Substituting formula (5) into formula (4), we get:

[0013] P r T ·R·S·P l =0 (6)

[0014] Consider R·S in the formula as a matrix E, which is called the essential matrix. It can be seen that the essential matrix E contains all the information about the relative motion of the camera in three-dimensional space between the two frames of images. The pinhole imaging model is usually used as the imaging model of the camera:

[0015] p l =KP l (7)

[0016] in,

[0017]

[0018] Among them, f x and f y are the focal lengths of the camera in the horizontal and vertical directions respectively. Substituting equation (7) into equation (6), we get:

[0019] p r T K -T EK -1 p l =0 (9)

[0020] K in the move -T EK -1It is regarded as a matrix F, which is called the basic matrix. Equation (9) establishes the connection between the different projection results and camera motion of the same three-dimensional space point between the two frames of images, so it can be used in the RANSAC algorithm to find the correct feature correspondence. The basic matrix has only 8 degrees of freedom due to the lack of scale information used for translation. Therefore, to solve the basic matrix through equation (9), at least 8 groups of corresponding pixel points need to be found between the two frames of images. Usually, in order to ensure the accuracy of the solution, more than 8 groups of corresponding pixel points are always selected to construct overdetermined equations and solve them. After solving the basic matrix, the essential matrix is ​​calculated by the following formula:

[0021] E=K T FK (10)

[0022] After obtaining the essential matrix, the relative motion of the camera in three-dimensional space corresponding to the two frames of images is restored through singular value decomposition, that is, three-dimensional rotation and three-dimensional translation, where there is an unknown scale relationship between the three-dimensional translation and the real translation. After obtaining the motion information of the camera in three-dimensional space, each pixel on the image can be projected into three-dimensional space by back projection to obtain the coordinates of the points on the image in three-dimensional space. Since there may be noise in the image, the corresponding results of the feature points between the two frames of images are not very accurate, and the accuracy of the estimated relative motion of the camera in three-dimensional space will be reduced. In order to further reduce the estimation error, after calculating the initial motion estimate of the camera, all the matching feature points on the previous frame of the image are projected onto another frame of the image through three-dimensional projection, and a three-dimensional projection error function is constructed:

[0023]

[0024] in:

[0025]

[0026] in represents all the matched feature points on the image, ∏c(·) represents the projection of the three-dimensional coordinates in the world coordinate system to the image coordinate system, ∏c -1(·) indicates that the two-dimensional coordinates in the image coordinate system are projected into the world coordinate system. The trajectory estimation of the camera in the three-dimensional space is fine-tuned using a nonlinear optimization method to minimize the projection error of all feature points. Compared with the direct method, which constructs the back-projection error based on the assumption that a point in the three-dimensional space projected onto the two frames of images will have the same pixel value, and finds the movement of the camera in the three-dimensional space by minimizing the difference between the pixel value of the projected pixel and the pixel value of the projected pixel, the feature point method finds the correspondence between the pixels of the two frames in the feature matching stage, so it can find the movement of the camera in the three-dimensional space by minimizing the coordinates of the projected point on the image and the coordinates of the point on the other image corresponding to the projected point.

[0027] It can be seen that the feature point method is not affected by the inconsistency of the image before and after the motion estimation stage of the optimized camera, that is, it is not affected by the change of illumination at this stage. However, illumination change does not have no effect on the feature point method SLAM. When constructing the projection error, the feature point method SLAM is actually based on another assumption different from the direct method SLAM, that is, there are a sufficient number of feature point matches between the two frames of images, and the matching results are accurate. This requires that the artificially designed feature extraction method can extract enough feature points under different lighting conditions, and the feature matching process should be robust to lighting changes. However, most image feature points are not designed to take into account extreme lighting conditions, such as dim environments, overexposed images, and changing lighting conditions. In these extreme scenes, these feature point extraction methods often cannot extract enough feature points, and it is difficult to take into account the changes in different scenes when performing specific optimization for these extreme scenes, which will also reduce the quality of feature point extraction in non-extreme scenes. Summary of the invention

[0028] The embodiments of the present invention provide a method and system for dynamically adjusting the threshold of feature point extraction of an image, so as to solve the problem that in the prior art, extreme lighting conditions are not taken into consideration when extracting image feature points, such as dim environments, overexposed images, constantly changing lighting conditions, etc., resulting in failure to extract enough feature points, difficulty in taking into account changes in different scenes, and reduced quality of feature point extraction in non-extreme scenes.

[0029] In a first aspect, an embodiment of the present invention provides a method for dynamically adjusting a feature point extraction threshold of an image, comprising:

[0030] Step S1, setting initial image feature point extraction parameters, wherein the initial image feature point extraction parameters include a feature point extraction threshold, a feature point extraction threshold increase factor, and a feature point extraction threshold decrease factor;

[0031] Step S2, determining the feature point extraction results and feature point tracking results of the current frame and the previous frame when performing image feature point extraction based on the initial image feature point extraction parameters;

[0032] Step S3, determining the average moving distance of feature points, the feature tracking success rate, and the number of feature points tracked based on the feature point extraction results and feature point tracking results of the current frame and the previous frame;

[0033] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate meets a first preset condition, increasing the feature point extraction threshold based on the feature point extraction threshold increase factor;

[0034] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate and the number of feature point tracking meet the second preset condition, the feature point extraction threshold is reduced based on the feature point extraction threshold reduction ratio.

[0035] Preferably, in step S1, the initial image feature point extraction parameters further include a maximum feature point tracking number, a minimum feature point tracking number, and a feature point tracking success rate threshold.

[0036] Preferably, the first preset condition is: the feature tracking success rate is not greater than the feature point tracking success rate threshold, or the feature tracking success rate is greater than the feature point tracking success rate threshold, and the feature point tracking number is greater than the maximum feature point tracking number;

[0037] The second preset condition is: the feature tracking success rate is greater than the feature point tracking success rate threshold, the number of feature point tracking is greater than the minimum feature point tracking number and not greater than the maximum feature point tracking number, or the feature tracking success rate is greater than the feature point tracking success rate threshold, the number of feature point tracking is not greater than the maximum feature point tracking number, and the feature points are unevenly distributed.

[0038] Preferably, in step S3, if it is determined that the camera has moved based on the average moving distance of the feature points, the method specifically includes:

[0039] If the average moving distance of the feature points of the current frame image relative to the previous frame image is less than a preset moving threshold, it is determined that the corresponding camera has not moved;

[0040] If the average moving distance of the feature points of the current frame image relative to the previous frame image is not less than a preset moving threshold, it is determined that the corresponding camera moves.

[0041] Preferably, in step S3, if it is determined that the camera has not moved based on the average moving distance of the feature points, the feature extraction threshold is not adjusted at the current moment.

[0042] Preferably, in step S3, when judging whether the feature point distribution in the second preset condition is met, the image is divided into several areas of equal size, and the number of successfully tracked feature points in each area is counted respectively. If the number of feature points in any area is less than the preset feature point number threshold, it is judged that the feature points currently being effectively tracked are unevenly distributed.

[0043] Preferably, in step S3, the feature tracking success rate is:

[0044]

[0045] In the above formula, n f Indicates the number of feature points obtained before the current feature point tracking, including the feature points successfully tracked at the previous moment and the new feature points extracted from the feature points at the current moment; n t Indicates the number of feature points successfully tracked at the current moment.

[0046] In a second aspect, an embodiment of the present invention provides a system for dynamically adjusting a threshold value for extracting feature points of an image, comprising:

[0047] An initialization setting module is used to set initial image feature point extraction parameters, wherein the initial image feature point extraction parameters include a feature point extraction threshold, a feature point extraction threshold increase factor, and a feature point extraction threshold decrease factor;

[0048] A feature extraction and tracking module, used to determine the feature point extraction results and feature point tracking results of the current frame and the previous frame when performing image feature point extraction based on the initial image feature point extraction parameters;

[0049] The threshold dynamic adjustment module determines the average moving distance of feature points, the feature tracking success rate, and the number of feature point tracking based on the feature point extraction results and feature point tracking results of the current frame and the previous frame;

[0050] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate meets a first preset condition, increasing the feature point extraction threshold based on the feature point extraction threshold increase factor;

[0051] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate and the number of feature point tracking meet the second preset condition, the feature point extraction threshold is reduced based on the feature point extraction threshold reduction ratio.

[0052] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for dynamically adjusting the feature point extraction threshold of an image as described in the embodiment of the first aspect of the present invention are implemented.

[0053] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for dynamically adjusting the feature point extraction threshold of an image as described in the embodiment of the first aspect of the present invention.

[0054] The embodiment of the present invention provides a method and system for dynamically adjusting the feature point extraction threshold of an image, which regards feature extraction and feature tracking as two interdependent processes. The quality of the feature points extracted in the feature extraction stage will affect the result of feature tracking, and the result of feature tracking will be fed back to the feature extraction stage to adjust the key parameters required for feature extraction; the feature point extraction threshold is dynamically adjusted according to the current environmental conditions, so that the traditional feature extraction method can effectively extract feature points even in extreme lighting environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 is a schematic diagram of epipolar geometry of photographing the same target from different positions using a camera according to an embodiment of the present invention;

[0057] Figure 2 It is a flowchart of a method for dynamically adjusting a threshold value for extracting feature points of an image according to an embodiment of the present invention;

[0058] Figure 3 is a flow chart of a dynamic threshold adjustment method according to an embodiment of the present invention;

[0059] Figure 4 A logic diagram for adjusting and regulating the feature point extraction threshold according to an embodiment of the present invention;

[0060] Figure 5 An image in the test data according to an embodiment of the present invention and a corresponding image taken under good lighting conditions;

[0061] Figure 6 is a threshold map for feature point extraction according to an embodiment of the present invention;

[0062] Figure 7 A schematic diagram of feature point extraction from the 110th frame to the 115th frame according to an embodiment of the present invention;

[0063] Figure 8 is a graph showing the total number of feature points, the number of valid feature points and the number of invalid feature points according to an embodiment of the present invention;

[0064] Fig. 9 A schematic diagram of feature point tracking according to an embodiment of the present invention;

[0065] Fig.10 A schematic diagram of a threshold map for extracting feature points from the 800th frame to the 860th frame and a distribution diagram of feature points at certain moments according to an embodiment of the present invention;

[0066] Fig.11 is a schematic diagram of feature tracking results according to an embodiment of the present invention;

[0067] Fig.12 A schematic diagram of feature extraction thresholds, the number of all feature points, the number of invalid feature points, and the feature extraction at a selected time from the 1300th frame to the 1600th frame according to an embodiment of the present invention;

[0068] Fig.13 is a graph showing the number of effective feature points according to an embodiment of the present invention;

[0069] Fig.14 FIG. 4 is a schematic diagram of a physical structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] In the embodiments of the present application, the term "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0072] The terms "first" and "second" in the embodiments of the present application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a system, product or device comprising a series of parts or units is not limited to the listed parts or units, but optionally also includes parts or units that are not listed, or optionally also includes other parts or units inherent to these products or devices. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0073] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0074] The existing commonly used image feature points such as Harris feature, Fast feature, ORB feature, SIFT feature, SURF feature, did not take into account the impact of extreme lighting conditions on the image at the beginning of design. At the same time, it can be found that whether extracting features directly based on image pixel values ​​or based on the image response map to a specific template, it is necessary to manually set thresholds to distinguish feature points from non-feature points. The current improvements to these feature extraction methods are mainly focused on improving the computational efficiency of feature extraction and improving the accuracy of feature matching, while ignoring the impact of image quality on feature extraction under extreme lighting conditions and lighting changes.

[0075] Therefore, the embodiments of the present invention provide a method and system for dynamically adjusting the threshold value of feature point extraction of an image, so that the standard for selecting feature points can change in real time with the change of the scene, ensuring that a sufficient number of effective feature points can be extracted under any lighting conditions and in any process of changing lighting conditions. The following will be explained and introduced through multiple embodiments.

[0076] In traditional feature extraction methods, feature extraction is an independent module, and it and the subsequent feature matching are two separate independent processes. In this method, feature extraction and feature tracking are regarded as two interdependent processes. The quality of the feature points extracted in the feature extraction stage will affect the result of feature tracking, and the result of feature tracking will be fed back to the feature extraction stage to adjust the key parameters required for feature extraction. The flowchart of the dynamic threshold adjustment method proposed in the embodiment of the present invention is shown in Figure 2. The method mainly includes four steps: parameter initialization, feature extraction, feature tracking, and threshold adjustment;

[0077] like Figure 2 and Figure 3 As shown in , a method for dynamically adjusting the threshold value of feature point extraction of an image is provided according to an embodiment of the present invention, which can be applied to feature point extraction of an image, and includes:

[0078] Step S1, setting initial image feature point extraction parameters, wherein the initial image feature point extraction parameters include a feature point extraction threshold, a feature point extraction threshold increase factor, and a feature point extraction threshold decrease factor;

[0079] Specifically, the initial state of the variables used in the system is set in the parameter initialization stage, including the maximum number of feature point tracking, the minimum number of feature point tracking, the feature point tracking success rate threshold, the feature point extraction threshold, the feature point extraction threshold increase factor, the feature point extraction threshold reduction factor, and the minimum pixel movement distance. Among them, the feature point extraction threshold is the threshold that the feature point extraction method needs to distinguish between good and bad features, and its initial value is related to the selected feature extraction method.

[0080] The following uses the Harris feature detection method to introduce the entire dynamic threshold adjustment method. Harris feature finds pixels on the image that have large pixel value changes in both directions. For a given image and a neighborhood window of fixed size, the sum of the squares of the differences between the pixels before and after the translation of the window is calculated, that is, the autocorrelation function of the pixel values ​​in the neighborhood window is calculated, which is expressed as:

[0081]

[0082] Among them, p represents the coordinates of a certain pixel point in the image, d∈Ω represents the offset of the coordinates of each pixel point in the neighborhood window relative to the coordinates of the pixel point p, and w(d) represents the weight value of the corresponding offset position in the neighborhood window. I(p) represents the pixel value at the pixel point p on the image. The weight value in the neighborhood window can usually be expressed using a mean function template or a Gaussian function template. Compared with the mean function template, the weight value of the Gaussian function template in the central area will be greater than the weight value of other positions around it, which is more conducive to accurately locating the position of the extracted features. Then use Taylor expansion to make a first-order approximation of I(p+d):

[0083] I(p+d)≈I(p)+I x (p)*u+I y (p)*v(14)

[0084] Among them, I x and I y are the image gradient values ​​in the horizontal and vertical directions, respectively, and u and v are the offsets in the horizontal and vertical directions, respectively. Substituting equation (14) into equation (13), we get:

[0085]

[0086] in:

[0087]

[0088] The two eigenvalues ​​of the matrix M determine whether the current pixel is a Harris feature point. When both eigenvalues ​​are large, it is a valid feature corner point. When one eigenvalue is large and the other is small, it means that the pixel is located on the edge of an object in the image. When both eigenvalues ​​are small, it means that the point is located in a weak texture area on the image. In order to more quickly determine whether the current pixel is a qualified feature corner point based on the matrix M, the following formula is used to directly calculate the feature response:

[0089] R = det (M) - k (trace (M)) 2 (17)

[0090] Among them, det(·) represents the determinant of the matrix, trace(·) represents the rank of the matrix, and k is a constant. When this feature response R is greater than a certain threshold, the current pixel is determined to be a Harris feature point. It can be seen that the feature response of the Harris feature does not have a clear range of values. Therefore, when setting the threshold for Harris feature point extraction, it is necessary to calculate a reasonable initial threshold based on the distribution of pixel values ​​on the current image. Specifically, when the system is initialized, the feature response value of each pixel is calculated for the first frame of the input image, and then the length and height of the sub-image block are set according to the size of the input image, and the image is evenly divided into n*m sub-image blocks. The local maximum feature response value is selected in each sub-image block, and then the local maximum feature response values ​​of the n*m ​​image blocks are sorted, and a feature response value R is selected according to a ratio α∈(0,1). i , so that among all local maximum eigenvalues, α*n*m values ​​are less than Ri, and the other (1-α)*n*m values ​​are greater than R i , and this characteristic response value R iAs the initial feature extraction threshold. This initial value selection method is applicable to methods that extract features based on the size of feature response values, such as Harris features. For Fast features, ORB features, SIFT features, SUFT features, and other methods that extract features based on local pixel value differences or feature response value differences, you can use the aforementioned method to extract Harris features when the first frame of the image is input. Since these feature extraction methods all look for corner points on the image, these extracted Harris features can be regarded as reliable corner points and should also be detected by other methods. For SIFT features and SUFT features, the corresponding feature response values ​​can be calculated at the positions of these Harris feature points, and the differences between the feature response values ​​of these points and the surrounding feature response values ​​are calculated. The difference values ​​calculated at each point are averaged as the initial feature extraction threshold. For FAST features and ORB features, the differences between the pixel values ​​at the positions of these Harris feature points and the surrounding pixel values ​​are directly calculated, and all the calculated differences are averaged as the initial feature extraction threshold. The maximum number of feature points tracked is used to limit the number of feature points that are continuously tracked between each frame in the system. This is because when the density of feature points distributed on the image exceeds a certain amount, a large number of feature points provide redundant information, and the amount of additional information provided for the subsequent solution of the camera position is very small. At the same time, it takes a lot of time to track a large number of feature points in each frame, which cannot meet the real-time requirements of the system. Therefore, it is necessary to reasonably set the maximum number of feature points for feature tracking according to the image size and available computing power. On the contrary, if the number of feature points used for feature tracking is too small, it means that the number of feature points that can be matched at present will be very small, which provides too few constraints for the subsequent estimation of the camera's posture, which can easily cause inaccurate estimation, especially when there are mismatches in the limited matching results. Therefore, it is also necessary to set the minimum number of feature point tracking. In the experiment of the embodiment of the present invention, the maximum number of feature point tracking and the minimum number of feature point tracking are set to 300 and 30, respectively. Since the method proposed in the embodiment of the present invention dynamically adjusts the threshold of feature extraction, in order to prevent the feature tracking of two frames of the same image from being continuously performed and the threshold from being continuously adjusted when the camera is not moving, causing the threshold to continuously change when the environment has not changed, it is necessary to set the minimum pixel movement distance. When it is found that the movement of the pixel point is less than the set minimum pixel movement distance according to the tracking result of the feature point, it means that neither the camera nor the scene is moving, and the feature extraction threshold should not be adjusted at this time.

[0091] Step S2, determining the feature point extraction results and feature point tracking results of the current frame and the previous frame when performing image feature point extraction based on the initial image feature point extraction parameters;

[0092] After the parameter initialization is completed, the feature extraction is performed. Since reasonable initial values ​​are selected for the necessary parameters in the initialization stage, the situation in which the feature points cannot be extracted and the situation in which too dense feature points are extracted will not occur in the feature extraction stage. In addition, it can ensure that the extracted features are basically distributed on the image at the mean value, and will not be densely concentrated in a certain area, that is, it is guaranteed that a sufficient number of feature points are extracted and the feature points are evenly distributed. In the feature tracking stage, this method uses the pyramid optical flow method with a distortion model to track features. For the previous and next two frames of images, in the case of extremely small three-dimensional camera motion and extremely short time, it can be assumed that the pixel point on the previous frame of the image has moved a short distance on the two-dimensional plane, and the pixel value has not changed. Under this assumption, the problem can be quantified as for a pixel point u=[u x u y ], it can be at point v = [u x +d x u y +d y ] is found. In addition, due to the movement of the camera, there is an affine deformation between the two captured images, so another affine change model is considered:

[0093]

[0094] Among them, d xx , d xy , d yx , d yy are the four parameters of the image affine deformation model. It can be considered that each pixel in a local image block moves on the image plane with the same two-dimensional motion, so the objective function of optical flow tracking feature points can be established as:

[0095]

[0096] Where x represents the coordinate of a pixel point, and Ω represents each pixel point in the sub-image. In order to more accurately estimate the motion of feature points on the two-dimensional plane, the input image is used to generate new images of different sizes from large to small using an image pyramid. Each layer of the image is obtained by downsampling the image of the next layer. Downsampling will use linear interpolation to generate a new image with half the length and height of the input image. With the image pyramid, feature point tracking can be started from the highest layer of the image, and the tracking results can be passed to the next layer to continue feature tracking. By continuously optimizing the motion of pixels on the image layer by layer, the motion of pixels on the two-dimensional image plane can be estimated from coarse to fine. For each layer of the pyramid image, the target equation, that is, equation (19), is solved using an optimization method, and the target function is differentiated with respect to all independent variables:

[0097]

[0098] Ideally, when each variable is optimized to the optimal point, there are:

[0099] D optimum =[0 0 0 0 0 0] (21)

[0100] Expanding formula (20) yields:

[0101]

[0102] in:

[0103]

[0104] The first-order Taylor expansion of J(Ax+d) in equation (22) is performed to linearize it:

[0105] J(Ax+d)≈J(x)+D*Δx(24)

[0106] in:

[0107] Δx=[d x d y d xx d xy d yx d yy ] (25)

[0108] Substituting equation (24) into equation (22), we get:

[0109]

[0110] In the Jacobian matrix D * All matrix elements in are the horizontal and vertical derivatives of image J at a certain pixel, that is, the horizontal gradient value and vertical gradient value of image J at a certain pixel. Since there is usually a very small change between image I and image J, the gradient value on image J can be approximately replaced by the gradient value of image I at the corresponding position. The advantage of doing this is that a large amount of calculation can be reduced in the subsequent calculation process. The gradient value calculation formula of image I is:

[0111]

[0112] In formula (28), x and y represent the horizontal and vertical coordinates of the image respectively. In order to prevent the influence of noise on the gradient calculation, the Sobel operator is used to calculate the gradient. The mathematical form of the Sobel operator is:

[0113]

[0114] The value calculated by the Sobel operator needs to be normalized before it can be used as the gradient value of a certain pixel. In order to solve a set of affine transformation parameters and translation parameters to minimize the value of the objective function, let equation 5-26 equal to zero to obtain:

[0115]

[0116] in:

[0117]

[0118] Calculate equation (30) to get the adjusted value of the independent variable, and then add the adjusted value to the initial value of the independent variable to get the new value. Continuously iterate and calculate the adjusted value of the independent variable, and continuously update the value of the independent variable until convergence. In equation (31), it can be seen that due to the coefficient matrix ∑ x ∈ Ω D*D* T The values ​​in are only related to image I, and image I is fixed, so the gradient value on image I is fixed, which makes the coefficient matrix ∑ on the left side of the equation x ∈ Ω D*D* T It only needs to be calculated once at the beginning of the iteration, which saves a lot of computing time. Note that since the independent variables are constantly fine-tuned during the iteration process, and J(x) always represents the value corresponding to the current independent variable, the matrix I(x)-J(x) on the right side of the equation changes with each iteration and needs to be recalculated in each round of iteration. Solve the system of equations to obtain the update amount of each independent variable and update the independent variable:

[0119]

[0120] The change of the independent variable is solved iteratively until a preset convergence condition is reached, such as the update amount of the translation is small enough, or the number of iterations reaches the set upper limit. Since the image pyramid is used, the solution of the independent variable will be solved layer by layer starting from the highest pyramid. When the affine transformation parameters and translation parameters calculated under a certain layer of pyramid need to be passed to the next layer of pyramid for further calculation, since the affine transformation is not affected by the scale, while the translation is affected by the scale, the affine transformation matrix A does not need to be changed, and the translation matrix d needs to be multiplied by the scale, that is, doubled. When the calculation of the bottom layer of the pyramid is completed, the final set of affine transformation parameters and translation parameters that need to be solved are obtained. With this set of parameters, the new position of a feature point on the image can be found on the next frame of the image, that is, the matching feature point is found. Note that for each feature point on the image, the above optical flow method needs to be used for feature tracking separately, so the number of feature points that need to be tracked will directly affect the amount of calculation. The optical flow method can always find a local optimal matching point for any feature point on another frame image by tracking feature points, even if the tracked feature point is a meaningless pixel point, such as a point on a white wall. Such matching results are usually inaccurate. Therefore, in order to eliminate these inaccurate feature matching results, this method uses the RANSAC algorithm based on the two-point method to quickly screen the matching results.

[0121] The main use of the RANSAC algorithm is to estimate the parameters of a preset mathematical model from a set of data with outliers through an iterative method, and it can also be used to find outliers in given data. It assumes that most of the data in the given data are non-outliers and conform to a fixed mathematical model. The parameters of the mathematical model are estimated by continuously randomly sampling data, and then other data are used to verify the correctness of the model. The more data that meets the estimated model, the higher the probability that the model is correct. When the RANSAC algorithm is usually used to filter the mismatch of image features, the model used is formula (10), that is, each iteration solves a basic matrix. Since the basic matrix has 9 unknowns and the matrix has one degree of freedom, at least 8 sets of matching feature points are required for each solution, which has certain requirements on the number of feature points and the amount of calculation. In this method, in order to improve the calculation speed of the RANSAC algorithm and reduce the dependence on the number of feature points, the RANSAC algorithm based on two sets of matching feature points is used, and the embodiment of the present invention needs to use the motion information of the gyroscope in addition. When there is a gyroscope in the system, as long as the system moves, the rotation of the system in three dimensions in space can be obtained. The corresponding rotation matrices can be expressed as R x (Δφ), R y (Δθ), R z(Δψ), then the rotation matrix of the camera corresponding to the two frames of images in three-dimensional space can be expressed as:

[0122] R=R x (Δφ)·R y (Δθ)·R z (Δψ) (35)

[0123] According to formula (35), the camera direction corresponding to one frame of the image can be turned to the camera direction corresponding to the other frame of the image. Therefore, the relative motion of the camera between the two frames of the image only exists in translation. Usually, the parameters of the camera are known, so the essential matrix E can be selected when selecting the mathematical model required by the RANSAC algorithm. The essential matrix consists of the translation and rotation of the camera in three-dimensional space:

[0124] E=[T] × R (36)

[0125] in:

[0126]

[0127] Where T x , T y , T z Represents the translation of the camera in three directions in three-dimensional space. When the camera direction corresponding to one frame of image is turned to the camera direction corresponding to another frame of image, the relative motion of the camera in three-dimensional space becomes:

[0128] T=ρ[sin(β)·cos(α)-sin(β)·sin(α) cos(β)] T (38)

[0129] R=I 3 (39)

[0130] Among them, ρ represents the modulus of the translation T in three-dimensional space, α and β represent two angles respectively, and I 3 represents a three-dimensional unit vector. Substituting equation (38) into (36) simplifies the essential matrix to:

[0131]

[0132] It can be seen that there are only two unknowns left in the essential matrix, so the feature correspondence between the two frames of images needs to satisfy:

[0133] x 1 (y 0 ·cos(β)+z 0 ·sin(α)sin(β))-y 2 ·(x 0cos(β)-z 0 ·cos(α)sin(β))-z 2 ·(y0cos(α)

[0134] sin(β)+x 0 ·sin(α) sin(β))=0 (41)

[0135] At least two sets of matched feature points can be used to solve α and β, and then a set of relative motion and direction of the camera in three-dimensional space can be obtained. Based on this set of information, other feature points are projected from one frame of image to another frame of image by three-dimensional projection and the back-projection error is calculated. A reasonable threshold is set to distinguish whether this set of matched feature points supports the currently estimated camera motion parameters. When using this RANSAC method based on two sets of matched feature points, two sets of matched feature points are randomly selected each time. In order to ensure that the two sets of matched feature points provide enough information for solving the equation, a distance threshold is set to ensure that there is a large enough distance between the two sets of matched feature points. If it is found that the distance between the two sets of matched feature points currently selected is less than the set distance threshold, the two sets of matched feature points are reselected. The random selection of two sets of matched feature points, the solution of the camera motion parameters, and the verification of how many of the other matched feature points meet the solved camera motion parameters are repeatedly executed. When the upper limit of the number of iterations is reached, a set of camera motion parameters that can meet the most matched feature points is selected. At the same time, an additional constraint is added, that is, the movement of the system should be continuous. Since image data is often collected at a high frame rate in the SLAM system, it can be assumed that the movement direction will not change significantly in a very short time. Compare the current estimated movement direction of the camera in space with the movement direction estimated at the previous moment. If the difference is greater than a preset threshold, skip the current set of camera motion parameters and switch to the next set. At the same time, filter out feature tracking results that do not support this set of parameters based on the selected set of camera motion parameters.

[0136] Step S3, determining the average moving distance of feature points, the feature tracking success rate, and the number of feature points tracked based on the feature point extraction results and feature point tracking results of the current frame and the previous frame;

[0137] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate meets a first preset condition, increasing the feature point extraction threshold based on the feature point extraction threshold increase factor;

[0138] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate and the number of feature point tracking meet the second preset condition, the feature point extraction threshold is reduced based on the feature point extraction threshold reduction ratio.

[0139] Among them, Figure 4 As shown in , the first preset condition is: the feature tracking success rate is not greater than the feature point tracking success rate threshold, or the feature tracking success rate is greater than the feature point tracking success rate threshold and the feature point tracking number is greater than the maximum feature point tracking number;

[0140] The second preset condition is: the feature tracking success rate is greater than the feature point tracking success rate threshold and the number of feature point tracking is less than the minimum feature point tracking number, or the feature tracking success rate is greater than the feature point tracking success rate threshold and the number of feature point tracking is not greater than the maximum feature point tracking number and the feature points are unevenly distributed.

[0141] After the feature tracking stage is completed, the threshold adjustment stage begins. The main purpose of the threshold adjustment stage is to decide how to adjust the feature extraction threshold based on the current feature tracking results. First, check the distribution of the currently successfully tracked feature points on the image. Divide the image into 4 equal areas of 2*2, and count the number of successfully tracked feature points in each area. If the number of feature point tracking in any area is less than a preset threshold, it is considered that the features that are still effectively tracked are unevenly distributed. In this implementation, the threshold is set to 5. Then, based on the average moving distance of the features calculated in the feature tracking stage on the image plane, it is determined whether the camera has moved. When the average moving distance is less than 2 pixels, it is considered that the camera has not moved compared to the previous moment, and the feature extraction threshold is not adjusted at the current moment. On the contrary, if the camera is detected to have moved, a series of conditions are used to decide how to adjust the feature extraction threshold. Specifically, first use the following formula to calculate the tracking success rate of the current feature:

[0142]

[0143] In the above formula, n f Indicates the number of feature points obtained before the current feature point tracking, including the feature points successfully tracked at the previous moment and the new feature points extracted from the feature points at the current moment; n t Indicates the number of feature points successfully tracked at the current moment.

[0144] If the current feature tracking success rate is less than the feature point tracking success rate threshold, it means that there are many meaningless feature points among all the current feature points, such as points on the white wall. This is because the current feature extraction threshold is too low, resulting in the feature response values ​​at some positions in the weak texture area also reaching the set threshold, so the feature extraction threshold should be increased. If the current feature tracking success rate is greater than the feature point tracking success rate threshold, it means that the current feature points are all valid, then further determine whether the number of currently tracked feature points is greater than the maximum number of feature point tracking. If it is greater, it means that there are a lot of textures in the current scene and the current feature extraction threshold can be set relatively large. The information provided by too dense feature points contains a lot of redundant components, which will not help improve the accuracy of camera pose estimation, but will reduce the efficiency of calculation. Therefore, it is also necessary to increase the feature extraction threshold at this time. If the number of currently tracked feature points is less than the maximum number of feature point tracking, then further determine whether the number of currently tracked feature points is too small. Too few tracked feature points means too few feature matching results. Such estimation of camera pose cannot provide enough constraints, which may lead to inaccurate estimation results. Therefore, if it is found that the number of feature points currently tracked is less than the minimum number of feature points tracked, it is necessary to lower the feature extraction threshold. On the other hand, if it is found that the currently extracted features are concentrated in a certain area on the image, this means that the current feature distribution is not uniform enough, and it is usually hoped that the available features are evenly distributed on the image, which is conducive to improving the accuracy of camera pose estimation. Therefore, in this case, it is also necessary to reduce the feature extraction threshold. For all the above rules, when it is necessary to increase the feature extraction threshold, multiply the current feature extraction threshold by a fixed magnification, that is, the feature point extraction threshold increase magnification, and when it is necessary to reduce the feature extraction threshold, multiply the current feature point extraction threshold by the feature point extraction threshold reduction magnification. In the embodiment of the present invention, the feature point extraction threshold increase magnification and the feature point extraction threshold reduction magnification are set to 1.1 and 0.9 respectively. The selection of the magnification needs to balance the speed and accuracy of the threshold adjustment. Since the camera usually collects images at a high frequency, the changes in image information will be smoother, so a smaller adjustment magnification is selected here.

[0145] In order to verify the effectiveness of the dynamic threshold adjustment method proposed in the embodiment of the present invention, a section of image data under poor ambient lighting conditions was captured in an indoor scene, and the frame rate of the data was 20 frames per second. Harris features were used in the experiment to extract image features, and the pyramid optical flow method was used to track feature points. In the embodiment of the invention, the numerical value was scaled down when calculating the Harris feature response during the experiment. At the initial moment, the feature extraction threshold was set to 600, and this value will be continuously adjusted during the operation of the system. Other pre-set parameters include the minimum number of feature point tracking set to 30, the feature point tracking success rate set to 0.85, the maximum number of feature point tracking set to 200, the feature point extraction threshold increase factor and the feature point extraction threshold reduction factor set to 1.1 and 0.9 respectively, and the minimum pixel movement distance set to 1. The scenes in the entire video include areas with obvious textures, white wall areas, and areas with uneven lighting. Due to the poor lighting conditions in the environment, the captured images are generally dark, which is a very big challenge for traditional visual SLAM algorithms. Some images in the data such as Figure 5 shown.

[0146] The first row in the figure is the images in the test data, and the second row is the images of the corresponding positions taken at similar positions under good lighting conditions. As can be seen from the figure, some of the scenes originally had certain texture information, but due to the dark environment, the images taken by the camera in the scenes without lights were almost black, while the images taken in the scenes with lights were bright in some parts and dark around. Such data is very challenging for the robustness of visual feature extraction and tracking. In this experiment, only the performance of feature extraction and matching was tested. For each input image, the feature points in the system that are still in a trackable state will be tracked using the optical flow method on the new image, and the positions of these feature points on the new image will be found. Then, new feature points will be extracted for the current image based on the results of feature tracking. In the process of traversing the entire data, the changes in the feature point extraction threshold in the system are shown as follows. Figure 6 shown.

[0147] As can be seen from the figure, when the system just started running, the feature point extraction threshold did not change because the camera did not move. However, due to the high feature point extraction threshold at this time, the number of extracted feature points was very small. When the camera started to move, the system gradually began to lower the feature point extraction threshold. From the 60th frame to the 90th frame, the threshold quickly dropped to a relatively reasonable range. As the feature point extraction threshold decreased, feature points could be gradually extracted from each new image frame. As shown by the gray line in the figure, as the gradient decreases, the number of feature points that can be extracted also increases and reaches a peak between the 110th and 115th frames. As shown by the second line in the figure, as the number of feature points that can be extracted from each frame increases, the number of feature points that can be effectively tracked in the system also increases. This is also the effective information that feature point SLAM really needs, that is, the correspondence between feature points between image frames is increasing. Although more feature points are extracted from the image between the 110th and 115th frames, most of the feature points are extracted from moving objects in the scene and are filtered out during the feature tracking stage. Therefore, the number of feature points that can be effectively tracked in the system does not increase. Figure 7 As shown in the figure. The gray dots in the figure represent the newly extracted feature points in each frame, and a large part of the feature points fall on the moving people and a shaking laptop. Since no additional features will be extracted in the vicinity of the existing feature points when extracting features from a new image, it can be seen from frame 130 onwards that there are a certain amount of feature points that can be tracked in the system and they begin to maintain stability, while the feature points extracted from the image begin to decrease and remain at a low level. From the change of the feature point extraction threshold in the entire sequence, it can be seen that the threshold is constantly changing with the change of the environment, but because the environment is always in a darker state, the threshold is always maintained at a low level. During the entire process of the feature point extraction threshold change, at certain moments, the threshold will also increase to dozens of times the lowest level. At these moments, the camera moves to an environment with relatively good lighting conditions, and the originally low feature point extraction threshold will lead to a large number of extracted feature points. On the other hand, it will also increase the number of meaningless feature points extracted, which will lead to a decrease in tracking rate. These two conditions make the feature point extraction threshold continuously increased during this period to meet the current environment. From the 2400th frame to the 2700th frame, the number of feature points in the system at each moment (the number of feature points that can be tracked at the previous moment plus the number of newly extracted feature points), the number of feature points that can be tracked after the feature tracking at the current moment, and the number of feature points that are filtered at the current moment are as follows: Figure 8As shown in Figure 2. It can be seen that the number of feature points exceeds the set threshold at certain moments, and there are a certain number of invalid feature points at each moment. The feature tracking situation at the 2500th, 2600th, and 2700th frames is shown in Figure 2. Fig. 9 As shown in the figure. Each row is a group of images, and the left and right sides of each group of images are two consecutive frames of images. The left image shows the valid feature points that existed in the system at the previous moment, and the gray points indicate that these points can still be correctly tracked in the current image. The right image shows the result of feature point tracking in the current image. In actual application, the green line segment can be used to represent the trajectory of the correctly tracked feature points, and the red line segment represents the wrong tracking result. The feature point tracking results shown here have not been filtered by the RANSAC algorithm. In the final feature tracking results, the feature points in the rectangular box will also be filtered out because they fall on a moving person. It can be seen from the figure that more valid feature points will be extracted in areas with relatively good ambient lighting conditions. At this time, increasing the feature point extraction threshold can effectively prevent excessive feature points from being extracted while ensuring a sufficient number of valid feature matches. At some moments, the scene presents a bright and rich texture in some local areas, while other areas are relatively dark. This situation usually occurs when the camera moves from an area with better lighting conditions to an area with poorer lighting conditions. This situation also exists in the 800th to 860th frames of the test data. During this period, the changes in the feature point extraction threshold in the system and the distribution of the effective feature points in the system on the image are shown in Figure 2. Fig.10 As shown in the figure. As can be seen from the figure, there is an area with good lighting conditions in the environment between the 800th frame and the 825th frame. Therefore, the feature point extraction threshold is relatively high at this time, and the feature points are mainly distributed in the brighter area of ​​the image. Since the upper area of ​​the image is darker, feature points cannot be extracted at a higher feature extraction threshold. In order to make the extracted features more evenly distributed on the image, the feature point extraction threshold is constantly decreasing. From the 830th frame to the 845th frame, it can be seen that the distribution of feature points gradually extends to the darker area. In the process of switching from brighter scenes to darker scenes, the feature points can always be evenly distributed on the entire image, and there will be no situation where feature points are concentrated in a small area or feature points cannot be extracted. On the other hand, the feature extraction threshold has not dropped to a too low level. As can be seen from the figure, even though the feature extraction threshold has been decreasing, no invalid feature points have been extracted in the white wall area on the left side of the image. The change of the feature point extraction threshold can always ensure that enough feature points can be extracted while ensuring the quality of the feature points. Some feature tracking results are shown in Figure 1. Fig.11 shown.

[0148] For visual SLAM algorithms, weak texture areas usually make it impossible for the algorithm to extract feature points. Since the dynamic threshold adjustment method proposed in the embodiment of the present invention will continuously reduce the feature extraction threshold when there are fewer valid feature points in the system, the new challenge becomes how to prevent the feature point extraction threshold from being reduced blindly in featureless scenes. Fig.12 The figure shows the feature point extraction threshold at each moment between the 1300th frame and the 1600th frame, the number of all feature points (the feature points that were effectively tracked at the previous moment and the feature points newly extracted in the current frame), and the number of invalid feature points. During this period of time, the camera was in a weakly textured environment, with most areas being white walls. It can be seen that although the threshold is constantly fluctuating, it always remains at a very low level, and the number of valid features in the system always fluctuates around zero. In this extreme environment, even a slight change in the threshold will result in a huge change in the number of feature points that can be extracted. Figure 5-11 The feature extraction from frame 1436 to frame 1438 is particularly shown. It can be seen that the feature point extraction threshold is slightly reduced in frame 1437, which results in a large number of invalid feature points being extracted. These feature points are filtered out in the feature tracking stage, which makes the feature tracking rate very small. Therefore, the threshold is raised a little bit again, so these invalid feature points are no longer extracted in the next frame of the image. It can be seen that since the number of valid features in the weak texture area is very small, the method proposed in the embodiment of the present invention will continuously try to lower the threshold to find possible valid feature points, which eventually causes the feature point extraction threshold to oscillate within a small range, and suddenly extract a large number of invalid feature points at certain moments. To address this problem, the frequency of changes in the feature point extraction threshold or the amplitude of adjustments can be artificially reduced when the feature extraction threshold is at a very low level. Finally, in Fig.13 The number of valid feature points in the system at each moment is shown in . It can be seen that under such data, the system can still extract a sufficient number of valid feature points in most areas, and the number of feature points is basically maintained within the set maximum number of feature points. In those weak texture areas, the system can also effectively identify invalid feature points and filter them out. The effective operation of this system also reflects from the side that under those extreme lighting conditions, even if the textures that originally exist in the scene are difficult for the human eye to recognize in the image, as long as the appropriate feature response threshold is found, the computer can effectively find these features and use them. In the experiment, we also tried to use the latest open source method, ORB-SLAM3, to test the data. This method was unable to extract feature points on such data, resulting in the system being unable to complete initialization and run.

[0149] The embodiment of the present invention further provides a system for dynamically adjusting the threshold value of feature point extraction of an image, based on the method for dynamically adjusting the threshold value of feature point extraction of an image in the above embodiments, comprising:

[0150] An initialization setting module is used to set initial image feature point extraction parameters, wherein the initial image feature point extraction parameters include a feature point extraction threshold, a feature point extraction threshold increase factor, and a feature point extraction threshold decrease factor;

[0151] A feature extraction and tracking module, used to determine the feature point extraction results and feature point tracking results of the current frame and the previous frame when performing image feature point extraction based on the initial image feature point extraction parameters;

[0152] The threshold dynamic adjustment module determines the average moving distance of feature points, the feature tracking success rate, and the number of feature point tracking based on the feature point extraction results and feature point tracking results of the current frame and the previous frame;

[0153] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate meets a first preset condition, increasing the feature point extraction threshold based on the feature point extraction threshold increase factor;

[0154] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate and the number of feature point tracking meet the second preset condition, the feature point extraction threshold is reduced based on the feature point extraction threshold reduction ratio.

[0155] Based on the same concept, the embodiment of the present invention also provides a schematic diagram of a physical structure, such as Fig.14 As shown, the server may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the steps of the method for dynamically adjusting the threshold value of feature point extraction of an image as described in the above embodiments. For example, it includes:

[0156] Step S1, setting initial image feature point extraction parameters, wherein the initial image feature point extraction parameters include a feature point extraction threshold, a feature point extraction threshold increase factor, and a feature point extraction threshold decrease factor;

[0157] Step S2, determining the feature point extraction results and feature point tracking results of the current frame and the previous frame when performing image feature point extraction based on the initial image feature point extraction parameters;

[0158] Step S3, determining the average moving distance of feature points, the feature tracking success rate, and the number of feature points tracked based on the feature point extraction results and feature point tracking results of the current frame and the previous frame;

[0159] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate meets a first preset condition, increasing the feature point extraction threshold based on the feature point extraction threshold increase factor;

[0160] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate and the number of feature point tracking meet the second preset condition, the feature point extraction threshold is reduced based on the feature point extraction threshold reduction ratio.

[0161] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0162] Based on the same concept, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores a computer program, which includes at least one code segment, and the at least one code segment can be executed by a main control device to control the main control device to implement the steps of the method for dynamically adjusting the feature point extraction threshold of an image as described in the above embodiments. For example, it includes:

[0163] Step S1, setting initial image feature point extraction parameters, wherein the initial image feature point extraction parameters include a feature point extraction threshold, a feature point extraction threshold increase factor, and a feature point extraction threshold decrease factor;

[0164] Step S2, determining the feature point extraction results and feature point tracking results of the current frame and the previous frame when performing image feature point extraction based on the initial image feature point extraction parameters;

[0165] Step S3, determining the average moving distance of feature points, the feature tracking success rate, and the number of feature points tracked based on the feature point extraction results and feature point tracking results of the current frame and the previous frame;

[0166] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate meets a first preset condition, increasing the feature point extraction threshold based on the feature point extraction threshold increase factor;

[0167] If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate and the number of feature point tracking meet the second preset condition, the feature point extraction threshold is reduced based on the feature point extraction threshold reduction ratio.

[0168] Based on the same technical concept, an embodiment of the present application also provides a computer program, which, when executed by a main control device, is used to implement the above method embodiment.

[0169] The program may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor.

[0170] Based on the same technical concept, the embodiment of the present application further provides a processor, which is used to implement the above method embodiment. The above processor can be a chip.

[0171] In summary, the embodiment of the present invention provides a method and system for dynamically adjusting the feature point extraction threshold of an image, which regards feature extraction and feature tracking as two interdependent processes. The quality of the feature points extracted in the feature extraction stage will affect the results of feature tracking, and the results of feature tracking will be fed back to the feature extraction stage to adjust the key parameters required for feature extraction; the feature point extraction threshold is dynamically adjusted according to the current environmental conditions, so that the traditional feature extraction method can effectively extract feature points even in extreme lighting environments.

[0172] The various embodiments of the present invention can be combined arbitrarily to achieve different technical effects.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidStateDisk), etc.

[0174] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamically adjusting the threshold of feature point extraction of an image, characterized in that: include: Step S1, setting initial image feature point extraction parameters, wherein the initial image feature point extraction parameters include a feature point extraction threshold, a feature point extraction threshold increase factor, and a feature point extraction threshold decrease factor; Step S2, determining the feature point extraction results and feature point tracking results of the current frame and the previous frame when performing image feature point extraction based on the initial image feature point extraction parameters; Step S3, determining the average moving distance of feature points, the feature tracking success rate, and the number of feature points tracked based on the feature point extraction results and feature point tracking results of the current frame and the previous frame; If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate meets a first preset condition, increasing the feature point extraction threshold based on the feature point extraction threshold increase factor; If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate and the number of feature point tracking meet the second preset condition, the feature point extraction threshold is reduced based on the feature point extraction threshold reduction ratio.

2. The method for dynamically adjusting the threshold value of feature point extraction of an image according to claim 1, characterized in that: In the step S1, the initial image feature point extraction parameters also include a maximum feature point tracking number, a minimum feature point tracking number, and a feature point tracking success rate threshold.

3. The method for dynamically adjusting the threshold value of feature point extraction of an image according to claim 2, characterized in that: The first preset condition is: the feature tracking success rate is not greater than the feature point tracking success rate threshold, or the feature tracking success rate is greater than the feature point tracking success rate threshold and the feature point tracking number is greater than the maximum feature point tracking number; The second preset condition is: the feature tracking success rate is greater than the feature point tracking success rate threshold and the number of feature point tracking is less than the minimum feature point tracking number, or the feature tracking success rate is greater than the feature point tracking success rate threshold and the number of feature point tracking is not greater than the maximum feature point tracking number and the feature points are unevenly distributed.

4. The method for dynamically adjusting the threshold value of feature point extraction of an image according to claim 1, characterized in that: In the step S3, if it is determined that the camera has moved based on the average moving distance of the feature points, the method specifically includes: If the average moving distance of the feature points of the current frame image relative to the previous frame image is less than a preset moving threshold, it is determined that the corresponding camera has not moved; If the average moving distance of the feature points of the current frame image relative to the previous frame image is not less than a preset moving threshold, it is determined that the corresponding camera moves.

5. The method for dynamically adjusting the threshold value of feature point extraction of an image according to claim 1, characterized in that: In the step S3, if it is determined that the camera has not moved based on the average moving distance of the feature points, the feature extraction threshold is not adjusted at the current moment.

6. The method for dynamically adjusting the threshold value of feature point extraction of an image according to claim 3, characterized in that: In step S3, when judging whether the feature point distribution in the second preset condition is met, the image is divided into several areas of equal size, and the number of successfully tracked feature points in each area is counted respectively. If the number of feature points in any area is less than the preset feature point number threshold, it is judged that the feature points currently being effectively tracked are unevenly distributed.

7. The method for dynamically adjusting the threshold value of feature point extraction of an image according to claim 3, characterized in that: In step S3, the feature tracking success rate is: In the above formula, n f Indicates the number of feature points obtained before the current feature point tracking, including the feature points successfully tracked at the previous moment and the new feature points extracted from the feature points at the current moment; n t Indicates the number of feature points successfully tracked at the current moment.

8. A system for dynamically adjusting the threshold of feature point extraction of an image, characterized in that: include: An initialization setting module is used to set initial image feature point extraction parameters, wherein the initial image feature point extraction parameters include a feature point extraction threshold, a feature point extraction threshold increase factor, and a feature point extraction threshold decrease factor; A feature extraction and tracking module, used to determine the feature point extraction results and feature point tracking results of the current frame and the previous frame when performing image feature point extraction based on the initial image feature point extraction parameters; The threshold dynamic adjustment module determines the average moving distance of feature points, the feature tracking success rate, and the number of feature point tracking based on the feature point extraction results and feature point tracking results of the current frame and the previous frame; If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate meets a first preset condition, increasing the feature point extraction threshold based on the feature point extraction threshold increase factor; If it is determined that the camera has moved based on the average moving distance of the feature points, and the feature tracking success rate and the number of feature point tracking meet the second preset condition, the feature point extraction threshold is reduced based on the feature point extraction threshold reduction ratio.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for dynamically adjusting the feature point extraction threshold of an image as claimed in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for dynamically adjusting the threshold value of feature point extraction of an image as claimed in any one of claims 1 to 7 are implemented.

Images