Visual odometer scale estimation method and device, electronic equipment and storage medium

Abstract

The invention provides a visual odometer scale estimation method and device, electronic equipment and a storage medium, and is applied to the technical field of navigation positioning, and the method comprises the steps: obtaining original image information, recognizing feature points, carrying out the matching in continuous frames, tracking the changes of the feature points in the continuous frames, constructing an essential matrix or a basic matrix through the matched feature points, and carrying out the scale estimation of a visual odometer. Calculating the normalized position and attitude variation of the feature points to obtain an inter-frame pose, restoring the three-dimensional coordinates of the feature points according to the inter-frame pose, and performing plane construction and ground selection to provide spatial reference for scale estimation; and the visual odometer scale is calculated by comparing the real height between the plane where the visual camera is located and the real ground and the ground elevation between the constructed plane and the selected ground. The method is small in calculation amount and high in scale estimation precision.

Description

Technical Field

[0001] The present application relates to the field of navigation and positioning technology, and in particular to a visual odometer scale estimation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of intelligent and unmanned technologies, the requirements for the accuracy and stability of navigation and positioning technologies are increasing. As representatives of intelligent systems, vehicle-mounted and airborne equipment play an important role in many fields such as autonomous driving, precision agriculture, logistics distribution, and aerial monitoring. These application scenarios are highly dependent on navigation and positioning technologies, requiring not only accurate positioning under open skies, but also high reliability and high accuracy in complex environments such as urban canyons, tunnels, and indoors.

[0003] The traditional solution is to use GNSS / IMU combined positioning for combined positioning. This method degenerates into pure inertial navigation in the case of long-term loss of lock, resulting in large positioning errors. The existing solution is to add visual information on the basis of this combined navigation, but the traditional visual odometry (VO) solves the position without scale information, which makes the positioning information unable to truly reflect the motion position. The aircraft altitude is calculated by matching the image obtained by the visual odometry system with the satellite map, and then the scale transformation between the visual odometry and the real world is calculated by the aircraft altitude and the average scene depth, so as to realize the application of visual odometry in the visual positioning of the aircraft. However, the current mainstream method is to use visual inertial odometry (VIO) to solve the position divergence problem in GNSS denial scenarios. The basic principle of visual inertial odometry is to first use IMU and VO to solve the position for inter-frame changes, then align and jointly initialize the solved positions, use IMU pre-integration to give scale information to the camera, and finally solve the IMU pre-integration residual equation, visual reprojection error equation and marginalization construction optimization equation through graph optimization to obtain the real position with scale information.

[0004] However, this method has three defects: 1. The accuracy of scale estimation is more dependent on the effect of joint initialization. If the initial state of the vehicle / UAV movement does not stimulate the IMU enough, it will lead to large scale estimation accuracy errors. 2. Since the position estimation of the IMU itself has a large cumulative error, even if it is successfully initialized, there will be a more obvious cumulative error during the subsequent movement of the vehicle. 3. The fusion method of IMU and VO is computationally intensive. In addition to the joint initialization part, the subsequent optimization of reprojection error, IMU pre-integration and marginalization consumes more resources, affecting the real-time positioning output of the algorithm on embedded devices.

[0005] Therefore, when performing navigation and positioning in the prior art, the VIO calculation amount is large and the position output accuracy is unstable. Summary of the invention

[0006] In view of the above shortcomings of the prior art, this application provides a visual odometer scale estimation method, device, electronic device and storage medium, which are applied to the field of navigation and positioning technology. By combining image feature extraction, pose estimation, plane construction, ground selection and elevation mapping, the scale estimation of the visual odometer is realized. Compared with the traditional method, the accuracy and stability of scale estimation are improved.

[0007] In a first aspect, the present application provides a visual odometry scale estimation method, the method comprising the steps of: S1: acquiring original image information, identifying feature points in the original image information, and performing feature matching on the feature points in consecutive frames; S2: constructing an essential matrix or a basic matrix according to the matched feature points, and calculating the normalized position and posture change of the feature points to obtain the inter-frame posture of the feature points; S3: restoring the three-dimensional coordinates of the feature points according to the inter-frame poses, and performing plane construction and ground selection according to the three-dimensional coordinates; S4: Obtain the real height between the plane where the visual camera is located and the real ground, and the ground elevation between the constructed plane and the selected ground, and obtain the visual odometer scale according to the ratio of the real height to the ground elevation.

[0008] The present application provides a visual odometer scale estimation method, which obtains original image information, identifies feature points and matches them in continuous frames, tracks the changes of feature points in continuous frames, and provides basic data for subsequent pose estimation; constructs an essential matrix or a basic matrix using the matched feature points, calculates the normalized position and attitude change of the feature points, obtains the inter-frame pose, and provides necessary information for three-dimensional coordinate restoration; three-dimensional coordinate restoration and plane construction: restores the three-dimensional coordinates of the feature points according to the inter-frame pose, and performs plane construction and ground selection to provide a spatial reference for scale estimation; calculates the visual odometer scale by comparing the real height between the plane where the visual camera is located and the real ground, and the ground elevation between the constructed plane and the selected ground. The method realizes the scale estimation of the visual odometer by combining techniques such as image feature extraction, pose estimation, plane construction, ground selection and elevation mapping. Compared with traditional methods, it can accurately restore visual scale information without relying on IMU data, avoids the influence of IMU cumulative error, and reduces the amount of calculation. By using ground information for scale estimation, the accuracy and stability of scale estimation are improved. This method can provide more reliable visual odometry scale information in complex environments, thereby improving the accuracy and stability of navigation positioning.

[0009] Further, step S1 includes: S11: Acquire original image information collected by a visual camera, and use Shi Tomasi corner point detection algorithm to identify the feature points in the original image information; S12: Using an optical flow tracking algorithm to track the feature points between consecutive frames to perform feature point matching.

[0010] The present application provides a visual odometer scale estimation method, which can effectively identify stable feature points in an image by using the Shi Tomasi corner detection algorithm, providing a reliable basis for subsequent feature matching. The optical flow tracking algorithm can efficiently track the movement of feature points between consecutive frames, thereby achieving feature point matching. These two technical features work together to solve the problem of feature point recognition and matching.

[0011] Furthermore, after step S12, the following steps are included: S13: comparing the position changes of the feature points in consecutive frames to calculate the disparity; S14: extracting key frames according to the disparity.

[0012] The present application provides a visual odometry scale estimation method, which can improve the estimation accuracy of motion and environment by calculating the disparity between consecutive frames to extract key frames, optimize the scale estimation process, and provide more reliable input for subsequent scale estimation and pose calculation, thereby improving the performance of the entire system.

[0013] Further, step S2 includes: S21: when the feature points in the scene are not coplanar, obtaining at least 8 pairs of matching feature points and constructing the basic matrix; S22: When the feature points in the scene are coplanar, an intrinsic parameter matrix of the visual camera is obtained to construct the intrinsic matrix; S23: Calculate the normalized position and posture change of the feature point according to the basic matrix or the essential matrix to obtain the inter-frame posture of the feature point.

[0014] The present application provides a visual odometer scale estimation method, which determines whether the feature points are coplanar and then selects an appropriate matrix construction method. When the feature points are not coplanar, at least 8 pairs of matching feature points are used to construct a basic matrix. The basic matrix can describe the symmetrical geometric relationship between the two images and is suitable for general scenes. When the feature points are coplanar, the intrinsic matrix is ​​constructed using the intrinsic parameter matrix of the visual camera. The essential matrix contains the rotation and translation information of the camera motion and is suitable for planar scenes. Through this method, the most appropriate matrix construction method can be selected according to the characteristics of the scene, so as to more accurately calculate the inter-frame pose of the feature points, providing a reliable foundation for subsequent three-dimensional coordinate restoration and plane construction. This flexible processing method improves the adaptability and accuracy of the visual odometer in different scenarios.

[0015] Further, step S3 includes: S31: restoring the three-dimensional coordinates of the feature points using a triangulation method according to the inter-frame pose; S32: Obtain the three-dimensional coordinates of the position of the visual camera at the current moment and the three-dimensional coordinates of the position of the visual camera at the previous moment, and construct a plane; S33: Calculate the normalized height of the feature point in the three-dimensional coordinates, traverse all the feature points according to the normalized height, and perform clustering, and select the plane where the feature point with the largest clustering value is located as the ground.

[0016] Further, step S33 includes: S331: arbitrarily selecting the feature point, and establishing a plane equation that passes through the feature point and is parallel to the constructed plane; S332: Calculate the normalized height of the feature point according to the equation, and take a height value of 20% of the normalized height as a preset threshold; S333: Traverse all the feature points, cluster the feature points whose height values ​​are less than or equal to the preset threshold, and select the plane where the feature points with the largest clustering value are located as the ground.

[0017] Further, step S4 includes: S41: Get the real height between the plane where the visual camera is located and the real ground. S42: Calculate the height difference between any of the feature points on the constructed plane and any of the feature points on the selected ground to obtain the ground elevation; S43: Obtain a visual odometer scale according to a ratio of the true height to the ground elevation.

[0018] In a second aspect, the present application provides a visual odometry scale estimation device, the device comprising: Feature point extraction module: obtains original image information, identifies feature points in the original image information, and performs feature matching on the feature points in consecutive frames; Posture calculation module: constructs an essential matrix or a basic matrix according to the matched feature points, and calculates the normalized position and posture change of the feature points to obtain the inter-frame posture of the feature points; Plane construction module: restore the three-dimensional coordinates of the feature points according to the inter-frame pose, and perform plane construction and ground selection according to the three-dimensional coordinates; Scale estimation module: obtains the real height between the plane where the visual camera is located and the real ground, as well as the ground elevation between the constructed plane and the selected ground, and obtains the visual odometer scale according to the ratio of the real height to the ground elevation.

[0019] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are performed.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, runs the steps in the method provided in the first aspect above.

[0021] Beneficial effects: The visual odometer scale estimation method, device, electronic device and storage medium proposed in the present application obtain the original image information, identify the feature points and match them in continuous frames, track the changes of the feature points in continuous frames, and provide basic data for subsequent pose estimation; construct the essential matrix or basic matrix using the matched feature points, calculate the normalized position and attitude change of the feature points, obtain the inter-frame pose, and provide necessary information for three-dimensional coordinate restoration; three-dimensional coordinate restoration and plane construction: restore the three-dimensional coordinates of the feature points according to the inter-frame pose, and perform plane construction and ground selection to provide a spatial reference for scale estimation; calculate the visual odometer scale by comparing the real height between the plane where the visual camera is located and the real ground, and the ground elevation between the constructed plane and the selected ground. The method realizes the scale estimation of the visual odometer by combining techniques such as image feature extraction, pose estimation, plane construction, ground selection and elevation mapping. Compared with traditional methods, it can accurately restore the visual scale information without relying on IMU data, avoid the influence of IMU cumulative error, and reduce the amount of calculation. By using ground information for scale estimation, the accuracy and stability of scale estimation are improved. This method can provide more reliable visual odometry scale information in complex environments, thereby improving the accuracy and stability of navigation positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a visual odometry scale estimation method proposed in this application.

[0023] Figure 2 This is a schematic diagram of the structure of a visual odometry scale estimation device proposed in this application.

[0024] Figure 3 A schematic diagram of the structure of the electronic device provided in this application.

[0025] Description of reference numerals: 201, feature point extraction module; 202, posture calculation module; 203, plane construction module; 204, scale estimation module; 3, electronic device; 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application usually described and marked in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0027] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first, second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0028] The following disclosure provides many different implementations or examples to achieve the purpose of the present invention, which solve the problems of the mainstream VIO technology in the prior art in that the scale estimation accuracy is overly dependent on the initialization effect, the IMU cumulative error has a large impact, and the amount of calculation is large.

[0029] Please refer to Figure 1 , the embodiment of the present application provides a visual odometer scale estimation method, the method comprising the following steps: S1: Obtain original image information, identify feature points in the original image information, and perform feature matching on the feature points in consecutive frames; S2: construct an essential matrix or basic matrix based on the matched feature points, and calculate the normalized position and posture change of the feature points to obtain the inter-frame pose of the feature points; S3: restore the three-dimensional coordinates of the feature points according to the inter-frame poses, and perform plane construction and ground selection based on the three-dimensional coordinates; S4: Obtain the true height between the plane where the visual camera is located and the real ground, as well as the ground elevation between the constructed plane and the selected ground, and obtain the visual odometer scale according to the ratio of the true height to the ground elevation.

[0030] Among them, in step S1, the original image information is obtained by the visual camera, which is a scene picture obtained by the visual camera. The detailed process of identifying the feature points in the original image information and performing feature matching on the feature points in the continuous frames is as follows: first, an image captured by the visual camera is obtained, and the image is used as the first frame image. The picture information in the image is the original image information, and then the algorithm is used to identify the feature points in the original image information; wherein the feature points are the corner points in the original image information, and finally, these feature points are tracked in the continuous frames, for example, a window is searched around the feature points of the first frame image, and the best matching points are found in the second frame, so that the feature points in the first frame can be matched in the continuous frames. Feature matching reflects the movement of feature points between continuous frames, providing a basis for calculating posture.

[0031] Wherein, in step S2, an essential matrix or a basic matrix is ​​constructed according to the matched feature points, wherein it is necessary to first determine whether the feature points are coplanar, wherein determining whether the feature points are coplanar is a prior art, and there are many methods for determining, such as vector and determinant methods, calculating the vectors of multiple feature points, and calculating the third-order determinant of the vector, if the value of the determinant is zero, then multiple feature points are coplanar, or calculating the normal vector of the plane where the feature points are located, and the feature points with consistent normal vectors are coplanar. When the feature points are not coplanar, a basic matrix is ​​constructed, and when the feature points are coplanar, the intrinsic matrix of the visual camera is constructed to construct the essential matrix, which contains the rotation and translation information of the visual camera motion, and is suitable for planar scenes. Regardless of which matrix is ​​used, the inter-frame pose can be obtained by calculating the normalized position and attitude change of the feature points, and the inter-frame pose includes the rotation and translation of the feature points at different time points, that is, the motion change of the feature points between consecutive frames. This method can adapt to different scene feature distributions and improve the accuracy and robustness of pose estimation.

[0032] In step S3, the three-dimensional coordinates of the feature points are restored according to the inter-frame pose. The three-dimensional coordinates can be restored by triangulation. The specific process is: the inter-frame pose is the feature point pairs matched between consecutive frames and their corresponding basic matrices or essential matrices are calculated by solving a linear equation system (such as the DLT algorithm) or the least squares method to obtain the three-dimensional point positions. This process involves constructing a linear equation system and using singular value decomposition to solve it. The three-dimensional points obtained through the above steps are usually homogeneous coordinates in the visual camera coordinate system, and the depth depends on the intrinsic parameters and pose of the visual camera. Therefore, normalization processing is also required to restore the actual three-dimensional coordinates.

[0033] After obtaining the restored three-dimensional coordinates, the plane is constructed and the ground is selected according to the three-dimensional coordinates. In some specific implementations, the plane of the visual camera is first constructed. For example, assuming that the coordinates of the visual camera at A at the current moment are (0,0,0), and the coordinates of the visual camera at B at the previous moment are ( , , ), the horizontal plane formed by AB is the plane of the visual camera, for the convenience of explanation, it is called AB plane below. Let the plane equation of AB plane be: Mx+Ny+Cz=0, where M, N, C are coefficients, and x, y, z are the parameters of the three-dimensional coordinates. Select any feature point on the AB plane, such as ( +0.5, , ), the constructed AB surface equation is: y-( / )z=0.

[0034] It can be seen that after any feature point in the original image information ( , , ), and the plane equation of the plane parallel to the AB plane is: y-( / )z= , where the feature points ( , , ) The normalized height calculated for normalization.

[0035] When selecting the plane where the ground is located, 20% of The height value is used as the preset threshold. In the three-dimensional coordinates, feature points with a height less than or equal to the preset threshold are closer to the ground. Therefore, these feature points are traversed and clustered (i.e., the number of points around each feature point is found). The feature points with the largest clustering value can be considered as feature points on the ground, and the plane formed by the feature points on the ground is the ground. In this way, the plane where the ground is located can be accurately selected.

[0036] In step S4, the actual height between the plane where the visual camera is located and the actual ground is known, that is, the vehicle height. For example, taking the experimental test vehicle as an example, the height is 1.83m, so the actual height is 1.83m. Through the above construction of the plane and the selection of the ground, the ground elevation between the constructed plane and the selected ground is also known, that is, any feature points are selected from the plane and the ground respectively, and the difference in their height values ​​in the three-dimensional coordinates is the ground elevation. Therefore, the scale S in the height direction = actual height / ground elevation. Further, the coordinates of the visual camera from B to A are: -1*( , , S* ), thus obtaining the scale-restored position.

[0037] In summary, the visual odometry scale estimation method proposed in this application uses the actual distance between the visual camera and the ground as the distance constraint, and the calculated position has real scale information. Compared with the traditional method that requires IMU assistance, this method greatly reduces the amount of calculation and is more suitable for running on embedded systems, which is conducive to the promotion of visual odometry.

[0038] Further, step S1 includes: S11: Obtain the original image information collected by the visual camera, and use the Shi Tomasi corner detection algorithm to identify feature points in the original image information; S12: Using an optical flow tracking algorithm to track feature points between consecutive frames for feature point matching.

[0039] Among them, the Shi Tomasi corner detection algorithm is used to identify feature points, and the optical flow tracking algorithm is used for feature point matching. These two technical features work together to solve the problem of feature point recognition and matching, and improve the accuracy and efficiency of feature point recognition and matching.

[0040] In specific implementation, the Shi Tomasi corner detection algorithm can identify stable corners by calculating the gradient matrix of each pixel in the original image information and analyzing its eigenvalue. This method usually sets a threshold, such as 0.01, and only when the eigenvalue is greater than the threshold, the point is considered a corner. In addition, in order to ensure the uniformity of the corner distribution, non-maximum suppression technology can be used to retain only the strongest corners within a certain range (such as a 10x10 pixel window).

[0041] The optical flow tracking algorithm can be implemented using the Lucas-Kanade method. This method assumes that the brightness of the image remains constant within a small spatial and temporal range, and estimates the motion of feature points by solving the optical flow equation. In order to improve the robustness of tracking, the pyramid Lucas-Kanade method can be used to gradually refine the motion estimate from low resolution to high resolution. Generally, using a 3 to 4-layer pyramid structure can achieve a good balance between accuracy and efficiency.

[0042] In practical applications, algorithm parameters can be adjusted according to specific scenarios. For example, in a feature-rich environment, the Shi Tomasi algorithm's corner detection threshold can be increased to reduce the number of detected corners, thereby reducing the computational burden. In a feature-sparse environment, the threshold can be lowered to detect more corners. For optical flow tracking, the size of the search window can be adjusted according to the camera's movement speed, using a larger search window for fast motion and a smaller window for slow motion.

[0043] Through this efficient and accurate feature point acquisition and matching method, the present application can provide reliable basic data for the subsequent visual odometer scale estimation. Accurate feature point matching results directly affect the quality of the construction of the essential matrix or basic matrix in the subsequent steps, and thus affect the calculation accuracy of the inter-frame pose. Therefore, this method not only solves the problem of feature point acquisition and matching, but also provides a guarantee for the accuracy and reliability of the entire visual odometer scale estimation process.

[0044] Furthermore, after step S12, the following steps are included: S13: comparing position changes of feature points in consecutive frames to calculate disparity; S14: Extract key frames according to the disparity.

[0045] In practical applications, disparity calculation can be achieved through a variety of methods. A common method is to use a block matching algorithm, which calculates the disparity by comparing the pixel values ​​of corresponding areas in two images. Another method is to use feature-based matching, such as SIFT or SURF features, which is more robust when dealing with large-scale disparity.

[0046] Different strategies can also be used to extract key frames. A simple method is based on disparity threshold, when the disparity exceeds a preset threshold, the frame is marked as a key frame. Another more complex method is to use information theory-based methods, such as entropy or mutual information, to evaluate the information content of each frame and select the frame with the largest amount of information as the key frame.

[0047] In a specific implementation, the disparity threshold can be set to 5% of the image width. For example, for a 1920x1080 image, the disparity threshold can be set to 96 pixels. When the disparity of more than 50% of the feature points is greater than this threshold, the frame can be marked as a key frame.

[0048] Compared with the traditional method of selecting key frames at fixed intervals, the technical solution of the present application has obvious advantages. Traditional methods may miss important information in key scenes, or retain too much redundant information in static scenes. The method of the present application can intelligently judge the degree of change of the scene through disparity analysis. When the scene changes greatly, such as the sudden appearance of a new object, the disparity change between two consecutive frames is large. The key frames are extracted based on the disparity, and the frames that exceed the preset threshold are extracted as key frames. It can more accurately judge the degree of scene change during the movement of the visual camera, and the inter-frame pose change of the visual camera is more obvious, thereby providing a basis for subsequent inter-frame pose calculation and scale estimation.

[0049] Further, step S2 includes: S21: When the feature points in the scene are not coplanar, obtain at least 8 pairs of matching feature points and construct a basic matrix; S22: When the feature points in the scene are coplanar, the intrinsic parameter matrix of the visual camera is obtained and the essential matrix is ​​constructed; S23: Calculate the normalized position and posture change of the feature point according to the basic matrix or the essential matrix to obtain the inter-frame posture of the feature point.

[0050] Among them, in practical applications, the coplanarity is determined by analyzing the spatial distribution of feature points. The RANSAC algorithm can be used to fit the plane. If most of the feature points (for example, more than 90%) fall on the same plane, the feature points are considered to be coplanar. Secondly, choose to construct the basic matrix or the essential matrix based on the judgment result. For the basic matrix, the 8-point algorithm combined with RANSAC can be used for robust estimation; for the essential matrix, the 5-point algorithm is used with the camera intrinsic parameters for calculation. Finally, the rotation matrix and translation vector are extracted from the selected matrix, and the scale is normalized to obtain the final inter-frame pose.

[0051] Compared with traditional methods, the technical solution of the present application improves the accuracy and robustness of pose estimation through scene-adaptive matrix selection. Compared with the method of fixedly using a certain matrix, this solution can better cope with complex and changing environments. Furthermore, by selecting a suitable matrix, the computational complexity can be reduced while ensuring accuracy. For example, using the essential matrix in a plane scene can reduce the number of matching point pairs required. This flexible processing method provides a more reliable foundation for the subsequent three-dimensional coordinate restoration and scale estimation of feature points.

[0052] Further, step S3 includes: S31: according to the inter-frame pose, the three-dimensional coordinates of the feature points are restored by using a triangulation method; S32: Obtain the three-dimensional coordinates of the position of the visual camera at the current moment and the three-dimensional coordinates of the position of the visual camera at the previous moment, and construct a plane; S33: Calculate the normalized height of the feature point in the three-dimensional coordinates, traverse all feature points according to the normalized height, and cluster them, and select the plane where the feature points with the largest clustering value are located as the ground.

[0053] Among them, in this application, the triangulation method can be implemented in a variety of ways, such as linear triangulation, midpoint method or optimal triangulation. The linear triangulation method has a fast calculation speed and is suitable for scenes with high real-time requirements; the midpoint method has high accuracy and is suitable for scenes with high accuracy requirements; the optimal triangulation method strikes a balance between accuracy and calculation speed and is suitable for most general scenes.

[0054] During the plane construction process, the least squares method or RANSAC algorithm can be used to fit the plane. The least squares method has a fast calculation speed and is suitable for scenes with less noise; the RANSAC algorithm is more robust to outliers and is suitable for complex scenes with more noise points. When calculating the normalized height of feature points, different normalization methods can be selected, such as maximum and minimum normalization or Z-score normalization.

[0055] In specific implementations, a suitable parameter range can be selected according to the actual application scenario. For example, when clustering, the clustering threshold can be set to 10% to 30% of the total number of feature points, which helps to filter out noise points and improve the accuracy of ground recognition.

[0056] Further, step S33 includes: S331: arbitrarily select a feature point and establish a plane equation that passes through the feature point and is parallel to the constructed plane; S332: Calculate the normalized height of the feature point according to the equation, and take a height value of 20% of the normalized height as a preset threshold; S333: Traverse all feature points, cluster the feature points whose height values ​​are less than or equal to a preset threshold, and select the plane where the feature points with the largest clustering value are located as the ground.

[0057] Among them, this technical solution randomly selects a feature point and establishes a plane equation parallel to the construction plane to provide a benchmark for subsequent calculations. Then, the normalized height of the feature point is calculated using this plane equation, and 20% of it is used as the preset threshold, taking into account the possible slight undulations on the ground. Finally, all feature points are traversed, and feature points with height values ​​less than or equal to the preset threshold are clustered, and the plane with the most feature points is selected as the ground, effectively filtering out non-ground feature points and improving the accuracy of ground selection.

[0058] Further, step S4 includes: S41: Get the real height between the plane where the visual camera is located and the real ground. S42: Calculate the height difference between any feature point on the constructed plane and any feature point on the selected ground to obtain the ground elevation; S43: The visual odometry scale is obtained based on the ratio of the true altitude to the ground elevation.

[0059] There are many ways to obtain the real height between the plane where the visual camera is located and the real ground. For example, in a vehicle-mounted terminal, the height of the vehicle body is the real height. During the movement of the drone, it can be directly measured using a laser rangefinder, ultrasonic sensor or barometer. The absolute height of the visual camera can also be obtained through GPS or other positioning systems, and then subtracted from the known ground height.

[0060] When calculating the height difference between any feature point on the constructed plane and any feature point on the selected ground, multiple pairs of feature points can be selected for calculation and then the average value can be taken to improve accuracy. Specifically, algorithms such as the least squares method can be used to optimize the calculation results.

[0061] As can be seen from the above, the visual odometer scale estimation method proposed in this application obtains the original image information, identifies the feature points and matches them in continuous frames, tracks the changes of the feature points in continuous frames, and provides basic data for subsequent pose estimation; uses the matched feature points to construct the essential matrix or basic matrix, calculates the normalized position and attitude change of the feature points, obtains the inter-frame pose, and provides necessary information for three-dimensional coordinate restoration; three-dimensional coordinate restoration and plane construction: restore the three-dimensional coordinates of the feature points according to the inter-frame pose, and performs plane construction and ground selection to provide a spatial reference for scale estimation; calculates the visual odometer scale by comparing the real height between the plane where the visual camera is located and the real ground, and the ground elevation between the constructed plane and the selected ground. This method realizes the scale estimation of the visual odometer by combining image feature extraction, pose estimation, plane construction, ground selection and elevation mapping. Compared with traditional methods, it can accurately restore the visual scale information without relying on IMU data, avoids the influence of IMU cumulative error, and reduces the amount of calculation. By using ground information for scale estimation, the accuracy and stability of scale estimation are improved. This method can provide more reliable visual odometry scale information in complex environments, thereby improving the accuracy and stability of navigation positioning.

[0062] In a second aspect, the present application provides a visual odometer size estimation device, the device comprising: Feature point extraction module 201: obtains original image information, identifies feature points in the original image information, and performs feature matching on the feature points in consecutive frames; Posture calculation module 202: constructs an essential matrix or a basic matrix according to the matched feature points, and calculates the normalized position and posture change of the feature points to obtain the inter-frame posture of the feature points; Plane construction module 203: restore the three-dimensional coordinates of the feature points according to the inter-frame poses, and perform plane construction and ground selection according to the three-dimensional coordinates; Scale estimation module 204: obtains the real height between the plane where the visual camera is located and the real ground, and the ground elevation between the constructed plane and the selected ground, and obtains the visual odometer scale according to the ratio of the real height to the ground elevation.

[0063] Among them, the feature point extraction module 201 is responsible for identifying and matching feature points from the original image, providing basic data for subsequent processing. The posture calculation module 202 uses the matched feature points to construct an essential matrix or a basic matrix, calculates the normalized position and posture change of the feature points, and thus obtains the inter-frame posture. The plane construction module 203 restores the three-dimensional coordinates of the feature points based on the inter-frame posture, and performs plane construction and ground selection. The scale estimation module 204 calculates the visual odometer scale by comparing the actual height between the plane where the terminal is located and the actual ground, and the ground elevation between the constructed plane and the selected ground.

[0064] The feature point extraction module 201 can use a variety of methods to identify and match feature points. For example, the Shi-Tomasi corner point detection algorithm can be used to identify feature points, and then the optical flow tracking algorithm can be used to match feature points between consecutive frames. In addition, feature descriptors such as SIFT or SURF can also be used to identify and match feature points.

[0065] When constructing the essential matrix or the basic matrix, the pose calculation module 202 needs to make a selection based on the scene characteristics. When the feature points in the scene are not coplanar, at least 8 pairs of matching feature points can be obtained to construct the basic matrix. When the feature points are coplanar, it is necessary to obtain the intrinsic parameter matrix of the visual camera to construct the essential matrix. This flexible selection can adapt to different scenes and improve the accuracy of pose calculation.

[0066] The plane construction module 203 can use triangulation when restoring the three-dimensional coordinates of the feature points. When selecting the ground, the normalized height of the three-dimensional coordinates of the feature points can be calculated, and then all the feature points can be traversed and clustered, and the plane where the feature points with the largest clustering value are located can be selected as the ground. This method can effectively identify the real ground plane and provide a reliable basis for subsequent scale estimation.

[0067] The scale estimation module 204 can use data from other sensors (such as a barometer or a laser rangefinder) when obtaining the true height. When calculating the ground elevation, multiple feature points on the construction plane and on the ground can be selected to calculate the average height difference between them to improve the stability and accuracy of the estimation.

[0068] The collaboration between these modules forms a complete visual odometry scale estimation system. The feature point extraction module 201 provides the necessary input data for the pose calculation module, and the output of the pose calculation module 202 provides the basis for the plane construction module. The results of the plane construction module 203 are finally used by the scale estimation module 204 to calculate the visual odometry scale. This close connection between modules ensures the efficient operation and accurate estimation of the entire system.

[0069] Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The present application provides an electronic device 3, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not shown). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiment to achieve the following functions: The present application embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method in any optional implementation of the above embodiment is executed to achieve the following functions: Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0070] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0071] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0072] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0073] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0074] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A visual odometry scale estimation method, characterized in that: The method comprises the steps of: S1: acquiring original image information, identifying feature points in the original image information, and performing feature matching on the feature points in consecutive frames; S2: constructing an essential matrix or a basic matrix according to the matched feature points, and calculating the normalized position and posture change of the feature points to obtain the inter-frame posture of the feature points; S3: restoring the three-dimensional coordinates of the feature points according to the inter-frame poses, and performing plane construction and ground selection according to the three-dimensional coordinates; S4: Obtain the real height between the plane where the visual camera is located and the real ground, and the ground elevation between the constructed plane and the selected ground, and obtain the visual odometer scale according to the ratio of the real height to the ground elevation.

2. A visual odometry scale estimation method according to claim 1, characterized in that: Step S1 includes: S11: Acquire original image information collected by a visual camera, and use Shi Tomasi corner detection algorithm to identify the feature points in the original image information; S12: Using an optical flow tracking algorithm to track the feature points between consecutive frames to perform feature point matching.

3. A visual odometry scale estimation method according to claim 2, characterized in that: Step S12 includes: S13: comparing the position changes of the feature points in consecutive frames to calculate the disparity; S14: extracting key frames according to the disparity.

4. A visual odometry scale estimation method according to claim 1, characterized in that: Step S2 includes: S21: when the feature points in the scene are not coplanar, obtaining at least 8 pairs of matching feature points and constructing the basic matrix; S22: When the feature points in the scene are coplanar, an intrinsic parameter matrix of the visual camera is obtained to construct the intrinsic matrix; S23: Calculate the normalized position and posture change of the feature point according to the basic matrix or the essential matrix to obtain the inter-frame posture of the feature point.

5. The visual odometry scale estimation method according to claim 1, characterized in that: Step S3 includes: S31: restoring the three-dimensional coordinates of the feature points using a triangulation method according to the inter-frame pose; S32: Obtain the three-dimensional coordinates of the position of the visual camera at the current moment and the three-dimensional coordinates of the position of the visual camera at the previous moment, and construct a plane; S33: Calculate the normalized height of the feature point in the three-dimensional coordinates, traverse all the feature points according to the normalized height, and perform clustering, and select the plane where the feature point with the largest clustering value is located as the ground.

6. A visual odometry scale estimation method according to claim 5, characterized in that: Step S33 includes: S331: arbitrarily selecting the feature point, and establishing a plane equation that passes through the feature point and is parallel to the constructed plane; S332: Calculate the normalized height of the feature point according to the equation, and take a height value of 20% of the normalized height as a preset threshold; S333: Traverse all the feature points, cluster the feature points whose height values ​​are less than or equal to the preset threshold, and select the plane where the feature points with the largest clustering value are located as the ground.

7. A visual odometry scale estimation method according to claim 6, characterized in that: Step S4 includes: S41: Get the real height between the plane where the visual camera is located and the real ground. S42: Calculate the height difference between any of the feature points on the constructed plane and any of the feature points on the selected ground to obtain the ground elevation; S43: Obtain a visual odometer scale according to a ratio of the true height to the ground elevation.

8. A visual odometry scale estimation device, characterized in that: The device comprises: Feature point extraction module: obtains original image information, identifies feature points in the original image information, and performs feature matching on the feature points in consecutive frames; Posture calculation module: constructs an essential matrix or a basic matrix according to the matched feature points, and calculates the normalized position and posture change of the feature points to obtain the inter-frame posture of the feature points; Plane construction module: restore the three-dimensional coordinates of the feature points according to the inter-frame pose, and perform plane construction and ground selection according to the three-dimensional coordinates; Scale estimation module: obtains the real height between the plane where the visual camera is located and the real ground, as well as the ground elevation between the constructed plane and the selected ground, and obtains the visual odometer scale according to the ratio of the real height to the ground elevation.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 7 are executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

Images