Feature processing method, visual tracking method, device, equipment and medium

Abstract

The invention relates to the technical field of computer vision and space calculation, and discloses a feature processing method, a visual tracking method, a device, equipment and a medium, and the feature processing method comprises the steps: obtaining a transformation relation between a current frame image coordinate system and a current world coordinate system according to a current frame image captured by the equipment; according to the transformation relation, converting the world coordinates of the historical road sign points in the point cloud library into image coordinates in the current frame image, so as to determine corresponding projection points of the historical road sign points on the current frame image; dividing the current frame image into a plurality of grids, determining a first type of feature points in each grid, the first type of feature points being feature points associated with feature points of the current frame image and feature points of a previous frame image; if the number of the first type of feature points in the grids is less than the preset value, adding a corresponding number of projection points in the corresponding grids; and carrying out optical flow tracking on the corresponding number of projection points, and determining potential feature points of the corresponding number of projection points on the current frame image so as to improve the stability of front-end feature tracking.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and spatial computing technology, and in particular to a feature processing method, a visual tracking method, a device, a equipment and a storage medium. Background Art

[0002] In recent years, the integration trend of vision systems and IMU (Inertial Measurement Unit) sensors has become increasingly obvious. The technology of using the measurement data of vision systems and IMU sensors to estimate the changes in the position and direction of the movement of smart devices over time is called visual-inertial odometry (VIO). Existing smart devices gradually use a visual-inertial odometry module to complete the simultaneous localization and mapping (SLAM) of the device. However, when the device moves too fast or the movement amplitude is too large, the accuracy of simultaneous positioning and mapping through the VIO system needs to be improved. Summary of the invention

[0003] In view of the shortcomings of the existing VIO system, a feature processing method, a visual tracking method, a device, a equipment and a storage medium are provided, which can optimize the input observation data and improve the use effect of the VIO system.

[0004] In order to achieve the above object, in a first aspect, a feature processing method is provided, which is applied to a VIO system, wherein the VIO system includes a point cloud library, and the point cloud library includes a plurality of historical landmarks, including: According to the current frame image captured by the device, the transformation relationship between the current frame image coordinate system and the current world coordinate system is obtained; according to the transformation relationship, the world coordinates of the historical landmark points in the point cloud library are converted into image coordinates in the current frame image to determine the corresponding projection points of the historical landmark points on the current frame image; the current frame image is divided into multiple grids, and the first type of feature points in each grid are determined, and the first type of feature points are feature points associated with the feature points of the current frame image and the previous frame image; if the first type of feature points in the grid are less than a preset value, a corresponding number of projection points are added in the corresponding grid; the corresponding number of projection points are subjected to optical flow tracking to determine the potential feature points of the corresponding number of projection points on the current frame image, and the potential feature points and the first type of feature points are used to construct a map.

[0005] In some optional embodiments, if the first type of feature points in the grid is less than a preset value, a corresponding number of projection points are added in the corresponding grid, including: all the projection points in the corresponding grid are selected in time order, and the corresponding number is the difference between the preset value and the number of first type feature points in the corresponding grid.

[0006] In some optional implementations, performing optical flow tracking on the corresponding number of projection points includes: performing image pyramid layering on the current frame image, and tracking the corresponding number of projection points layer by layer through optical flow tracking.

[0007] In some optional embodiments, determining the potential feature points of the corresponding number of projection points on the current frame image includes: using the corresponding number of projection points as the initial points of optical flow tracking to determine the potential feature points corresponding to the projection points on the current frame image, and obtaining the moving distance between the projection points and the corresponding potential feature points; based on a first threshold value pre-set for verification, comparing the moving distance between the projection points and the corresponding potential feature points, if the moving distance is less than or equal to the first threshold, the corresponding potential feature points are used to construct a map; and / or, based on a second threshold value pre-set for verification, calculating the sum of residuals between the pixel blocks of the projection points and the corresponding potential feature point pixel blocks, if the sum of residuals is less than or equal to the second threshold, the corresponding potential feature points are used to construct a map.

[0008] In some optional embodiments, the feature processing method also includes an updating step, which includes: updating the historical landmark points of the point cloud library based on the obtained potential feature points; and / or, when new feature points are extracted from the current frame image, recording them in the point cloud library as historical landmark points, and the current frame image includes the first type of feature points, the potential feature points and the new feature points.

[0009] In some optional embodiments, the feature processing method also includes a maintenance step, which includes: when the device is not located in the current constructed map, removing the historical landmark points of the point cloud library from the current constructed map, and deleting the pixel block information of the historical landmark points in the point cloud library.

[0010] In a second aspect, a visual tracking method is provided, including: obtaining a previous frame image, a current frame image, and state parameters from the previous frame image to the current frame image captured by a device; performing optical flow tracking on the current frame image and the previous frame image to obtain a first type of feature points in the current frame image that are associated with the feature points of the previous frame image, and generating a local spatial map of the current frame; implementing the feature processing method as described above to obtain potential feature points of the current frame image; updating the local spatial map of the current frame based on the first type of feature points and the potential feature points, and then calculating the position and posture of the device in combination with the state parameters.

[0011] In a third aspect, a feature processing device is provided, which is applied to a VIO system, wherein the VIO system includes a point cloud library, and the point cloud library includes multiple historical landmark points, including: a first module, which is used to obtain the transformation relationship between the current frame image coordinate system and the current world coordinate system according to the current frame image captured by the device; a second module, which is used to convert the world coordinates of the historical landmark points in the point cloud library into image coordinates in the current frame image according to the transformation relationship, so as to determine the corresponding projection points of the historical landmark points on the current frame image; a third module, which is used to divide the current frame image into multiple grids, and determine the first type of feature points in each grid, wherein the first type of feature points are feature points associated with the feature points of the current frame image and the previous frame image; a fourth module, which is used to add a corresponding number of projection points in the corresponding grid if the first type of feature points in the grid are less than a preset value; a fifth module, which is used to perform optical flow tracking on the corresponding number of projection points, determine potential feature points of the corresponding number of projection points on the current frame image, and use the potential feature points and the first type of feature points to construct a map.

[0012] In a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the feature processing method as described above when executing the computer program.

[0013] In a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the feature processing method as described above is implemented.

[0014] The present application performs a separate optical flow tracking of historical landmark points, so that feature points that cannot be tracked and associated in the previous frame image can be obtained in the current frame image, which are used for map construction and subsequent pose calculation in the VIO system, and can effectively improve the corresponding calculation accuracy. It is especially suitable for situations where the number of feature points in the current frame image is too small or too concentrated. The feature processing method of the present application can improve the stability of front-end feature tracking, so that the device maintains good positioning and mapping accuracy in a variety of motion conditions. At the same time, by associating the potential feature points finally obtained with the historical landmark points, it can greatly save computing power and reduce computing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 is a flow chart of the feature processing method described in the embodiment; Figure 2 is a schematic diagram of grid division and feature point selection described in the embodiment; Figure 3 is a flow chart of the visual tracking method described in the embodiment; Figure 4 is a schematic diagram of the structure of the feature processing device described in the embodiment; Figure 5 is a schematic diagram of the structure of the visual tracking device described in the embodiment; Figure 6 It is a schematic diagram of the structure of the electronic device described in the embodiment. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Whether it is AR or VR equipment, or mobile robots, 6DOF positioning is required in space. For spatial positioning, most existing technologies use front-end feature tracking modules, and the feature tracking adopts optical flow tracking. However, the front-end feature tracking does not use historical landmark feature data to improve stability, and the management ideas of landmark features are unclear. In order to improve the tracking effect and positioning accuracy, these features need to be managed and processed.

[0019] The present application is described below with reference to specific embodiments in conjunction with the accompanying drawings.

[0020] like Figure 1 As shown, a feature processing method is applied to a VIO system, including steps S301 to S305: Step S301, obtaining the transformation relationship between the current frame image coordinate system and the current world coordinate system according to the current frame image captured by the device.

[0021] Specifically, in the visual inertial odometry (VIO) process, the device will collect the current frame image based on the currently used camera and predict the current frame pose T through the IMU. wi-pre , T wi-pre Represents the prediction of the transformation from the IMU coordinate system to the world coordinate system, and is based on the known T ic , T ic It is the coordinate transformation relationship from the currently used camera coordinate system to the IMU coordinate system, through the formula T wc =T wi-pre T ic Get the transformation relationship between the current frame image coordinate system and the current world coordinate system, that is, T wc It is the transformation relationship from the current frame image coordinate system to the current world coordinate system.

[0022] Furthermore, if the device is a VR glasses and adopts a binocular system, and the left-eye camera is the main data acquisition device, the left-eye camera can be selected as the camera currently used.

[0023] Step S302: According to the transformation relationship, the world coordinates of the historical landmark points in the point cloud library are converted into image coordinates in the current frame image to determine the projection points corresponding to the historical landmark points on the current frame image.

[0024] The VIO system includes a point cloud library, which includes multiple historical landmark points. The point cloud library is a collection of points in three-dimensional space that can describe the shape and features of objects. The point cloud library can store previously acquired historical landmark points, that is, it stores feature points previously acquired for positioning and map construction. These feature points are representative in the image and can remain stable under different viewing angles, thereby performing matching and recognition between multiple images.

[0025] In the VIO system, the historical landmark points maintained in all the current optimized sliding windows are obtained based on the point cloud library, and the T wc The historical landmark points in the three-dimensional space are projected onto the current frame image using the projection transformation relationship. That is, the projection points of the historical landmark points on the current frame image are obtained. The specific formula used is as follows:

[0026] in, x is a homogeneous coordinate vector representing a point on the image plane; u , v is the two-dimensional coordinate on the current image plane; f proj is the projection model function of the camera, which projects points in the three-dimensional space onto the two-dimensional image plane, and the projection models of different models of cameras are different; T wc -1 is T wc The inverse matrix is ​​the transformation relationship from the current world coordinate system to the current frame image coordinate system; the vector It is a homogeneous coordinate representation, in which the last element is 1, which is used to keep the scale unchanged when performing projection transformation.

[0027] Get each projection point uv After obtaining the coordinates, you can proceed to the next step.

[0028] Step S303, dividing the current frame image into a plurality of grids, and determining the first type of feature points in each grid, where the first type of feature points are feature points associated with the feature points of the current frame image and the previous frame image.

[0029] The image of the current frame can be divided into Figure 2 The grid shown in the figure can be directly used if the current frame image has been divided into a grid when feature points are extracted and supplemented by the FAST corner detection algorithm or other algorithms. The current frame image will perform optical flow tracking with the previous frame image, and then feature points associated with the feature points of the previous frame image can be obtained in the current frame image, and they are used as the first type of feature points, that is, the first type of feature points of the current frame image, which can be correlated with the feature points of the previous frame image. In addition, there is no order of precedence between step 302 and step 303, and no limitation is made here.

[0030] Step S304: if the number of the first type of feature points in the grid is less than a preset value, a corresponding number of projection points are added to the corresponding grid.

[0031] After the divided grids and the first type of feature points are determined, the projection points can be judged and used. The specific judgment method is: Traverse all divided grids. If the number of the first type of feature points in a grid is greater than or equal to the preset value M, then the grid does not need to be processed to increase the projection points. If the number of the first type of feature points in a grid is less than the preset value M, the projection points in the grid are selected from the projection points of the current frame image for additional use. The specific corresponding number of increased projection points can be determined according to preset requirements.

[0032] In some optional embodiments, all projection points in the corresponding grid are selected in time order, and the corresponding number is the difference between the preset value and the number of first-category feature points in the corresponding grid.

[0033] Specifically, when it is necessary to add projection points in a certain grid, when the number of selectable projection points in the grid exceeds the number to be selected, the projection points can be sorted according to their time attributes. The time attribute of the projection points is the generation time of the corresponding historical landmark points, so as to select the required projection points closer to the current time. In this way, it can ensure that the latest saved historical landmark points are used as much as possible, improve the reliability of using historical landmark points to correspond to projection points, and provide more effective front-end features for the construction of subsequent maps.

[0034] The corresponding number of increased projection points can be obtained through specific calculations to determine the selection criteria for increasing projection points. Compared with selecting according to a unified set value, the utilization efficiency of the first-class feature points and projection points can be improved, and unnecessary projection points can be selected when there are enough first-class feature points in a grid, thereby reducing the waste of computing power. For example, assuming that there are L first-class feature points in a grid, and the total number of feature points required in the grid is preset to M, then the number of projection points N=ML that needs to be selected in the grid can be determined, such as Figure 2 The preset value M=1 represented in corresponds to that in a grid without the first type of feature points, if there is a projection point corresponding to a historical landmark point, a projection point closest in time is selected as the feature point of the grid, thereby effectively improving the accuracy of subsequent map construction.

[0035] Step S305 , performing optical flow tracking on a corresponding number of projection points, determining potential feature points of the corresponding number of projection points on the current frame image, and using the potential feature points and the first type of feature points to construct a map.

[0036] Based on all the qualified projection points selected in step S304, assuming there are K projection points, the K projection points are used as the initial points of optical flow tracking, and optical flow tracking is performed in the current frame image to obtain the coordinate position after optical flow tracking. The result of optical flow tracking is to find the matching pixel block (also called patch) and its corresponding feature point. The matching pixel block refers to the pixel block on the current frame image that is closest to the pixel block where the initial point is located. The feature point corresponding to the matching pixel block is the potential feature point, so the potential feature point matches the corresponding projection point. At the same time, the potential feature point also matches the corresponding historical landmark point. After obtaining the potential feature point corresponding to the added projection point, these potential feature points and the first type of feature points are used together as front-end features for map construction and subsequent posture calculation.

[0037] The feature processing method of the present application performs a separate optical flow tracking on historical landmark points, so that feature points that cannot be tracked and associated in the previous frame image, that is, feature points that do not belong to the first category of feature points, can be obtained in the current frame image. This effectively improves the corresponding calculation accuracy when used for map construction and subsequent pose calculation, and avoids the disadvantages of map construction and subsequent pose calculation due to too few or too concentrated feature points.

[0038] In addition, by dividing the grids on the current frame image, for grids without first-category feature points or with only a small number of first-category feature points, historical landmark points are actively extracted to supplement the feature points, which can ensure that there are a sufficient number of feature points on the current frame image, and at the same time, the distribution of the acquired feature points on the current frame image is relatively uniform, which is particularly suitable for the situation where the first-category feature points of the current frame image are relatively concentrated; further, for grids with a large number of first-category feature points, there is no need to extract the corresponding historical landmark points, which can greatly save computing power, especially there is no need to match all feature points on the current frame image that are not of the first category with historical landmark points, thereby reducing the computing power requirements for matching historical landmark points.

[0039] The feature processing method of the present application utilizes the projection points of historical landmark points as the feature points of the current frame image, and can connect the feature points in the current frame image with the feature points of the historical frame image. The historical frame image is obtained before the previous frame image, which solves the problem that the feature points cannot be associated after the image is interrupted, and improves the continuity between the images used to generate the local map. By associating the historical landmark points in the point cloud library with the feature points of the current frame image and then updating and optimizing the local space map, the accuracy and stability of the device positioning can be improved, and the situation where the same feature point is marked as two feature points can be reduced.

[0040] For example, the historical frame image has a feature point of the first air-conditioning outlet. The first air-conditioning outlet in the historical frame image is used as a historical landmark point and saved in the point cloud library. Due to the rapid movement of the device, or the image is lost, or the first air-conditioning outlet is blocked by the line of sight and other similar reasons, the feature point of the first air-conditioning outlet does not appear in the previous frame image, while the current frame image captures the "second air-conditioning outlet". However, in fact, the "second air-conditioning outlet" is the first air-conditioning outlet. For the prior art, the feature points of the current frame image are associated with the feature points of the previous frame image by optical flow tracking, and the obtained local space map has the "second air-conditioning outlet". The prior art will mistakenly believe that there are the first air-conditioning outlet and the "second air-conditioning outlet", that is, it is impossible to identify the "second air-conditioning outlet" as the first air-conditioning outlet of the historical map, which can easily cause a waste of computing power and a decrease in the accuracy of map construction. The present application can reduce the situation where the same feature point is not identified by associating historical landmark points, so that the updated current frame local map is more accurate.

[0041] In some optional embodiments, the current frame image is layered into an image pyramid, and a corresponding number of projection points are tracked layer by layer through optical flow tracking.

[0042] That is, in step S305, the current frame image may be further processed by image pyramid layering, and when optical flow tracking is performed on the projection points, layer-by-layer tracking may be performed on the basis of optical flow tracking to balance the accuracy and robustness of optical flow tracking.

[0043] In some optional embodiments, determining potential feature points of a corresponding number of projection points on the current frame image also includes the following steps: using the corresponding number of projection points as initial points for optical flow tracking to determine the potential feature points corresponding to the projection points on the current frame image, and obtaining the moving distance between the projection points and the corresponding potential feature points; that is, obtaining the displacement between the projection points and the corresponding potential feature points through the results of optical flow tracking, so as to facilitate the subsequent use verification of the potential feature points.

[0044] In some optional embodiments, based on a first threshold value preset for verification, the moving distance between the projection point and the corresponding potential feature point is compared, and if the moving distance is less than or equal to the first threshold value, the corresponding potential feature point is used to construct the map.

[0045] That is, in step 305, after setting the first threshold for verification according to actual needs, the results of optical flow tracking can be reviewed and checked to improve the reliability of the potential feature points finally determined and ensure the stability of the front-end features. Specifically, the moving distance between the projection point and the corresponding potential feature point is determined. The moving distance refers to the moving distance of the pixel block corresponding to the projection point on the current frame image. For example, the first threshold is set to 3 pixels, that is, when matching the pixel block where the projection point is located with the pixel block on the current frame image, if the potential feature point is matched, but the pixel block corresponding to the potential feature point and the pixel block corresponding to the projection point move more than 3 pixels, it is considered that the two pixel blocks do not match, and the corresponding potential feature points are not used for subsequent map construction to realize the review of the matching results. In addition, when there are too many unused potential feature points, projection points can be added again to supplement the required potential feature points.

[0046] In some optional embodiments, based on a second threshold value preset for verification, the sum of residuals between the pixel block of the projection point and the corresponding pixel block of the potential feature point is calculated. If the sum of residuals is less than or equal to the second threshold value, the corresponding potential feature point is used to construct the map.

[0047] That is, in step 305, after the second threshold for verification is set according to actual needs, the result of optical flow tracking is reviewed and checked to improve the reliability of the potential feature points finally determined and ensure the stability of the front-end features. Specifically, the residual sum between the pixel block of the projection point and the corresponding pixel block of the potential feature point is calculated. Since the projection point is the projection of the historical landmark point on the current frame image, the pixel block of the projection point is the pixel block corresponding to the historical landmark point, that is, the pixel block corresponding to the historical landmark point recorded in the point cloud library, and the pixel block of the potential feature point refers to the pixel block where the feature point of the historical landmark point corresponding to the current frame image is located. Therefore, what is finally calculated is the residual sum r between the pixel block where the historical landmark point is located and the pixel block where the feature point of the current frame image is located. sum , in order to verify the matching results, assuming that the size of the pixel block is n p pixels, then:

[0048] Among them, I f is the pixel block where the feature point of the current frame image is located, that is, the pixel block of the potential feature point, I l The pixel block where the historical landmark point is located; x i is the pixel index corresponding to the pixel block where the landmark point is located, and is the average value of all pixels in the corresponding pixel block, T flIt is the coordinate transformation relationship of the selected projection point to the corresponding potential feature point in the current frame image.

[0049] Then, based on the calculated residual sum, we make a judgment. sum When the set second threshold is exceeded, it can be considered that there is no match, that is, the determined potential feature points are judged to be unreliable. In order to avoid affecting the accuracy of subsequent map construction, the corresponding potential feature points are not used for subsequent map construction. The specific value of the second threshold is determined according to actual needs and is not limited here.

[0050] In some optional embodiments, the feature processing method further includes an updating step, and the updating step includes: updating the historical landmark points of the point cloud library according to the obtained potential feature points.

[0051] Updating the historical landmark points in the point cloud library mainly involves updating the pixel blocks where the historical landmark points are located. That is, for the matched pixel blocks, compare the predicted posture of the current frame with the posture when the historical landmark points are updated and stored. If the relative translation distance and relative rotation angle exceed a certain threshold, it can be considered that the pixel block of the feature point of the current frame has a relatively large parallax compared with the pixel block of the matched historical landmark points. Then, the pixel block recording the feature point of the current frame can be updated to replace the pixel block of the matched historical landmark points. If there is a sliding window optimization module, its corresponding position will be updated after the optimization is completed.

[0052] By performing optical flow tracking on historical landmark points and matching them to the feature points on the current frame image, the pixel blocks of the historical landmark points in the point cloud library are also updated accordingly. This can update the data of the point cloud library in real time, ensure the validity of the historical landmark points in the point cloud library, and realize a virtuous cycle of calling historical data.

[0053] In some optional embodiments, the updating step further includes: when new feature points are extracted in the current frame image, they are recorded in the point cloud library as historical landmark points, and the current frame image includes the first type of feature points, potential feature points and new feature points.

[0054] Whenever a new valid feature point is extracted, that is, after the current frame image performs optical flow tracking based on the previous frame image, the obtained feature point is neither associated with the previous frame image nor matches the historical landmark point. In other words, the new valid feature point is a valid feature point in the current frame image that does not belong to the first category of feature points or potential feature points. The pixel block where the new feature point is located is recorded, for example, recorded in the point cloud library as a historical landmark point, and the spatial position of the current frame corresponding to the pixel block is used to update the pixel block, thereby optimizing the update process of the pixel block. Therefore, when the device collects the current frame image, the feature points can include three types: the first category of feature points, potential feature points and new feature points.

[0055] In some optional embodiments, the feature processing method also includes a maintenance step, which includes: when the device is not located in the current construction map, removing the historical landmark points of the point cloud library from the current construction map, and deleting the pixel block information of the historical landmark points in the point cloud library.

[0056] That is, the device constructs maps and calculates postures based on the collected images. When the device leaves the previous spatial environment, there is no need to call the corresponding historical landmark points stored in the point cloud library. Therefore, according to the local spatial map currently constructed by the device, whenever some feature points are removed from the local spatial map, if the feature points are potential feature points, there will be corresponding historical landmark points stored. In this case, the pixel block information of the corresponding historical landmark points will be deleted, and the historical landmark point information in the point cloud library will be timely maintained, unnecessary memory usage will be cleared, and the management of landmark information will be realized.

[0057] like Figure 3 As shown, a visual tracking method includes: Step S1: Obtain the state parameters of the previous frame image, the current frame image and the image from the previous frame image to the current frame image captured by the device.

[0058] Step S2: Perform optical flow tracking on the current frame image and the previous frame image to obtain the first type of feature points in the current frame image that are associated with the feature points in the previous frame image, and generate a local space map of the current frame.

[0059] Step S3: Implement the above feature processing method to obtain potential feature points of the current frame image.

[0060] Step S4: Based on the first type of feature points and potential feature points, the local space map of the current frame is updated, and then combined with the state parameters, the position and posture of the device are calculated.

[0061] When the VIO system of the present application completes the SLAM of the device to which it belongs, it associates and matches the feature points of the current frame with very early historical landmark points through the associated use of historical landmark features, thereby making the local map recognition of the current frame more accurate and significantly saving the required computing power, thereby optimizing the observation data of the highly nonlinear VIO input and effectively improving the positioning capability of the device.

[0062] Based on the above method embodiments, a feature processing device and a visual tracking device are provided to implement the embodiments and preferred embodiments of the above method. Those that have been explained will not be repeated here. The terms "module", "unit", "sub-unit", etc. used below refer to a combination of software and / or hardware that can implement predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0063] like Figure 4 As shown, this embodiment provides a feature processing device, which is applied to a VIO system. The VIO system includes a point cloud library, and the point cloud library includes a plurality of historical landmarks. The feature processing device includes a first module 10, a second module 20, a third module 30, a fourth module 40, and a fifth module 50, wherein: The first module 10 is used to obtain the transformation relationship between the current frame image coordinate system and the current world coordinate system according to the current frame image captured by the device.

[0064] The second module 20 is used to convert the world coordinates of the historical landmark points in the point cloud library into image coordinates in the current frame image according to the transformation relationship, so as to determine the projection points corresponding to the historical landmark points on the current frame image.

[0065] The third module 30 is used to divide the current frame image into a plurality of grids, and determine the first type of feature points in each grid, where the first type of feature points are feature points associated with the feature points of the current frame image and the previous frame image.

[0066] The fourth module 40 is used to add a corresponding number of projection points in the corresponding grid if the first type of feature points in the grid is less than a preset value.

[0067] The fifth module 50 is used to perform optical flow tracking on a corresponding number of projection points, determine potential feature points of the corresponding number of projection points on the current frame image, and use the potential feature points and the first type of feature points to construct a map.

[0068] like Figure 5 As shown, the visual tracking device includes a data acquisition module 110, an association module 120, a projection module 130, a feature point quantity judgment module 140, a matching module 150, a tracking inspection module 160, and a point cloud library update module 170, wherein: The data acquisition module 110 is used to obtain the transformation relationship between the current frame image coordinate system and the current world coordinate system, obtain historical landmark points in the point cloud library, and obtain the state parameters of the previous frame image, the current frame image, and the previous frame image to the current frame image captured by the device.

[0069] The association module 120 is used to associate the feature points of the current frame image with the feature points of the previous frame image by optical flow tracking to obtain the first type of feature points of the current frame image and the local spatial map of the current frame; the first type of feature points refers to the feature points in the current frame image that can be associated with the feature points of the previous frame image.

[0070] The projection module 130 is used to project the historical landmark points onto the current frame image according to the transformation relationship, and obtain the projection points corresponding to the historical landmark points on the current frame image.

[0071] The feature point quantity determination module 140 is used to divide the current frame image into grids, and if the first type of feature points in the grid is less than a preset value, a corresponding number of projection points are added in the corresponding grid.

[0072] The matching module 150 is used to perform optical flow tracking on a corresponding number of projection points to determine potential feature points of the corresponding number of projection points on the current frame image.

[0073] The tracking and checking module 160 is used to set a first threshold and a second threshold, and determine the moving distance between the projection point and the corresponding potential feature point, and calculate the residual sum between the pixel block of the projection point and the corresponding potential feature point pixel block. If the moving distance is less than or equal to the first threshold, and the corresponding residual sum is less than or equal to the second threshold, the corresponding potential feature point and the first type of feature point are used together to construct the map.

[0074] The point cloud library update and maintenance module 170 is used to update the historical landmark points of the point cloud library based on the obtained potential feature points. When new feature points are extracted, they are recorded in the point cloud library as historical landmark points. When the historical landmark points of the point cloud library are removed from the local space map, the pixel block information corresponding to the historical landmark points is deleted.

[0075] In some optional embodiments, a local map updating module is further included to update the local spatial map of the current frame according to the first type of feature points and potential feature points of the current frame image.

[0076] In some optional embodiments, a calculation module is further included, which is used to calculate the position and posture of the device in the updated local space map of the current frame based on the state parameters and the updated local space map of the current frame.

[0077] Based on the above method embodiment, Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, and the following reference Figure 6 , which shows an electronic device (eg, Figure 6 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0078] like Figure 6As shown, the electronic device 500 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, ROM 502 and RAM 503 are connected to each other via a bus 504, and an editing / output (I / O) interface 505 is also connected to the bus 504.

[0079] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509, which may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The electronic device 500 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown, and more or fewer devices may be implemented or possessed instead.

[0080] Based on the above method embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0082] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0086] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0087] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0088] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0090] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A feature processing method, applied to a VIO system, wherein the VIO system includes a point cloud library, wherein the point cloud library includes a plurality of historical landmarks, wherein: include: According to the current frame image captured by the device, the transformation relationship between the current frame image coordinate system and the current world coordinate system is obtained; According to the transformation relationship, the world coordinates of the historical landmark points in the point cloud library are converted into image coordinates in the current frame image to determine the projection points corresponding to the historical landmark points on the current frame image; Dividing the current frame image into a plurality of grids, determining a first type of feature point in each grid, wherein the first type of feature point is a feature point associated with a feature point of a previous frame image of the current frame image; If the first type of feature points in the grid is less than a preset value, a corresponding number of projection points are added to the corresponding grid; The corresponding number of projection points are subjected to optical flow tracking to determine potential feature points of the corresponding number of projection points on the current frame image, and the potential feature points and the first type of feature points are used to construct a map.

2. The feature processing method according to claim 1, characterized in that: If the first type of feature points in the grid are less than a preset value, adding a corresponding number of projection points in the corresponding grid includes: All the projection points in the corresponding grid are selected according to time sorting, and the corresponding number is the difference between the preset value and the number of the first type of feature points in the corresponding grid.

3. The feature processing method according to claim 1, characterized in that: The performing optical flow tracking on the corresponding number of projection points comprises: The current frame image is layered into an image pyramid, and the corresponding number of projection points are tracked layer by layer through optical flow tracking.

4. The feature processing method according to claim 1, characterized in that: The determining of the potential feature points of the corresponding number of projection points on the current frame image includes: Using the corresponding number of projection points as initial points for optical flow tracking to determine potential feature points corresponding to the projection points on the current frame image, and obtaining a moving distance between the projection points and the corresponding potential feature points; Based on a first threshold value preset for verification, comparing the moving distance between the projection point and the corresponding potential feature point, if the moving distance is less than or equal to the first threshold value, the corresponding potential feature point is used to construct a map; And / or, based on a second threshold value preset for verification, the sum of residuals between the pixel block of the projection point and the corresponding pixel block of the potential feature point is calculated, and if the sum of residuals is less than or equal to the second threshold value, the corresponding potential feature point is used to construct the map.

5. The feature processing method according to claim 1, characterized in that: The feature processing method further comprises an updating step, wherein the updating step comprises: According to the obtained potential feature points, updating the historical landmark points of the point cloud library; And / or, when a new feature point is extracted from the current frame image, it is recorded in the point cloud library as a historical landmark point, and the current frame image includes the first type of feature points, the potential feature points and the new feature points.

6. The feature processing method according to claim 1, characterized in that: The feature processing method further includes a maintenance step, wherein the maintenance step includes: When the device is not located in the current constructed map, the historical landmark points of the point cloud library are removed from the current constructed map, and the pixel block information of the historical landmark points in the point cloud library is deleted.

7. A visual tracking method, characterized in that: include: Obtain the state parameters of the previous frame image, the current frame image and the image from the previous frame image to the current frame image captured by the device; Performing optical flow tracking on the current frame image and the previous frame image to obtain a first type of feature points in the current frame image that are associated with feature points in the previous frame image, and generating a local space map of the current frame; Implementing the feature processing method as described in any one of claims 1 to 6 to obtain potential feature points of the current frame image; The local space map of the current frame is updated according to the first type of feature points and the potential feature points, and then the position and posture of the device is calculated in combination with the state parameters.

8. A feature processing device, applied to a VIO system, wherein the VIO system includes a point cloud library, wherein the point cloud library includes a plurality of historical landmarks, wherein: include: The first module is used to obtain the transformation relationship between the current frame image coordinate system and the current world coordinate system according to the current frame image captured by the device; The second module is used to convert the world coordinates of the historical landmark points in the point cloud library into image coordinates in the current frame image according to the transformation relationship, so as to determine the projection point corresponding to the historical landmark point on the current frame image; A third module is used to divide the current frame image into a plurality of grids, and determine a first type of feature points in each grid, wherein the first type of feature points are feature points associated with feature points of the current frame image and a previous frame image; The fourth module is used to add a corresponding number of projection points in the corresponding grid if the first type of feature points in the grid is less than a preset value; The fifth module is used to perform optical flow tracking on the corresponding number of projection points, determine potential feature points of the corresponding number of projection points on the current frame image, and use the potential feature points and the first type of feature points to construct a map.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images