Image feature point tracking method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a method, device, electronic device, and system for tracking image feature points. In the embodiments of the present application, by predicting the position of the first feature point to be matched in the previous frame image in the current frame image, the position of the first feature point in the previous frame image in the current frame image is tracked based on the prediction result, thereby continuing to track the first feature point whose tracking was interrupted and extending the tracking link of the first feature point; further, based on the predicted position of each first feature point in the current frame image and the feature point to be matched in the current frame image, a grid division method is used to determine a second feature point that matches each first feature point from the feature points to be matched in multiple grid areas of the current frame image. On the basis of extending the first feature point tracking link, the accuracy of the feature point tracking link is further improved, thereby improving the accuracy of the map constructed based on the feature point tracking link.

Description

Technical Field

[0001] The present application relates to the field of visual image technology, and in particular to a method, device, electronic device and storage medium for tracking image feature points. Background Art

[0002] The fields of robotics and self-driving cars are among the hottest and most capital-intensive fields today. They share certain technical similarities, collectively referred to as simultaneous localization and mapping (SLAM). SLAM based on visual technology is called VSLAM.

[0003] Vision-based technology receives image streams captured by visual sensors and extracts and tracks feature points. The tracked feature points are used to create map landmarks. The quality (quantity and accuracy) of these landmarks depends largely on the quality (quantity, accuracy, and length) of the image feature point tracking. Low quality of image feature point tracking is currently a challenge. Summary of the Invention

[0004] Various aspects of the present application provide a method, device, electronic device, and storage medium for tracking image feature points to improve the quality of image feature point tracking, thereby improving the quality of map landmarks and VSLAM.

[0005] An exemplary embodiment of the present application provides a method for tracking image feature points, including:

[0006] Detecting a first feature point to be matched in the previous frame image and a feature point to be matched in the current frame image;

[0007] Obtain the predicted position of each first feature point in the current frame image;

[0008] Based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a grid division method is used to determine second feature points matching each first feature point from the feature points to be matched in the current frame image.

[0009] An exemplary embodiment of the present application further provides a device for tracking image feature points, comprising:

[0010] A detection module, configured to detect a first feature point to be matched in a previous frame image and a feature point to be matched in a current frame image;

[0011] An acquisition module, configured to acquire a predicted position of each first feature point in the current frame image;

[0012] The determination module is used to determine, based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a second feature point that matches each first feature point from the feature points to be matched in the current frame image using a grid division method.

[0013] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; wherein the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program to perform the steps in the above-mentioned image feature point tracking method.

[0014] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned image feature point tracking method.

[0015] The technical solutions provided by the embodiments of the present application predict the position of the first feature point to be matched in the previous frame image in the current frame image, and thus track the position of the first feature point in the previous frame image in the current frame image based on the prediction result, thereby achieving accurate positioning of the first feature point whose tracking is interrupted and continuing to track, extending the tracking link of the first feature point whose tracking is interrupted, and improving the accuracy of the map constructed according to the tracking link of the feature point; further, based on the predicted position of each first feature point in the current frame image and the feature point to be matched in the current frame image, the current frame image is divided into multiple grid areas by using a grid division method, and the second feature point matching each first feature point is determined from the feature points to be matched in the multiple grid areas of the current frame image. On the basis of extending the tracking link of the first feature point, the accuracy of determining the second feature point matching the first feature point is further improved, that is, the accuracy of feature point tracking is improved, thereby improving the accuracy of the map constructed according to the tracking link of the feature point. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A flowchart of a method for tracking image feature points provided by an exemplary embodiment of the present application;

[0018] Figure 2a A schematic diagram of a feature point prediction position provided by an exemplary embodiment of the present application;

[0019] Figure 2b A schematic diagram of grid division of a pair of current frame images provided by an exemplary embodiment of the present application;

[0020] Figure 2c Another schematic diagram of gridding a current frame image provided by an exemplary embodiment of the present application;

[0021] Figure 3 A schematic structural diagram of an image feature point tracking device provided by an exemplary embodiment of the present application;

[0022] Figure 4 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0023] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0025] Figure 1 This is a flow chart of a method for tracking image feature points provided by an exemplary embodiment of the present application. This method is applicable to applications where feature points in consecutive frame images are located and tracked, such as Figure 1 As shown, the method includes:

[0026] 101. Detecting a first feature point to be matched in a previous frame image and a feature point to be matched in a current frame image;

[0027] 102. Obtain a predicted position of each first feature point in the current frame image;

[0028] 103. Based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a second feature point matching each first feature point is determined from the feature points to be matched in the current frame image using a grid division method.

[0029] In this embodiment, there are at least two frames of images before the current frame image, wherein one frame image can be referred to as the previous frame image, and the other frame image can be referred to as the previous frame image. The feature points to be matched in the previous frame image refer to the feature points that were not tracked when the previous frame image and the previous frame image were tracking feature points. For the sake of ease of description and distinction, the feature points to be matched in the previous frame image are referred to as the first feature points. Tracking feature points means finding the position or features of the feature points in the image before the current frame image in the current frame image. Figure 2a The process shown here shows finding a point in image 2 that corresponds to feature point 1 in image 1, where image 1 is the previous frame, image 2 is the current frame, and image 1 contains feature point 1. After tracking the feature points, a feature point tracking link is formed for each feature point. Each feature point tracking link contains tracking link information, which includes but is not limited to: the feature descriptor of the feature point, its position in each frame before the current frame, its predicted position in the frame after each frame, and its tracking number.

[0030] In this embodiment, feature points in an image may be detected using a preset feature point recognition algorithm, such as the Harris corner detection algorithm or the DOG (Difference of Gaussian) algorithm, to detect certain pixels or regions in the image with stable features. The specific implementation of detecting feature points in an image using the Harris corner detection algorithm or the DOG (Difference of Gaussian) algorithm can be found in the detailed description of existing implementations and will not be elaborated upon here.

[0031] In this embodiment, when tracking feature points of the input previous frame image and the previous previous frame image, there may be a situation where some feature points fail to be tracked or are tracked incorrectly, and the tracking link of the first feature point is interrupted. After the tracking link of the first feature point is interrupted, the first feature point and its tracking link will be stored in the system variable H, and the successfully tracked feature point pair will be assigned a tracking number.

[0032] In order to continue tracking the first feature point and extend the tracking link of the first feature point, after detecting the first feature point to be matched in the previous frame image, the first feature point can be continued to be tracked based on the input current frame image. Specifically, the predicted position of each first feature point in the current frame image can be obtained. In addition, the preset feature point recognition algorithm mentioned in the above embodiment can be used to detect the feature points to be matched in the current frame image that cannot be matched with the previous frame image. Then, based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a grid division method is used to determine the second feature point that matches each first feature point from the feature points to be matched in the current frame image.

[0033] It should be noted that the specific implementation of each step in the above embodiment can be found in the relevant description of the following embodiment, which will not be repeated here.

[0034] In this embodiment, the grid division method mainly performs grid division on the current frame image based on a preset grid size, divides the current frame image into multiple grid areas, and based on the predicted position of the first feature point contained in the target grid area of ​​the multiple grid areas in the current frame image, the historical predicted position of the first feature point in the image before the current frame image, and the feature points to be matched, the second feature point matching each first feature point is determined from the current frame image by constraining conditions such as the mean and standard deviation of the angles between each predicted position contained in the target grid area and the straight line corresponding to the first feature point position in the previous frame image and the coordinate axis, the preset pixel angle range and the preset descriptor distance threshold.

[0035] The technical solution provided by the embodiment of the present application predicts the position of the first feature point to be matched in the previous frame image in the current frame image, and thus tracks the position of the first feature point in the previous frame image in the current frame image according to the prediction result, thereby achieving accurate positioning of the first feature point whose tracking is interrupted, and continuing tracking, extending the tracking link of the first feature point whose tracking is interrupted, and improving the accuracy of the map constructed according to the tracking link of the feature point; further, based on the predicted position of each first feature point in the current frame image and the feature point to be matched in the current frame image, the current frame image is divided into multiple grid areas by using a grid division method, and the second feature point matching each first feature point is determined from the feature points to be matched in the multiple grid areas of the current frame image. On the basis of extending the tracking link of the first feature point, the accuracy of determining the second feature point matching the first feature point is further improved, that is, the accuracy of feature point tracking is improved, thereby improving the accuracy of the map constructed according to the tracking link of the feature point.

[0036] In this embodiment, the predicted position of each first feature point in the current frame image is obtained. A specific implementation method is as follows: for each first feature point, the predicted position of each first feature point in the current frame image is obtained using the optical flow tracking method.

[0037] Furthermore, after obtaining the predicted position of each first feature point in the current frame image, a grid division method can be used to determine a second feature point that matches each first feature point from the feature points to be matched in the current frame image based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image. A specific implementation method is as follows:

[0038] A1. Based on a preset grid size, the current frame image is gridded to obtain a plurality of grid regions and their position distribution in the current frame image. Based on the position distribution, the attribution relationship between each predicted position and the feature point to be matched and each grid region is determined.

[0039] A2. Based on the attribution relationship between each predicted position, the feature point to be matched, and each grid area, determine the mean and standard deviation of the first angles between the first connecting line of all predicted positions in the target grid area and the corresponding first feature point position in the previous frame of the image and the coordinate axis;

[0040] A3. Based on the angle mean and the angle standard deviation, select candidate prediction positions contained in the target grid area from the prediction positions;

[0041] A4. Select feature points that satisfy both a preset pixel angle range and a preset descriptor distance threshold from the neighborhood of the candidate prediction position as second feature points that match each first feature point.

[0042] In this embodiment, the grid size includes the length and width of the grid area. The length and width of the grid area can be equal or unequal. When the length and width of the grid area are equal, the current frame image can be evenly divided into N*N grid areas. When the length and width of the grid area are unequal, the current frame image can be divided into N*M grid areas, where N≠M. After the current frame image is grid-divided in the above manner, multiple grid areas and the position distribution of the multiple grid areas in the current frame image may be obtained. Taking a 3*3 grid area as an example, the position distribution of the grid in the current frame image is as follows: Figure 2b As shown. Taking the 3*4 grid area as an example, the position distribution of the grid in the current frame image is as follows Figure 2c Furthermore, based on the location distribution, the attribution relationship between each predicted location and the feature point to be matched and each grid area can be determined.

[0043] In an optional embodiment, a specific implementation method of determining the attribution relationship between each predicted position and the feature point to be matched and each grid area based on the position distribution is: based on the position distribution, determine the grid area to which each preset position and the feature point to be matched in the current frame image belongs.

[0044] Furthermore, based on the attribution relationship between each predicted position, the feature point to be matched, and each grid area, the first connecting line of all predicted positions in the target grid area and their corresponding first feature point positions in the previous frame image, the mean value and standard deviation of the first angles between the first connecting line and the coordinate axis can be determined. The specific implementation method of this step is as follows:

[0045] B1. Select any grid area from the multiple grid areas. For ease of description and distinction, this grid area may be referred to as the target grid area.

[0046] B2. For all predicted positions in the target grid area, determine a plurality of first angles between a first connecting line between each predicted position in the target grid and its corresponding first feature point position in the previous frame image and the coordinate axis;

[0047] B3. determining a first angle average based on the multiple first angles;

[0048] B4. Determine an angle standard deviation based on multiple first angles and the first angle mean.

[0049] In this embodiment, the target grid area may or may not include the predicted position. Based on the inclusion of the predicted position, a specific implementation method for determining multiple first angles between the first connecting line between each predicted position in the target grid and its corresponding first feature point position in the previous frame image and the coordinate axis is as follows:

[0050] C1. Determine the connecting line between each predicted position in the target grid and the corresponding first feature point position in the previous frame image;

[0051] C2. taking the angle between each first connecting line and the X-axis of the coordinate axis as a plurality of first included angles;

[0052] or,

[0053] C2. Calculate the angle between each first connecting line and the Y axis of the coordinate axis, and use the supplementary angle of the angle between each first connecting line and the Y axis of the coordinate axis as the multiple first included angles.

[0054] Furthermore, after determining multiple first angles between the first connecting line and the coordinate axis between each predicted position in the target grid and its corresponding first feature point position in the previous frame image, the average value can be calculated based on the multiple first angles to obtain the first angle mean.

[0055] After determining the first angle mean and the angle standard deviation, a candidate prediction position included in the target grid area may be selected from the prediction positions based on the first angle mean and the angle standard deviation. The specific implementation thereof is as follows:

[0056] D1. Determine the angle reference interval based on the first angle mean and angle standard deviation;

[0057] D2. Retain the candidate prediction positions in the target grid area that are within the angle reference interval.

[0058] In this embodiment, the specific implementation method for determining the angle reference interval based on the first angle mean and angle standard deviation is as follows: the difference between the first angle mean and the angle standard deviation is used as the minimum boundary value of the angle reference interval, and the sum of the first angle mean and the angle standard deviation is used as the maximum boundary value of the angle reference interval, thereby determining the angle reference interval and retaining candidate prediction positions in the target grid area that fall within the angle reference interval. Taking the first angle mean as Aavg and the angle standard deviation as Aδ as an example, the angle reference interval is (Aavg - Aδ, Aavg + Aδ).

[0059] Further optionally, after the candidate prediction position is determined, a feature point that satisfies both a preset pixel angle range and a preset descriptor distance threshold is selected from the neighborhood of the candidate prediction position as a second feature point that matches each of the first feature points.

[0060] In an optional embodiment, selecting a feature point that satisfies a preset pixel angle range from a neighborhood of the candidate prediction position as a second feature point that matches each first feature point includes:

[0061] E1. Obtain the candidate prediction positions contained in the neighborhood of each candidate prediction position in the target grid area;

[0062] E2. Based on the candidate prediction positions included in the neighborhood of each candidate prediction position in the target grid area, determine a plurality of second angles between a second connecting line between each candidate prediction position included in the neighborhood of each candidate prediction position and each to-be-matched feature point in the target grid area and the coordinate axis;

[0063] E3. Obtain a tracking link for each first feature point and save it to a system variable. The tracking link contains tracking link information, including: a feature descriptor of each first feature point, its position in each frame before the current frame, its predicted position in the frame after each frame, and a tracking number.

[0064] E4. Based on the tracking link information of each first feature point, determine the average of the second angles between the third connecting line between each predicted position and the previous predicted position in the tracking link of each first feature point and the coordinate axis;

[0065] E5. Determine a preset pixel angle range based on the second angle mean and the preset offset angle;

[0066] E6. Take the multiple second angles between the second connecting line between each candidate prediction position contained in the neighborhood of each candidate prediction position and each feature point to be matched in the target grid area and the coordinate axis, and the feature points to be matched within the preset pixel angle range, as the second feature points matching each first feature point.

[0067] Among them, based on the candidate prediction positions included in the neighborhood of each candidate prediction position in the target grid area, a specific implementation method of determining multiple second angles between the second connecting line between each candidate prediction position included in the neighborhood of each candidate prediction position and each feature point to be matched in the target grid area and the coordinate axis is as follows:

[0068] F1. Determine a second connecting line between each candidate prediction position contained in the neighborhood of each candidate prediction position and each feature point to be matched in the target grid area;

[0069] F2. The angle between each second connecting line and the X axis of the coordinate axis is used as a plurality of second included angles;

[0070] or,

[0071] F3. Calculate the angle between each second connecting line and the Y axis of the coordinate axis, and use the supplementary angle of the angle between each second connecting line and the Y axis of the coordinate axis as a plurality of second included angles.

[0072] Furthermore, after determining the multiple second angles between the second connecting lines of each candidate prediction position and each feature point to be matched in the target grid area contained in the neighborhood of each candidate prediction position and the coordinate axis, the average value can be calculated based on the multiple second angles to obtain the second angle mean. Further, the second angle mean has a preset offset angle, and the preset pixel angle range can be determined based on the second angle mean and the preset offset angle. More specifically, the difference between the second angle mean and the preset offset angle can be used as the minimum boundary value of the pixel angle range, and the sum of the second angle mean and the preset offset angle can be used as the maximum boundary value of the pixel angle range, thereby determining the pixel angle range. For example, taking the second angle mean as Ah and the preset offset angle as θ as an example, the pixel angle range is (Ah-θ, Ah +θ).

[0073] In an optional embodiment, a specific implementation of selecting a feature point that meets a preset descriptor distance threshold from a neighborhood of a candidate prediction position as a second feature point that matches each first feature point is as follows:

[0074] G1. Calculate the descriptor distance between the feature descriptor of each first feature point and the feature descriptor of the feature point matched in the current frame image;

[0075] G2. Use a feature point whose descriptor distance is less than a preset descriptor distance threshold as a second feature point that matches each first feature point.

[0076] Furthermore, after obtaining the second feature point that matches each first feature point, the first feature point matched to the feature point and its tracking link can be deleted from the system constant H, and the remaining first feature points to be matched in the system constant H can continue to be predicted and tracked using the method in the above embodiment.

[0077] It should be noted that, for the first feature point to be matched in the system variable H, each time feature point tracking is performed and the tracking fails to track a feature point that matches the first feature point to be matched, the loss count of the first feature point to be matched is increased by 1. When the loss count is greater than the preset value and the first feature point has not tracked a feature point that matches it, the first feature point is deleted from the system variable H.

[0078] Figure 3 This is a schematic diagram of the structure of an image feature point tracking device provided by an exemplary embodiment of the present application. Figure 3 Shown, including:

[0079] A detection module 31 is configured to detect a first feature point to be matched in a previous frame image and a feature point to be matched in a current frame image;

[0080] An acquisition module 32 is configured to acquire a predicted position of each first feature point in the current frame image;

[0081] The determination module 33 is used to determine, based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a second feature point that matches each first feature point from the feature points to be matched in the current frame image using a grid division method.

[0082] In an optional embodiment, the acquisition module 32, when used to obtain the predicted position of each first feature point in the current frame image, is specifically used to:

[0083] For each of the first feature points, an optical flow tracking method is used to obtain a predicted position of each of the first feature points in the current frame image.

[0084] In an optional embodiment, the determining module 33 is configured to determine, based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a second feature point matching each first feature point from the feature points to be matched in the current frame image using a grid division method, specifically for:

[0085] Based on a preset grid size, the current frame image is grid-divided to obtain multiple grid areas and the position distribution of the multiple grid areas in the current frame image, and based on the position distribution, the attribution relationship between each predicted position, the feature point to be matched and each grid area is determined; based on the attribution relationship between each predicted position, the feature point to be matched and each grid area, the first angle mean and angle standard deviation between the first connecting line of all predicted positions contained in the target grid area and the first feature point position corresponding to it in the previous frame image and the coordinate axis are determined; based on the angle mean and the angle standard deviation, candidate predicted positions contained in the target grid area are selected from the predicted positions; and feature points that simultaneously meet a preset pixel angle range and a preset descriptor distance threshold are selected from the neighborhood of the candidate predicted positions as second feature points that match each first feature point.

[0086] In an optional embodiment, the determination module 33, when used to determine, based on the attribution relationship between each predicted position, the feature point to be matched, and each grid area, a mean and a standard deviation of first angles between a first connecting line of all predicted positions in the target grid area and their corresponding first feature point positions in the previous image and a coordinate axis, is specifically configured to:

[0087] For all predicted positions in the target grid area, determine multiple first angles between a first connecting line between each predicted position in the target grid and its corresponding first feature point position in the previous frame image and the coordinate axis; based on the multiple first angles, determine a first angle mean; based on the multiple first angles and the first angle mean, determine an angle standard deviation.

[0088] In an optional embodiment, the determination module 33, when configured to select, from the predicted positions, the candidate predicted positions included in the target grid area based on the included angle mean and the angle standard deviation, is specifically configured to:

[0089] An angle reference interval is determined based on the angle mean and the angle standard deviation; and candidate prediction positions in the target grid area that are within the angle reference interval are retained.

[0090] In an optional embodiment, the determining module 33 selects a feature point that satisfies a preset pixel angle range from a neighborhood of the candidate prediction position as a second feature point that matches each first feature point, including:

[0091] Obtain the candidate prediction positions contained in the neighborhood of each candidate prediction position in the target grid area; based on the candidate prediction positions contained in the neighborhood of each candidate prediction position in the target grid area, determine a plurality of second angles between the second connecting line of each candidate prediction position contained in the neighborhood of each candidate prediction position and each feature point to be matched in the target grid area and the coordinate axis; obtain the tracking link of each first feature point and save it to the system variable, wherein the tracking link contains tracking link information, and the tracking link information includes: the feature descriptor of each first feature point, the position in each frame image before the current frame image, and the position in each frame image before the current frame image. The predicted position and tracking number in the next frame of image; based on the tracking link information of each first feature point, determine the average of the second angles between the third connecting line of each predicted position and the previous predicted position in the tracking link of each first feature point and the coordinate axis; based on the average of the second angles and the preset offset angle, determine the preset pixel angle range; take the multiple second angles between the second connecting line of each candidate predicted position contained in the neighborhood of each candidate predicted position and each feature point to be matched in the target grid area and the coordinate axis, and the corresponding feature points to be matched within the preset pixel angle range, as the second feature points matching each first feature point.

[0092] In an optional embodiment, the determining module 33 selects a feature point that satisfies a preset descriptor distance threshold from a neighborhood of the candidate prediction position as a second feature point that matches each first feature point, including:

[0093] Calculate the descriptor distance between the feature descriptor of each first feature point and the feature descriptor of the feature point matched in the current frame image; and use the feature point whose descriptor distance is less than the preset descriptor distance threshold as the second feature point matched with each first feature point.

[0094] It should be noted that the specific implementation of the above modules or units provided in this embodiment can refer to the relevant description of the above method embodiment, and will not be repeated here.

[0095] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 4 As shown, it includes: a memory 40a and a processor 40b; wherein the memory 40a is used to store a computer program; the processor 40b is coupled to the memory 40a, and is used to execute the computer program to perform the following steps:

[0096] Detect first feature points to be matched in the previous frame image and feature points to be matched in the current frame image; obtain a predicted position of each first feature point in the current frame image; and determine, from the feature points to be matched in the current frame image, a second feature point that matches each first feature point using a grid division method based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image.

[0097] In an optional embodiment, when the processor 40b is used to obtain the predicted position of each first feature point in the current frame image, it is specifically used to:

[0098] For each of the first feature points, an optical flow tracking method is used to obtain a predicted position of each of the first feature points in the current frame image.

[0099] In an optional embodiment, when the processor 40b is configured to determine, from the feature points to be matched in the current frame image, a second feature point matching each first feature point based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, using a grid division method, the processor 40b is specifically configured to:

[0100] Based on a preset grid size, the current frame image is grid-divided to obtain multiple grid areas and the position distribution of the multiple grid areas in the current frame image, and based on the position distribution, the attribution relationship between each predicted position, the feature point to be matched and each grid area is determined; based on the attribution relationship between each predicted position, the feature point to be matched and each grid area, the first angle mean and angle standard deviation between the first connecting line of all predicted positions contained in the target grid area and the first feature point position corresponding to it in the previous frame image and the coordinate axis are determined; based on the angle mean and the angle standard deviation, candidate predicted positions contained in the target grid area are selected from the predicted positions; and feature points that simultaneously meet a preset pixel angle range and a preset descriptor distance threshold are selected from the neighborhood of the candidate predicted positions as second feature points that match each first feature point.

[0101] In an optional embodiment, the processor 40b, when used to determine, based on the attribution relationship between each predicted position, the feature point to be matched, and each grid area, a mean and a standard deviation of first angles between a first connecting line of all predicted positions included in the target grid area and their corresponding first feature point positions in the previous image and a coordinate axis, is specifically configured to:

[0102] For all predicted positions in the target grid area, determine multiple first angles between a first connecting line between each predicted position in the target grid and its corresponding first feature point position in the previous frame image and the coordinate axis; based on the multiple first angles, determine a first angle mean; based on the multiple first angles and the first angle mean, determine an angle standard deviation.

[0103] In an optional embodiment, when the processor 40b is configured to select, from the predicted positions based on the included angle mean and the angle standard deviation, the candidate predicted position included in the target grid area, is specifically configured to:

[0104] An angle reference interval is determined based on the angle mean and the angle standard deviation; and candidate prediction positions in the target grid area that are within the angle reference interval are retained.

[0105] In an optional embodiment, the processor 40b selects a feature point that satisfies a preset pixel angle range from a neighborhood of the candidate prediction position as a second feature point that matches each first feature point, including:

[0106] Obtain the candidate prediction positions contained in the neighborhood of each candidate prediction position in the target grid area; based on the candidate prediction positions contained in the neighborhood of each candidate prediction position in the target grid area, determine a plurality of second angles between the second connecting line of each candidate prediction position contained in the neighborhood of each candidate prediction position and each feature point to be matched in the target grid area and the coordinate axis; obtain the tracking link of each first feature point and save it to the system variable, wherein the tracking link contains tracking link information, and the tracking link information includes: the feature descriptor of each first feature point, the position in each frame image before the current frame image, and the position in each frame image before the current frame image. The predicted position and tracking number in the next frame of image; based on the tracking link information of each first feature point, determine the average of the second angles between the third connecting line of each predicted position and the previous predicted position in the tracking link of each first feature point and the coordinate axis; based on the average of the second angles and the preset offset angle, determine the preset pixel angle range; take the multiple second angles between the second connecting line of each candidate predicted position contained in the neighborhood of each candidate predicted position and each feature point to be matched in the target grid area and the coordinate axis, and the corresponding feature points to be matched within the preset pixel angle range, as the second feature points matching each first feature point.

[0107] In an optional embodiment, the processor 40b selects a feature point that satisfies a preset descriptor distance threshold from a neighborhood of the candidate prediction position as a second feature point that matches each first feature point, including:

[0108] Calculate the descriptor distance between the feature descriptor of each first feature point and the feature descriptor of the feature point matched in the current frame image; and use the feature point whose descriptor distance is less than the preset descriptor distance threshold as the second feature point matched with each first feature point.

[0109] Furthermore, if Figure 4 As shown, the electronic device also includes: a communication component 40c, a power supply component 40d and other components. Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 The electronic device of this embodiment can be implemented as an electronic device such as a desktop computer, a laptop computer, a smart phone or an IOT device.

[0110] It should be noted that the specific implementation of the above modules or units provided in this embodiment can refer to the relevant description of the above method embodiment, and will not be repeated here.

[0111] It should be noted that the specific implementation of the above modules or units provided in this embodiment can refer to the relevant description of the above method embodiment, and will not be repeated here.

[0112] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions, characterized in that when the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned image feature point tracking method.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0117] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0118] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0119] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0121] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for tracking image feature points, characterized in that: include: Detecting a first feature point to be matched in the previous frame image and a feature point to be matched in the current frame image; Obtaining a predicted position of each first feature point in the current frame image; Based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, using a grid division method, determining a second feature point that matches each first feature point from the feature points to be matched in the current frame image; The method further comprises: determining, based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a second feature point matching each first feature point from the feature points to be matched in the current frame image using a grid division method, comprising: Based on a preset grid size, the current frame image is grid-divided to obtain a plurality of grid areas and a position distribution of the plurality of grid areas in the current frame image, and based on the position distribution, an attribution relationship between each predicted position, the feature point to be matched, and each grid area is determined; Determine, based on the attribution relationship between each predicted position, the feature point to be matched, and each grid area, a mean and a standard deviation of first angles between a first connecting line of all predicted positions in the target grid area and their corresponding first feature point positions in the previous image and a coordinate axis; selecting, from the predicted positions, a candidate predicted position included in the target grid area based on the included angle mean and the angle standard deviation; A feature point that satisfies both a preset pixel angle range and a preset descriptor distance threshold is selected from the neighborhood of the candidate prediction position as a second feature point that matches each first feature point.

2. The method according to claim 1, characterized in that Obtaining a predicted position of each first feature point in the current frame image includes: For each of the first feature points, an optical flow tracking method is used to obtain a predicted position of each first feature point in the current frame image.

3. The method according to claim 1, characterized in that Determining, based on the attribution relationship between each predicted position, the feature point to be matched, and each grid area, a first angle mean and an angle standard deviation between a first connecting line of all predicted positions contained in the target grid area and their corresponding first feature point positions in the previous frame of image and a coordinate axis, including: For all predicted positions in the target grid area, determining a plurality of first angles between a first connecting line between each predicted position in the target grid area and its corresponding first feature point position in the previous frame image and a coordinate axis; Determining a first angle average based on the multiple first angles; An angle standard deviation is determined based on the multiple first angles and the first angle mean.

4. The method according to claim 1, wherein Selecting, from the predicted positions, a candidate predicted position included in the target grid area based on the included angle mean and the angle standard deviation, comprising: Determining an angle reference interval based on the angle mean and the angle standard deviation; The candidate prediction positions in the target grid area that are within the angle reference interval are retained.

5. The method according to claim 1, wherein Selecting a feature point that satisfies a preset pixel angle range from a neighborhood of the candidate prediction position as a second feature point that matches each first feature point, including: Obtain the candidate prediction positions contained in the neighborhood of each candidate prediction position in the target grid area; Determining, based on the candidate prediction positions included in the neighborhood of each candidate prediction position in the target grid area, a plurality of second angles between a second connecting line between each candidate prediction position included in the neighborhood of each candidate prediction position and each feature point to be matched in the target grid area and the coordinate axis; Obtain a tracking link for each first feature point and save it to a system variable, wherein the tracking link includes tracking link information, including: a feature descriptor of each first feature point, a position in each frame before the current frame, a predicted position in a frame after each frame, and a tracking number; Based on the tracking link information of each first feature point, determining an average of second angles between a third connecting line between each predicted position and a previous predicted position in the tracking link of each first feature point and the coordinate axis; Determining the preset pixel angle interval based on the second angle mean and the preset offset angle; The multiple second angles between the second connecting line of each candidate prediction position contained in the neighborhood of each candidate prediction position and each feature point to be matched in the target grid area and the coordinate axis, and the feature points to be matched corresponding to the preset pixel angle range, are used as the second feature points matching each first feature point.

6. The method according to claim 1, characterized in that Selecting a feature point that satisfies a preset descriptor distance threshold from a neighborhood of the candidate prediction position as a second feature point that matches each first feature point, comprising: Calculating a descriptor distance between a feature descriptor of each first feature point and a feature descriptor of a feature point matched in the current frame image; The feature point whose descriptor distance is less than the preset descriptor distance threshold is used as the second feature point that matches each first feature point.

7. A device for tracking image feature points, characterized in that: include: A detection module, configured to detect a first feature point to be matched in a previous frame image and a feature point to be matched in a current frame image; An acquisition module, configured to acquire a predicted position of each first feature point in the current frame image; a determining module, configured to determine, based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a second feature point that matches each first feature point from the feature points to be matched in the current frame image using a grid division method; The determining module is specifically configured to determine, based on the predicted position of each first feature point in the current frame image and the feature points to be matched in the current frame image, a second feature point matching each first feature point from the feature points to be matched in the current frame image using a grid division method: Based on a preset grid size, the current frame image is grid-divided to obtain a plurality of grid areas and a position distribution of the plurality of grid areas in the current frame image, and based on the position distribution, an attribution relationship between each predicted position, the feature point to be matched, and each grid area is determined; Determine, based on the attribution relationship between each predicted position, the feature point to be matched, and each grid area, a mean and a standard deviation of first angles between a first connecting line of all predicted positions in the target grid area and their corresponding first feature point positions in the previous image and a coordinate axis; selecting, from the predicted positions, a candidate predicted position included in the target grid area based on the included angle mean and the angle standard deviation; A feature point that satisfies both a preset pixel angle range and a preset descriptor distance threshold is selected from the neighborhood of the candidate prediction position as a second feature point that matches each first feature point.

8. An electronic device, characterized in that: include: A memory and a processor; wherein the memory is used to store a computer program; The processor is coupled to the memory and is configured to execute the computer program to perform the steps of the image feature point tracking method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the image feature point tracking method according to any one of claims 1 to 6.

Images