Image feature tracking method, device and electronic equipment

Abstract

The present application provides an image feature tracking method, device and electronic device, wherein the method includes: determining a feature point set of each current image frame for an acquired image frame sequence, wherein the feature point set includes: optical flow tracking points; performing descriptor matching on each feature point in the feature point set to obtain tracking information of the descriptor tracking point; and determining the feature tracking result of the current image frame based on the tracking information of the optical flow tracking point and the descriptor tracking point. In the process of determining the image feature tracking result, the present application effectively combines the advantages of optical flow feature tracking and descriptor feature tracking. For image sequences captured in complex scenes such as fast or intense motion scenes, scenes with insufficient light, etc., devices with limited computing power can also accurately and stably track feature points representing the same object, thereby improving the visual tracking effect and reducing the visual tracking power consumption of the device.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image feature tracking method, device and electronic device. Background Art

[0002] Image feature tracking is an important functional module of the visual SLAM (simultaneous localization and mapping) system, which directly affects the efficiency and robustness of the SLAM system. In related technologies, optical flow method or descriptor method can be used to perform feature tracking when the camera quickly captures scene images. Among them, the use of optical flow method can efficiently track features in fast motion, but the weak matching characteristics of the optical flow method itself make the tracking points prone to drift; and the use of descriptor method for feature tracking, although the matching accuracy is higher and the drift is less in the tracking results, the feature points are easy to lose in fast motion, and the use of descriptor method alone usually requires tracking more feature points than the optical flow method, and consumes more computing resources to achieve the ideal tracking effect.

[0003] In actual application scenarios of augmented reality (AR) on electronic devices such as mobile phones, if the user carries the electronic device and moves it quickly, or if factors such as the limited imaging quality of the image sensor affect it, it is often impossible to simultaneously take into account the efficiency and robustness of feature tracking, resulting in a poor AR experience for the user. Summary of the invention

[0004] Embodiments of the present application provide an image feature tracking method, device, and electronic device.

[0005] According to a first aspect of an embodiment of the present application, there is provided an image feature tracking method, comprising:

[0006] For the acquired image frame sequence, determining a feature point set of each current image frame, wherein the feature point set includes: an optical flow tracking point;

[0007] Performing descriptor matching on each feature point in the feature point set to obtain tracking information of the descriptor tracking point;

[0008] The feature tracking result of the current image frame is determined according to the tracking information of the optical flow tracking point and the descriptor tracking point.

[0009] In conjunction with any method embodiment provided in the present application, determining a feature point set of each current image frame includes:

[0010] Acquire optical flow tracking information of a current image frame, and determine an optical flow tracking point in the current image frame based on the optical flow tracking information;

[0011] Extracting feature points from the current image frame to obtain feature extraction points in the current image frame;

[0012] Determining a feature point set of the current image frame according to the optical flow tracking points and the feature extraction points;

[0013] The feature point set of the current image frame includes two types of feature points: the optical flow tracking points and the feature extraction points.

[0014] In combination with any method embodiment provided in the present application, performing descriptor matching on each feature point in the feature point set to obtain tracking information of the descriptor tracking point includes:

[0015] Calculating a descriptor for each of the feature points in the feature point set to obtain a descriptor for the target feature point;

[0016] Feature point matching is performed based on the descriptor of the target feature point to determine tracking information of the descriptor tracking point.

[0017] In combination with any of the method embodiments provided in the present application, the matching of feature points based on the descriptor of the target feature point to determine the tracking information of the descriptor tracking point includes:

[0018] For each first target feature point in the previous image frame, determining a search range center of the first target feature point in the current image frame, wherein the first target feature point is a feature point having a descriptor in the previous image frame;

[0019] Determine a descriptor of each second target feature point within a target search range, wherein the target search range is a range determined based on the center of the search range;

[0020] Feature point matching is performed based on the descriptor of the first target feature point and the descriptors of each of the second target feature points to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

[0021] In combination with any method embodiment provided in the present application, the first target feature point has a corresponding target optical flow tracking point in the current image frame;

[0022] The step of determining the search range center of the first target feature point in the current image frame includes:

[0023] Determining whether the three-dimensional position of the first target feature point has been initialized;

[0024] If the three-dimensional position of the first target feature point has been initialized, determining a predicted projection position of the initialized three-dimensional point in the current image frame;

[0025] Determining a first distance between the target optical flow tracking point and the projected predicted position;

[0026] If the first distance is less than or equal to a preset first distance threshold, determining the target optical flow tracking point as the center of the search range;

[0027] If the first distance is greater than the first preset distance threshold, the projected predicted position is determined as the center of the search range.

[0028] In combination with any of the method embodiments provided in the present application, the step of determining the search range center of the first target feature point in the current image frame further includes:

[0029] If the three-dimensional position of the first target feature point is not initialized, the target optical flow tracking point is determined as the center of the search range.

[0030] In combination with any method embodiment provided in the present application, the first target feature point has no corresponding target optical flow tracking point in the current image frame;

[0031] The step of determining the search range center of the first target feature point in the current image frame includes:

[0032] Determining whether the three-dimensional position of the first target feature point has been initialized;

[0033] If the three-dimensional position of the first target feature point has been initialized, determining a predicted projection position of the initialized three-dimensional point in the current image frame;

[0034] The projected predicted position is determined as the center of the search range.

[0035] In combination with any of the method embodiments provided in the present application, the step of determining the search range center of the first target feature point in the current image frame further includes:

[0036] If the three-dimensional position of the first target feature point is not initialized, determining a homography matrix representing a mapping relationship between the current image frame and the previous image frame;

[0037] The center of the search range of the first target feature point is determined according to the homography matrix.

[0038] In combination with any method embodiment provided in the present application, the determining a homography matrix representing a mapping relationship between the current image frame and the previous image frame includes:

[0039] Determining the number of the optical flow tracking points in the feature point set;

[0040] If the number of the optical flow tracking points is greater than a preset number threshold, determining pixel coordinate information of each optical flow matching point pair according to the optical flow tracking information of the optical flow tracking points;

[0041] Determine the homography matrix according to the pixel coordinate information of each of the optical flow matching point pairs;

[0042] The optical flow matching point pair is a feature point pair matched in the previous image frame and the current image frame using an optical flow tracking method.

[0043] In conjunction with any of the method embodiments provided in the present application, the tracking information of the descriptor tracking point includes: a tracking point position and related tracking information, wherein the related tracking information is used to indicate a method for determining a search range center during a process of determining the descriptor tracking point;

[0044] The step of determining the feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point includes:

[0045] Determining an initial feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point;

[0046] Feature points are removed from the initial feature tracking result according to relevant tracking information of each tracking point to obtain a target feature tracking result.

[0047] In combination with any of the method embodiments provided in the present application, if a first target feature point in a previous image frame is matched to an optical flow tracking point and a descriptor tracking point in the current image frame respectively;

[0048] The step of determining the initial feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point comprises:

[0049] For the same first target feature point, determining a second distance between the optical flow tracking point and the descriptor tracking point;

[0050] If the second distance is less than or equal to a preset second distance threshold, determining the optical flow tracking point as a target tracking point of the first target feature point;

[0051] If the second distance is greater than the preset second distance threshold, the descriptor tracking point is determined as the target tracking point of the first target feature point.

[0052] In combination with any of the method embodiments provided in the present application, the step of removing feature points from the initial feature tracking result according to the relevant tracking information of each tracking point to obtain the target feature tracking result includes:

[0053] Determining the score of each tracking point in the initial feature tracking result according to the relevant tracking information of each tracking point and a preset weight scoring rule;

[0054] Sorting the tracking points in descending order according to the tracking point scores to obtain a tracking point sorting list, wherein the tracking point sorting list includes: a corresponding relationship between tracking point indexes and tracking point scores;

[0055] According to the tracking point sorting list, tracking points with similar distances in the current image frame are eliminated to obtain the target feature tracking result.

[0056] In conjunction with any method embodiment provided in the present application, the ranking of the scores of the tracking points in descending order of the tracking point scores includes:

[0057] If the scores of multiple tracking points are equal, the scores of the tracking points are updated using the number of times each of the tracking points is tracked in the previous image frame;

[0058] The plurality of tracking points are sorted based on the updated tracking point scores.

[0059] According to a second aspect of an embodiment of the present application, a scene creation method is provided, including:

[0060] Get image frame sequence;

[0061] Using any of the image feature tracking method embodiments provided in the present application to track image feature points of each current image frame in the image frame sequence, to obtain an image feature tracking result;

[0062] estimating the position information and three-dimensional point cloud information of the terminal device according to the image feature tracking results of the image frame sequence;

[0063] An augmented reality scene is created based on the posture information and three-dimensional point cloud information of the terminal device.

[0064] According to a third aspect of an embodiment of the present application, there is provided an image feature tracking device, comprising:

[0065] The feature point determination module is configured to determine a feature point set of each current image frame for the acquired image frame sequence, wherein the feature point set includes: an optical flow tracking point;

[0066] A tracking information determination module is configured to perform descriptor matching on each feature point in the feature point set to obtain tracking information of the descriptor tracking point;

[0067] The feature tracking module is configured to determine the feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point.

[0068] In conjunction with any device embodiment provided in this application, the feature point determination module includes:

[0069] an optical flow tracking submodule, configured to obtain optical flow tracking information of a current image frame, and determine an optical flow tracking point in the current image frame based on the optical flow tracking information;

[0070] A feature point extraction submodule is configured to extract feature points from the current image frame to obtain feature extraction points in the current image frame;

[0071] A feature point set determination submodule, configured to determine a feature point set of the current image frame according to the optical flow tracking points and the feature extraction points;

[0072] The feature point set of the current image frame includes two types of feature points: the optical flow tracking points and the feature extraction points.

[0073] In conjunction with any device embodiment provided in this application, the tracking information determination module includes:

[0074] A descriptor determination submodule is configured to calculate a descriptor for each of the feature points in the feature point set to obtain a descriptor for a target feature point;

[0075] The descriptor tracking submodule is configured to perform feature point matching based on the descriptor of the target feature point to determine tracking information of the descriptor tracking point.

[0076] In conjunction with any device embodiment provided in the present application, the descriptor tracking submodule includes:

[0077] A search center determination unit is configured to determine, for each first target feature point in a previous image frame, a search range center of the first target feature point in the current image frame, wherein the first target feature point is a feature point having a descriptor in the previous image frame;

[0078] A descriptor determining unit configured to determine a descriptor of each second target feature point within a target search range, wherein the target search range is a range determined based on a center of the search range;

[0079] The matching unit is configured to perform feature point matching based on the descriptor of the first target feature point and the descriptor of each of the second target feature points to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

[0080] In combination with any device embodiment provided in the present application, the first target feature point has a corresponding target optical flow tracking point in the current image frame;

[0081] The search center determination unit includes:

[0082] a position initialization determination subunit, configured to determine whether the three-dimensional position of the first target feature point has been initialized;

[0083] a projection position determination subunit, configured to determine a projection prediction position of the initialized three-dimensional point in the current image frame when the three-dimensional position of the first target feature point has been initialized;

[0084] A first distance determination subunit is configured to determine a first distance between the target optical flow tracking point and the projection prediction position;

[0085] A first search center determination subunit is configured to determine the target optical flow tracking point as the center of the search range when the first distance is less than or equal to a preset first distance threshold;

[0086] The second search center determination subunit is configured to determine the projection prediction position as the search range center when the first distance is greater than the first preset distance threshold.

[0087] In combination with any device embodiment provided in the present application, the search center determination unit is configured to determine the target optical flow tracking point as the center of the search range when the three-dimensional position of the first target feature point is not initialized.

[0088] In combination with any device embodiment provided in the present application, the first target feature point has no corresponding target optical flow tracking point in the current image frame;

[0089] The search center determination unit includes:

[0090] a position initialization determination subunit, configured to determine whether the three-dimensional position of the first target feature point has been initialized;

[0091] a projection position determination subunit, configured to determine a projection prediction position of the initialized three-dimensional point in the current image frame when the three-dimensional position of the first target feature point has been initialized;

[0092] The third search center determination subunit is configured to determine the projection prediction position as the search range center.

[0093] In combination with any device embodiment provided in the present application, the search center determination unit further includes:

[0094] a homography matrix determination subunit, configured to determine a homography matrix representing a mapping relationship between the current image frame and the previous image frame when the three-dimensional position of the first target feature point is not initialized;

[0095] The fourth search center determination subunit is configured to determine the search range center of the first target feature point according to the homography matrix.

[0096] In conjunction with any device embodiment provided in the present application, the homography matrix determination subunit includes:

[0097] a quantity determination unit, configured to determine the quantity of the optical flow tracking points in the feature point set;

[0098] a pixel coordinate determining unit configured to determine pixel coordinate information of each optical flow matching point pair according to optical flow tracking information of the optical flow tracking points if the number of the optical flow tracking points is greater than a preset number threshold;

[0099] A matrix determination unit, configured to determine the homography matrix according to the pixel coordinate information of each of the optical flow matching point pairs;

[0100] The optical flow matching point pair is a feature point pair matched in the previous image frame and the current image frame using an optical flow tracking method.

[0101] In conjunction with any device embodiment provided in the present application, the tracking information of the descriptor tracking point includes: a tracking point position and related tracking information, wherein the related tracking information is used to indicate a method for determining a search range center during a process of determining the descriptor tracking point;

[0102] The feature tracking module comprises:

[0103] an initial tracking result determination submodule, configured to determine an initial feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point;

[0104] The target tracking result determination submodule is configured to remove feature points from the initial feature tracking result according to relevant tracking information of each tracking point to obtain a target feature tracking result.

[0105] In combination with any device embodiment provided in the present application, if a first target feature point in a previous image frame is matched to an optical flow tracking point and a descriptor tracking point in the current image frame respectively;

[0106] The initial tracking result determination submodule includes:

[0107] A second distance determination unit is configured to determine a second distance between the optical flow tracking point and the descriptor tracking point for the same first target feature point;

[0108] A first target determination unit is configured to determine the optical flow tracking point as a target tracking point of the first target feature point when the second distance is less than or equal to a preset second distance threshold;

[0109] The second target determination unit is configured to determine the descriptor tracking point as a target tracking point of the first target feature point when the second distance is greater than the preset second distance threshold.

[0110] In conjunction with any device embodiment provided in this application, the target tracking result determination submodule includes:

[0111] A score determination unit, configured to determine the score of each tracking point in the initial feature tracking result according to relevant tracking information of each tracking point and a preset weight scoring rule;

[0112] A sorting unit is configured to sort the tracking points in descending order of the tracking point scores to obtain a tracking point sorting list, wherein the tracking point sorting list includes: a corresponding relationship between the tracking point index and the tracking point score;

[0113] The tracking result determination unit is configured to eliminate the tracking points with similar distances in the current image frame according to the tracking point sorting list to obtain the target feature tracking result.

[0114] In conjunction with any device embodiment provided in this application, the sorting unit includes:

[0115] A score updating subunit is configured to update the scores of the tracking points using the number of times each of the tracking points is tracked in a previous image frame if the scores of the multiple tracking points are equal;

[0116] The sorting subunit is configured to sort the multiple tracking points based on the updated tracking point scores.

[0117] According to a fourth aspect of an embodiment of the present application, a scene creation device is provided, including:

[0118] An image acquisition module is configured to acquire a sequence of image frames;

[0119] A feature tracking module is configured to track image feature points of each current image frame in the image frame sequence using any image feature tracking device provided in the present application to obtain an image feature tracking result;

[0120] An information estimation module, configured to estimate the position and posture information and three-dimensional point cloud information of the terminal device according to the image feature tracking results of the image frame sequence;

[0121] The scene creation module is configured to create an augmented reality scene based on the posture information and three-dimensional point cloud information of the terminal device.

[0122] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described above is implemented.

[0123] According to a sixth aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described methods when executing the program.

[0124] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first and second aspects above is implemented.

[0125] According to the sixth aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the first and second aspects above is implemented.

[0126] By using the image feature tracking method provided by the embodiment of the present application, in the process of tracking the image features of the current image frame, the feature point set of the current image frame can be determined based on the optical flow tracking information; then, the descriptor tracking method is used to determine the descriptor tracking points included in the above feature point set, and finally, the image feature tracking result of the current image frame is finally determined according to the tracking information corresponding to the optical flow tracking points and the descriptor tracking points respectively. In the process of determining the image feature tracking result, the embodiment of the present application effectively combines the advantages of optical flow feature tracking and descriptor feature tracking. For image sequences captured in complex scenes such as fast or intense motion scenes, low light scenes, etc., devices with limited computing power can also accurately and stably track the feature points representing the same object, thereby improving the visual tracking effect and reducing the visual tracking power consumption of the device.

[0127] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0128] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0129] Figure 1 is a flow chart of an image feature tracking method according to an exemplary embodiment of the present application;

[0130] Figure 2 is a flow chart of another image feature tracking method according to an exemplary embodiment of the present application;

[0131] Figure 3 is a schematic diagram of an application scenario of another image feature tracking method according to an exemplary embodiment of the present application;

[0132] Figure 4 is a flow chart of another image feature tracking method according to an exemplary embodiment of the present application;

[0133] Figure 5 is a flow chart of another image feature tracking method according to an exemplary embodiment of the present application;

[0134] Figure 6 is a flow chart of another image feature tracking method according to an exemplary embodiment of the present application;

[0135] Figure 7 is a flow chart of another image feature tracking method according to an exemplary embodiment of the present application;

[0136] Figure 8 is a schematic diagram of an application scenario of another image feature tracking method according to an exemplary embodiment of the present application;

[0137] Fig. 9 is a schematic diagram of an application scenario of an image feature tracking method according to an exemplary embodiment of the present application;

[0138] Fig.10 is a flow chart of another image feature tracking method according to an exemplary embodiment of the present application;

[0139] Figure 11-1 is a schematic diagram of an application scenario of an image feature tracking method according to an exemplary embodiment of the present application;

[0140] Figure 11-2is a schematic diagram of an application scenario of another image feature tracking method according to an exemplary embodiment of the present application;

[0141] Fig.12 is a flow chart of another image feature tracking method according to an exemplary embodiment of the present application;

[0142] Fig.13 is a schematic diagram of an application scenario of another image feature tracking method according to an exemplary embodiment of the present application;

[0143] Fig.14 is a flow chart of a scene creation method according to an exemplary embodiment of the present application;

[0144] Fig.15 is a block diagram of an image feature tracking device according to an exemplary embodiment of the present application;

[0145] Fig.16 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0146] Fig.17 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0147] Fig.18 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0148] Fig.19 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0149] Fig. 20 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0150] Fig.21 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0151] Fig. 22 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0152] Fig.23 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0153] Fig.24 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0154] Fig.25is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0155] Fig.26 is a block diagram of another image feature tracking device according to an exemplary embodiment of the present application;

[0156] Fig. 27 is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present application;

[0157] Fig.28 It is a schematic structural diagram of another electronic device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0158] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0159] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, which can represent any one or more elements selected from the set consisting of A, B, and C.

[0160] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0161] The present application provides an image feature tracking method, which can be applied to application scenarios such as intelligent driving, human-computer interaction, security monitoring, and augmented reality (AR).

[0162] Image feature point tracking is a basic problem in computer vision. In visual SLAM, it is necessary to first find a combination of adjacent image matching points and then calculate the camera's position and posture based on these matching point pairs, that is, the rotation and translation information of the camera relative to the initial position.

[0163] In the embodiment of the present application, the execution subjects involved include: a computer system and a camera for real-time image acquisition. The camera sends the scene image data acquired in real time to the computer system, so that the computer system processes the scene image data, determines the corresponding feature point pairs between the current image frame and the previous image frame, and calculates the position change of the camera according to the information of the corresponding feature point pairs, so that the computer system performs subsequent data processing according to the position change of the camera, such as constructing an augmented reality scene at the current moment.

[0164] The above-mentioned computer system can be a computer system in electronic devices such as a personal computer system, a vehicle-mounted device, a mobile terminal, or a computer system arranged in a server, a server cluster, or a cloud platform. The above-mentioned camera can be a camera of a terminal device, and the above-mentioned terminal device can be an electronic device such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In the specific implementation process, the camera and the computer system are independent of each other, but are interconnected at the same time, and jointly implement the image feature tracking method provided in the embodiment of the present application. Taking a computer system as an example, the image feature tracking method provided in the present application is described in detail below.

[0165] The embodiment of the present application provides a method for tracking image features by fusing an optical flow method and a descriptor method, which can be applied to a SLAM system.

[0166] Before introducing the image feature tracking method provided by the present application, the feature point matching between two frames of images, namely image feature tracking, based on the optical flow method and the descriptor method is first introduced.

[0167] Regarding the optical flow method for image feature tracking, two consecutive image frame data and the array of feature points detected in the previous frame are input into a preset optical flow tracking function, for example, the cv::calcOpticalFlowPyrLK function. The optical flow tracking function will return a set of new feature point positions, indicating the positions of the corresponding points of the feature points of the previous frame in the current image frame.

[0168] Regarding the descriptor method for image feature tracking: extract feature point information from two frames of images respectively, wherein each feature point information includes: a key point (Keypoint) and a descriptor (Descriptor); perform image feature matching based on each feature point information in the two frames of images.

[0169] Among them, the key point usually indicates the location information of the feature point in the image; the descriptor is used to describe the information of the key point and the surrounding pixels. The descriptor can eliminate the changes in the scale and direction of the image features caused by the change of the perspective, and can more accurately match the feature points between images. According to the design characteristics of the descriptor, two key points with similar appearance have similar descriptors. Therefore, through descriptor matching, it is possible to accurately determine which feature points in the two frames of images are the same point.

[0170] Based on this, see Figure 1 According to a flowchart of an image feature tracking method shown in an exemplary embodiment, the method may include:

[0171] In step 11, for the acquired image frame sequence, a feature point set of each current image frame is determined, wherein the feature point set includes: optical flow tracking points and feature extraction points;

[0172] The computer system can obtain a sequence of image frames captured by a camera in real time, and the current image frame belongs to any image in the sequence of image frames.

[0173] In the embodiment of the present application, a feature point set may be determined for each current image frame. The feature point set of the current image frame is a set consisting of multiple feature points.

[0174] The above feature points are features that can reflect the same object in the image captured when the camera moves and rotates, that is, when the viewing angle changes. That is, through these feature points, the same object in the images captured by the camera at different viewing angles can be found, and then the change in the camera's posture can be determined based on the position change of the above feature points in different images.

[0175] In the embodiment of the present application, the feature point set of the current image frame includes two types of feature points: optical flow tracking points and feature extraction points.

[0176] Among them, the above-mentioned optical flow tracking point refers to the feature point tracked in the current image frame using the optical flow method. According to relevant knowledge, optical flow describes the movement of the corresponding projection point after the three-dimensional moving point is projected onto the two-dimensional image, which can be regarded as the instantaneous displacement of the two-dimensional projection point, that is, the optical flow is the displacement of the same point in two adjacent frames. Based on this, the position of a feature point in the previous image frame is known, and the optical flow information of the feature point is used to calculate the position of the feature point in the current image frame; in the embodiment of the present application, the feature point determined in the current image frame using the optical flow method is called an optical flow tracking point.

[0177] According to the different sources of optical flow tracking information, the above optical flow tracking points can be divided into two categories:

[0178] The first type of optical flow tracking points is determined by optical flow tracking information provided externally; for example, the optical flow tracking point information comes from an optical flow tracker at the front end, and the optical flow tracker can independently perform feature point extraction and pure optical flow tracking.

[0179] The second type of optical flow tracking points is determined by a computer system extracting feature points from two adjacent image frames, i.e., the previous image frame and the current image frame, and performing optical flow tracking based on the feature points. It is suitable for situations where no optical flow tracking information is provided externally.

[0180] The above-mentioned feature extraction points refer to feature points obtained by the computer system by extracting features from the current image frame using relevant feature point extraction algorithms. Among them, the above-mentioned feature point extraction algorithms may include: SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), FAST (Features From Accelerated Segment Test) algorithm and other algorithms that can extract image feature points.

[0181] In step 12, descriptor matching is performed on each feature point in the feature point set to obtain tracking information of the descriptor tracking point;

[0182] In the embodiment of the present application, the optical flow tracking point is also attempted to be tracked twice using the descriptor method, and the feature point corresponding to the optical flow tracking point in the previous image frame is simultaneously tracked using the descriptor method, so that the feature point is tracked with higher accuracy and the probability of tracking drift is reduced.

[0183] In the present application, the tracking information of the above-mentioned descriptor tracking point includes: the position of the descriptor tracking point in the current image frame and related tracking information; wherein the above-mentioned related tracking information is used to indicate the method of determining the center of the search range during the process of determining the descriptor tracking point.

[0184] See also Figure 2 According to a flowchart of another image feature tracking method shown in an exemplary embodiment, the above step 12 may include:

[0185] In step 121, a descriptor is calculated for each of the feature points in the feature point set to obtain a descriptor of the target feature point;

[0186] In an embodiment of the present application, for each feature point in the above-mentioned feature point set, regardless of whether the feature point belongs to an optical flow tracking point or a feature extraction point, the computer system will use relevant descriptor calculation methods such as BRIEF algorithm, ORB (Oriented FAST and Rotated BRIEF) algorithm, SIFT algorithm, SURF algorithm, etc., which can calculate feature point descriptors, to calculate the descriptor for each feature point.

[0187] It should be noted here that in the process of calculating the descriptor of the feature point, the computer system will calculate the descriptor for the position of each feature point in the feature point set. Since the descriptor of the feature point is used to describe the information of the key point and its surrounding pixels, it can eliminate the changes in the scale and direction of the image caused by the change in viewing angle, so a good descriptor usually has invariance, robustness, and distinguishability. Among them, invariance means that the feature will not change with the enlargement, reduction, and rotation of the image; robustness means insensitivity to noise, lighting or other small deformations; distinguishability means that the descriptor of each feature point is unique and exclusive, and minimizes the similarity between each other. In view of the above characteristics of the descriptor, in the above process of traversing the feature points to calculate the descriptor, not every feature point can calculate the corresponding descriptor.

[0188] In the embodiment of the present application, the feature points from which the descriptors can be calculated are referred to as target feature points, wherein the target feature points may be optical flow tracking points or feature extraction points.

[0189] In addition, the algorithms mentioned in the embodiments of the present application are not limited to realizing one function. For some algorithms such as the above-mentioned SIFT algorithm and SURF algorithm, the computer system can use these algorithms to simultaneously realize two functions of feature point extraction and descriptor calculation.

[0190] In step 122, feature point matching is performed based on the descriptor of the target feature point to determine tracking information of the descriptor tracking point in the feature point set.

[0191] The descriptor tracking point of the feature point set refers to a feature point in the feature point set of the current image frame that can be tracked using the descriptor method based on the target feature point of the previous image frame.

[0192] In the process of feature point matching using the descriptor method, it is necessary to match the feature point descriptor of the previous image frame with the feature point descriptor of the current image frame.

[0193] See also Figure 3 According to a flowchart of another method for image feature tracking shown in an exemplary embodiment, the above step 122 may include:

[0194] Step 1221, for each first target feature point in the previous image frame, determining a search range center of the first target feature point in the current image frame;

[0195] The first target feature point is a feature point in the previous image frame from which the descriptor can be calculated. In the process of feature matching using the descriptor method, for feature points in the previous image frame, in order to match corresponding points in the current image frame, it is first necessary to determine a search range in the current image frame so as to perform feature matching within the search range. The key to determining the search range is to determine the position of the center of the search range.

[0196] In the embodiment of the present application, at least one of the following methods may be used to determine the search range center of a first target feature point in the current image frame:

[0197] Method 1 is applicable to application scenarios where the first target feature point has a corresponding target optical flow tracking point in the current image frame.

[0198] See also Figure 4 According to a flowchart of another method for image feature tracking shown in an exemplary embodiment, the above step 1221 may include:

[0199] In step 1201, it is determined whether the three-dimensional position of the first target feature point has been initialized;

[0200] In the embodiment of the present application, before determining the center of the search range of the first target feature point, the computer system may first determine whether the first target feature point has been initialized to a corresponding three-dimensional point. The initialization timing of the three-dimensional point is related to the strategy of the visual odometry (VO) in the SLAM system.

[0201] In step 1202, if the three-dimensional position of the first target feature point has been initialized, determining a predicted projection position of the initialized three-dimensional point in the current image frame;

[0202] In one embodiment of the present application, if the three-dimensional position of the first target feature point has been initialized, that is, a three-dimensional point has been initialized, the projection prediction position of the three-dimensional point in the current image frame can be determined.

[0203] In step 1203, a first distance between the target optical flow tracking point and the projected predicted position is determined;

[0204] In one embodiment of the present application, if there are both a target optical flow tracking point Pg and a projection prediction position Pt corresponding to a first target feature point in the current image frame, the distance between the two positions can be calculated, which can be called the first distance in the embodiment of the present application and is expressed as D1.

[0205] In step 1204, if the first distance is less than or equal to a preset first distance threshold, the target optical flow tracking point is determined as the center of the search range;

[0206] After determining the first distance D1, in one embodiment of the present application, the first distance may be compared with a preset first distance threshold, and the center of the search range of the first target feature point in the current image frame may be determined according to the comparison result.

[0207] Assuming that the first distance is a pixel distance, the preset first distance threshold may be a preset number of pixels, such as 5 pixels.

[0208] In an embodiment of the present application, if the first distance is less than or equal to a preset first distance threshold, the target optical flow tracking point Pg may be determined as the center of the search range of the first target feature point. Otherwise, the following step 1205 is executed.

[0209] In step 1205, if the first distance is greater than the first preset distance threshold, the projected predicted position is determined as the center of the search range.

[0210] The above steps 1202 to 1205 can be understood as the first implementation case of step 1221, that is, when the first target feature point has a corresponding target optical flow tracking point and the three-dimensional position of the first target feature point has been initialized, how to determine the center of the search range of the first target feature point in the current image frame to improve the accuracy of determining the center of the search range.

[0211] See also Figure 5 According to a flowchart of another method for image feature tracking shown in an exemplary embodiment, after the above step 1201, the above step 1221 may further include:

[0212] In step 1206, if the three-dimensional position of the first target feature point is not initialized, the target optical flow tracking point is determined as the center of the search range.

[0213] The embodiment of the present application can be understood as the second implementation situation of step 1221, that is, when the first target feature point has a corresponding target optical flow tracking point but the three-dimensional position is not initialized, how to determine the center of the search range of the first target feature point in the current image frame.

[0214] In an embodiment of the present application, if the first target feature point has a corresponding target optical flow tracking point in the current image frame, in the process of feature matching using the descriptor method, the above-mentioned target optical flow tracking point can be used as prior information to determine the center of the search range, thereby achieving the purpose of quickly determining the center of the search range, thereby improving the efficiency of feature matching using the descriptor method.

[0215] Method 2 is applicable to application scenarios where the first target feature point is not matched to the target optical flow tracking point in the current image frame.

[0216] See also Figure 6 According to a flowchart of another method for image feature tracking shown in an exemplary embodiment, the above step 1221 may include:

[0217] In step 1001, it is determined whether the three-dimensional position of the first target feature point has been initialized;

[0218] This step 1001 is similar to the above step 1201 and will not be repeated here.

[0219] In step 1002, if the three-dimensional position of the first target feature point has been initialized, determining a predicted projection position of the initialized three-dimensional point in the current image frame;

[0220] This step 1002 is similar to the above step 1202, and they can be referred to each other and will not be described again here.

[0221] In step 1003, the projected predicted position is determined as the center of the search range.

[0222] The above embodiment can be understood as the third implementation situation of the above step 1221, that is, when there is no corresponding target optical flow tracking point at the first target feature point, but its three-dimensional position has been initialized, the projection prediction position of the above three-dimensional position in the current image frame can be determined as the search range center of the first target feature point.

[0223] If the judgment result of step 1001 is negative, please refer to Figure 7 According to a flowchart of another method for image feature tracking shown in an exemplary embodiment, after the above step 1001, the above step 1221 may further include:

[0224] In step 1004, if the three-dimensional position of the first target feature point is not initialized, determining a homography matrix representing a mapping relationship between the current image frame and the previous image frame;

[0225] Among them, the homography matrix describes the mapping relationship between a set of two-dimensional points on one image and a set of two-dimensional points on another image. The condition for this mapping relationship to hold is that the two-dimensional points in the image correspond to the three-dimensional points in the real world and all fall on the same plane; but in fact, the objects photographed by the camera are usually not a large plane, but a real scene including many three-dimensional objects. At this time, using the homography matrix for mapping will produce certain errors. However, for an image sequence such as two adjacent frames in a continuous video, this error is relatively small, so it can be used to estimate the potential position of the matching point, that is, to predict the center of the search range of the first target feature point in the current image frame.

[0226] Regarding how to determine the homography matrix, in one embodiment of the present application, if the number of optical flow tracking points in the above-mentioned feature point set exceeds a preset number threshold, the above-mentioned homography matrix can be determined based on the pixel coordinates of the optical flow tracking point pairs in two adjacent frame images.

[0227] See also Figure 8 According to a flowchart of another method for image feature tracking shown in an exemplary embodiment, the above step 1004 may include:

[0228] In step 1014, the number of the optical flow tracking points in the feature point set is determined;

[0229] In step 1024, if the number of the optical flow tracking points is greater than a preset number threshold, the pixel coordinate information of each optical flow matching point pair is determined according to the optical flow tracking information of the optical flow tracking points;

[0230] Among them, the above-mentioned preset number threshold is the minimum number of matching point pairs required to ensure the validity of the homography matrix; if the number of optical flow tracking points is less than or equal to the above threshold, the calculated homography matrix may not accurately reflect the mapping relationship between the two frames of images.

[0231] The optical flow matching point pair is a feature point pair tracked in the previous image frame and the current image frame using an optical flow tracking method.

[0232] In step 1034, the homography matrix is ​​determined according to the pixel coordinate information of each optical flow matching point pair;

[0233] That is, the feature point pairs matched in two adjacent image frames by the optical flow tracking method are determined according to the optical flow tracking point information in the current image frame, and then a homography matrix is ​​calculated according to the pixel coordinates of all the feature point pairs.

[0234] In another embodiment of the present application, if the number of optical flow tracking points in the above-mentioned feature point set does not exceed the above-mentioned preset number threshold, such as 15 pixels; assuming that the number of optical flow tracking points in the above-mentioned feature point set is 13, correspondingly, the pixel coordinate information of 13 optical flow matching point pairs can be determined in the previous image frame and the current image frame. In this case, the homography matrix determined only based on the pixel coordinates of the 13 optical flow matching point pairs is often unreliable, that is, the homography matrix determined based on the optical flow tracking point information is usually unreliable. At this time, a terminal device such as a smart phone can use data sensed by other sensors that sense the device's posture, such as an inertial measurement unit IMU and other sensors, to fit a homography matrix.

[0235] In step 1005, the center of the search range of the first target feature point is determined according to the homography matrix.

[0236] The above embodiment can be understood as the fourth implementation situation of the above step 1221, that is, the first target feature point is not matched to the corresponding target optical flow tracking point in the current image frame, and the corresponding three-dimensional position is not initialized. The above homography matrix can be used to determine the search range center of the first target feature point.

[0237] In step 1222, a descriptor of each second target feature point within a target search range is determined, wherein the target search range is a range determined based on the center of the search range;

[0238] In the embodiment of the present application, feature points of the descriptor can be calculated in the current image frame, which are called second target feature points.

[0239] When the search range center of the first target feature point is determined, the target search range of the first target feature point can be determined in the current image frame according to a preset strategy, for example, with the search range center as the center and a preset number of pixels as the radius, so that the descriptor of the first target feature point can be subsequently matched with the descriptor of each second target feature point within the target search range.

[0240] In step 1223, feature point matching is performed based on the descriptor of the first target feature point and the descriptors of each of the second target feature points to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

[0241] In the process of feature matching by descriptor method, the feature matching is performed on the feature point descriptor. The specific matching process is: calculating the distance between the descriptor of the first target feature point and the descriptor of each second target feature point within the target search range, and obtaining multiple distance values;

[0242] Among them, the feature point descriptor is usually a vector. The distance between two feature point descriptors can reflect the similarity of the two feature points, that is, whether the two feature points represent the same object. Different distance metrics can be selected according to the type of descriptor. If it is a floating point type descriptor, its Euclidean distance can be used; for binary descriptors (BRIEF), its Hamming distance can be used, where the Hamming distance between two different binary strings refers to the number of different bits in the two binary strings.

[0243] The minimum distance value is compared with a preset distance threshold; if the above minimum distance value is less than the preset distance threshold, the second target feature point corresponding to the minimum distance value is determined as the descriptor tracking point corresponding to the above first target feature point; otherwise, it is determined that the descriptor tracking point corresponding to the above first target feature point is not matched in the current image frame.

[0244] That is to say, the descriptor matching result of the above-mentioned first target feature point includes two situations: 1. The corresponding feature point is matched in the second target feature point of the current image frame; 2. The corresponding feature point is not matched in the second target feature point of the current image frame.

[0245] For example, see Fig. 9 A schematic diagram of image feature tracking is shown according to an exemplary embodiment.

[0246] For a feature point in the previous image frame, assume that it is marked as P1i, where 1i represents the serial number of the feature point in the tth frame image, and i is a positive integer from 1 to n; for the feature point P1i, the corresponding descriptor can be calculated.

[0247] First, for the feature point P1i in the previous image frame, the center of the search range is determined in the current image frame, i.e., the t+1th frame image, which can be marked as P2j; then, the target search range is determined with the above-mentioned search range center P2j, for example, the area of ​​a circle determined by taking the feature point P2j as the center and a preset number of pixels, such as 10 pixels, as the radius; and the descriptors of P1i are matched one by one with the descriptors of the second target feature points in the above-mentioned target search range, for example, the descriptors of the 8 second target feature points, to obtain the descriptor matching results.

[0248] By analogy, the descriptor tracking points included in the feature point set of the current image frame are determined in the above manner, and the tracking information of each descriptor tracking point is determined. The tracking information of a descriptor tracking point may include: the position of the descriptor tracking point, related tracking information, etc., wherein the related tracking information is used to indicate the method for determining the center of the search range in the above tracking process using the descriptor method; it may include: optical flow method, three-dimensional projection method, homography matrix method.

[0249] At this point, the tracking information of the optical flow tracking points and the descriptor tracking points included in the feature point set of the current image frame is determined.

[0250] In the process of determining the descriptor tracking point, the optical flow tracking information can be used as prior information to quickly and accurately determine the center of the search range and reduce the search range of the descriptor tracking, thereby effectively improving the tracking effect.

[0251] In step 13, the feature tracking result of the current image frame is determined according to the tracking information of the optical flow tracking point and the descriptor tracking point.

[0252] In one embodiment of the present application, the computer system uses an optical flow method and a descriptor method to determine the tracked feature points in the feature point set of the current image frame.

[0253] See also Fig.10 According to a flowchart of another image feature tracking method shown in an exemplary embodiment, the above step 13 may include:

[0254] In step 131, an initial feature tracking result of the current image frame is determined according to the tracking information of the optical flow tracking point and the descriptor tracking point;

[0255] In an embodiment of the present application, for a first target feature point in the previous image frame, an optical flow tracking point and a descriptor tracking point may be matched in the current image frame, respectively. In view of the above situation, one of the two tracking points may be further selected as the target tracking point of the first target feature point.

[0256] See also Figure 11-1 According to a flowchart of another image feature tracking method shown in an exemplary embodiment, the above step 131 may include:

[0257] In step 1311, for the same first target feature point, a second distance between the optical flow tracking point and the descriptor tracking point is determined;

[0258] See also Figure 11-2 According to a scene diagram of another image feature tracking method shown in an exemplary embodiment, it is assumed that a first target feature point P1i in the previous image frame is matched to a corresponding optical flow tracking point P2g in the current image frame; and is also matched to a corresponding descriptor tracking point P2m. In the embodiment of the present application, the distance between the optical flow tracking point P2g and the descriptor tracking point P2m can be calculated, which can be called the second distance D2.

[0259] In step 1312, determining whether the second distance is greater than a preset second distance threshold;

[0260] In the embodiment of the present application, the second distance may be compared with a preset second distance threshold, and the target tracking point of the first target feature point may be determined based on the comparison result.

[0261] In step 1313, if the second distance is less than or equal to the preset second distance threshold, the optical flow tracking point is determined as the target tracking point of the first target feature point;

[0262] In step 1314, if the second distance is greater than the preset second distance threshold, the descriptor tracking point is determined as the target tracking point of the first target feature point.

[0263] Assume that the above-mentioned second distance D2 belongs to the pixel distance, and the above-mentioned preset second distance threshold is a preset number of pixel points, such as 3 pixel points; if D2 is less than or equal to 3, the above-mentioned optical flow tracking point P2g is determined as the target tracking point of the first target feature point P1i in the current image frame; since the accuracy of the tracking point position information determined by the optical flow tracking method can be sub-pixel level, and the accuracy of the tracking point position information determined by the descriptor method is pixel level; therefore, when the positions of two tracking points for the same first target feature point in the current image frame are very close, the optical flow tracking point can be preferentially determined as the target tracking point to improve the accuracy of the tracking point position of the above-mentioned first target feature point in the current image frame.

[0264] On the contrary, if the second distance D2 is greater than 3, the above-mentioned descriptor tracking point P2m is determined as the target tracking point of the first target feature point P1i in the current image frame. In the embodiment of the present application, when the distance between the two tracking points is far, that is, when the position deviation of the feature points tracked by the optical flow method and the descriptor method is large, since the tracking point determined by the descriptor method has higher accuracy, the descriptor tracking point is determined as the target tracking point of the first target feature point, which can avoid the possible drift of the optical flow tracking point.

[0265] By analogy, the matching point pair information between the current image frame and the previous image frame is obtained, and then the initial feature tracking result of the current image frame is determined. The above initial feature tracking result indicates which feature points in the feature point set of the current image frame belong to the tracked points, that is, the tracking points.

[0266] In step 132, feature points of the initial feature tracking result are removed according to the relevant tracking information of each tracking point to obtain a target feature tracking result.

[0267] In the embodiment of the present application, the initial feature tracking result generally includes many tracking points. For example, assuming that the feature point set of the current image frame determined in step 11 includes 100 feature points, 80 tracking points are determined in step 131. Among them, the 80 tracking points may include tracking points that are relatively close.

[0268] In view of the above situation, feature points of the above initial feature tracking results can be eliminated to obtain effective tracking points of the current image frame, so as to reduce the amount of calculation for subsequent image feature tracking.

[0269] In one embodiment of the present application, the computer system can use relevant methods such as brute-force matching, cross matching, K nearest neighbor matching method, etc. to eliminate tracking points that are relatively close in the current image frame and obtain valid tracking points in the current image frame to reduce the amount of calculation in the subsequent feature tracking process.

[0270] In another embodiment of the present application, the computer system may also determine the target tracking point of the current image frame using a multiple weighted scoring method based on the above initial feature tracking results.

[0271] See also Fig.12 According to a flowchart of another image feature tracking method shown in an exemplary embodiment, the above step 132 may include:

[0272] In step 1321, the score of each tracking point in the initial feature tracking result is determined according to the relevant tracking information of each tracking point and a preset weight scoring rule;

[0273] In the embodiment of the present application, for each target tracking point in the current image frame, different weights may be assigned according to different relevant tracking information. The relevant tracking information may include at least one of the following:

[0274] Tracking methods, including: optical flow tracking method, descriptor tracking method;

[0275] Search range determination methods include: optical flow tracking method, three-dimensional point projection method, homography matrix method;

[0276] The number of tracking times indicates the number of times the object was tracked in the previous image frame sequence.

[0277] In the present application, the above-mentioned preset weight scoring rules may include: a correspondence between relevant tracking information and preset scoring values, which is exemplarily shown in the following Table 1:

[0278] Table 1

[0279] Related tracking information Assignment Descriptor-Optical Flow Prior 5 points Descriptor-Projection 5 points Descriptor-homography matrix 3 points Optical flow method 1 point

[0280] As shown in Table 1, if the relevant tracking information of a tracking point is a descriptor-optical flow prior, it means that the tracking point is a descriptor tracking point, and the search range center determined by the optical flow tracking information may correspond to the above step 1204 or step 1206.

[0281] Similarly, if the relevant tracking information of a tracking point is a descriptor-projection, it means that the tracking point is a descriptor tracking point and the search range center is determined by using the three-dimensional point projection predicted position, which can correspond to the above step 1205 or the above step 1003.

[0282] If the relevant tracking information of a tracking point is a descriptor-homography matrix, it means that the tracking point is a descriptor tracking point and the center of the search range is determined by the homography matrix, which corresponds to the above step 1005.

[0283] If the relevant tracking information of a tracking point is the optical flow method, it means that the tracking point is an optical flow tracking point.

[0284] For a simple example, assume that the above initial tracking result includes 5 tracking points, which are represented as: P21, P22, P23, P24, P25. After scoring according to the above preset weight scoring rules based on their respective relevant tracking information, the scores obtained are: 5, 3, 1, 5, 3; the corresponding relationship between the above tracking point index and the tracking point score can be shown in Table 2:

[0285] Table 2

[0286]

[0287]

[0288] In actual application scenarios, there may be a large number of tracking points in the above initial tracking results, for example, 80. The computer system will determine the score of each tracking point according to the above preset weight scoring rule.

[0289] In step 1322, the tracking points are sorted in descending order according to the tracking point scores to obtain a tracking point sorting list, wherein the sorting list includes: a correspondence between tracking point index information and scores;

[0290] As in the above example, the obtained tracking point sorting list can be shown in Table 3:

[0291] Table 3

[0292] Tracking point index Points P21 5 points P24 5 points P22 3 points P25 3 points P23 1 point

[0293] In another embodiment of the present application, if at least two tracking points obtain the same score using the above-mentioned preset weight scoring rule, such as P21 and P24, and P22 and P25 shown in Table 3, in order to effectively distinguish their sorting information, the scores of the above-mentioned tracking points can also be updated by using the number of times each tracking point is tracked in the previous image frame, and the above-mentioned multiple tracking points can be sorted based on the above-mentioned updated tracking point scores.

[0294] For example, taking P21 and P24 as examples, assuming that P21 has been tracked 5 times in a preset number of image frames, such as 5, the score of P21 can be updated to 5.5 points; P24 has been tracked 3 times in a preset number of image frames, such as 5, the score of P24 can be updated to 5.3 points. The updated scores are used to adjust the order of P21 and P24 in the sorting list. Similarly, the order of P22 and P25 in the tracking point sorting list can be effectively distinguished according to this method.

[0295] In step 1323, tracking points with similar distances in the current image frame are eliminated according to the tracking point sorting list to obtain the target feature tracking result.

[0296] In the embodiment of the present application, the computer system may eliminate tracking points with similar distances in the current image frame according to the sorting information of the tracking point indexes in the tracking point sorting list to obtain the target feature tracking result.

[0297] Still using the above example, assuming that Table 3 is fine-tuned according to the number of tracking times, the updated tracking point sorting list is shown in Table 4:

[0298] Table 4

[0299]

[0300]

[0301] The computer system may then use a mask of a preset size in the current image frame to sequentially remove tracking points with similar distances according to the sorted index information.

[0302] See also Fig.13According to an exemplary embodiment, another application scenario diagram of an image feature tracking method is shown. Still as in the above example, in the current image frame 100, the computer system first uses a mask with a radius of r, with P21 as the center, to determine whether there are other tracking points within the mask coverage range according to the tracking point sorting list shown in Table 4 above. As shown in the figure, tracking point P22 appears within the mask coverage range of tracking point P21, indicating that P22 is close to P21, so tracking point P22 can be eliminated. Next, with tracking point P24 ranked second as the center, feature points are eliminated using the above mask, and so on, feature points are eliminated in descending order according to the index sorting, and the target tracking result of the current image frame is obtained, as shown in picture 200.

[0303] In the embodiment of the present application, after the initial feature tracking result of the current image frame is determined by fusing the optical flow tracking method and the descriptor tracking method, feature points that are close to each other can be eliminated according to the feature point sorting list, so as to extract a reasonable number of tracking feature points from the current image frame, which can not only improve the extraction accuracy of image features, but also reduce the computational complexity of subsequent image feature tracking, thereby improving feature tracking efficiency.

[0304] In summary, using the image feature tracking method provided by the embodiment of the present application, in the process of tracking the image features of the current image frame, the feature point set of the current image frame can be determined based on the optical flow tracking information; then, the descriptor tracking method is used to determine the descriptor tracking points included in the above feature point set, and finally, the image feature tracking result of the current image frame is finally determined according to the tracking information corresponding to the optical flow tracking points and the descriptor tracking points respectively. In the process of determining the image feature tracking result, the embodiment of the present application effectively combines the advantages of optical flow feature tracking and descriptor feature tracking. For image sequences captured in complex scenes such as fast or intense motion scenes, low light scenes, etc., devices with limited computing power can also accurately and stably track feature points representing the same object, thereby improving the visual tracking effect and reducing the visual tracking power consumption of the device.

[0305] Based on the image feature tracking method described in any of the above embodiments, the present application also provides a scene creation method, which can be applied to the visual SLAM system of a terminal device.

[0306] See also Fig.14 According to a flowchart of a scene creation method shown in an exemplary embodiment, the method may include:

[0307] In step 21, an image frame sequence is acquired;

[0308] In step 22, the image feature tracking method is used to track image feature points of each current image frame in the image frame sequence to obtain an image feature tracking result;

[0309] In an embodiment of the present application, the SLAM system can use any of the above-mentioned image feature tracking method embodiments to perform feature tracking on the current image frame acquired in real time to obtain an image feature tracking result.

[0310] In step 23, according to the image feature tracking result of the image frame sequence, the posture information and the three-dimensional point cloud information of the terminal device are estimated;

[0311] In one embodiment of the present application, the VIO (Visual-Inertial Odometry) method in the SLAM system can be used to estimate the real-time posture information and three-dimensional point cloud information of the terminal device through bundle adjustment (BundleAdjustment) based on the image feature tracking results of the image frame sequence and the motion or motion change data sensed by the inertial components such as IMU, magnetometer and other sensors carried by the terminal device.

[0312] In step 24, an augmented reality scene is created based on the posture information and three-dimensional point cloud information of the terminal device.

[0313] A core engine of AR applications is the three-dimensional positioning capability of terminal devices such as smartphones. Only when the computer system of the terminal device accurately estimates the six degrees of freedom information of the smartphone, namely the rotation and translation information, based on the real-time image frame sequence, and accurately estimates the position and posture of the smartphone, can the placement of virtual objects in the real scene be accurate, thereby enhancing the AR experience of the terminal device.

[0314] In AR effects, the placement of virtual objects mainly depends on the accuracy of the camera pose and the accuracy of the 3D point position output by the visual SLAM system. The camera pose is determined based on the image feature tracking results of adjacent image frames. That is, in the SLAM system, the feature points are continuously tracked through the real-time image sequence, combined with the data measured by the inertial sensor carried by the target object, and the camera pose and 3D point position are estimated through the Bundle Adjustment (BA) method. Generally, the better the feature point tracking, the more accurate the estimated 3D point position and camera pose.

[0315] The image feature tracking method provided in the embodiment of the present application also has higher tracking efficiency and accuracy for image frame sequences taken when the user carries a terminal device in rapid motion or in poor ambient light. Therefore, the visual SLAM system can output a higher-precision camera pose, and the pose output is more stable. Therefore, better AR effects can be achieved, enhancing the AR experience of the terminal device.

[0316] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0317] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, and the present disclosure is not limited by the described action sequence, because according to the present disclosure, some steps can be performed in other sequences or simultaneously. Those skilled in the art should understand that in the specific implementation of the above method, the writing sequence of each step does not mean a strict execution sequence and does not constitute any limitation on the implementation process, and the specific execution sequence of each step should be determined by its function and possible internal logic.

[0318] Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0319] Corresponding to the aforementioned application function implementation method embodiment, the present disclosure also provides an application function implementation device and a corresponding terminal embodiment.

[0320] Corresponding to the aforementioned application function implementation method embodiment, the present disclosure also provides an application function implementation device and a corresponding terminal embodiment.

[0321] See also Fig.15 A block diagram of an image feature tracking device according to an exemplary embodiment may include:

[0322] The feature point determination module 31 is configured to determine a feature point set of each current image frame for the acquired image frame sequence, wherein the feature point set includes: an optical flow tracking point;

[0323] The tracking information determination module 32 is configured to perform descriptor matching on each feature point in the feature point set to obtain tracking information of the descriptor tracking point;

[0324] The feature tracking module 33 is configured to determine the feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point.

[0325] See also Fig.16 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.15 Based on the device embodiment shown, the feature point determination module 31 may include:

[0326] The optical flow tracking submodule 311 is configured to obtain optical flow tracking information of a current image frame, and determine an optical flow tracking point in the current image frame based on the optical flow tracking information;

[0327] The feature point extraction submodule 312 is configured to extract feature points from the current image frame to obtain feature extraction points in the current image frame;

[0328] A feature point set determination submodule 313 is configured to determine a feature point set of the current image frame according to the optical flow tracking points and the feature extraction points;

[0329] The feature point set of the current image frame includes two types of feature points: the optical flow tracking points and the feature extraction points.

[0330] See also Fig.17 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.15 or Fig.16 Based on the device embodiment shown, the tracking information determination module 32 may include:

[0331] A descriptor determination submodule 321 is configured to calculate a descriptor for each of the feature points in the feature point set to obtain a descriptor for a target feature point;

[0332] The descriptor tracking submodule 322 is configured to perform feature point matching based on the descriptor of the target feature point to determine tracking information of the descriptor tracking point.

[0333] See also Fig.18 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.17 Based on the device embodiment shown, the descriptor tracking submodule 322 may include:

[0334] A search center determination unit 3221 is configured to determine, for each first target feature point in a previous image frame, a search range center of the first target feature point in the current image frame, wherein the first target feature point is a feature point having a descriptor in the previous image frame;

[0335] A descriptor determining unit 3222 is configured to determine a descriptor of each second target feature point within a target search range, wherein the target search range is a range determined based on a center of the search range;

[0336] The matching unit 3223 is configured to perform feature point matching based on the descriptor of the first target feature point and the descriptors of each of the second target feature points to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

[0337] In an apparatus embodiment of the present application, for a situation where the first target feature point has a corresponding target optical flow tracking point in the current image frame;

[0338] See also Fig.19 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.18 Based on the device embodiment shown, the search center determination unit 3221 may include:

[0339] The position initialization determination subunit 301 is configured to determine whether the three-dimensional position of the first target feature point has been initialized;

[0340] The projection position determination subunit 302 is configured to determine, when the three-dimensional position of the first target feature point has been initialized, a projection prediction position of the initialized three-dimensional point in the current image frame;

[0341] A first distance determination subunit 303 is configured to determine a first distance between the target optical flow tracking point and the projection prediction position;

[0342] A first search center determination subunit 304 is configured to determine the target optical flow tracking point as the search range center when the first distance is less than or equal to a preset first distance threshold;

[0343] The second search center determination subunit 305 is configured to determine the projection prediction position as the search range center when the first distance is greater than the first preset distance threshold.

[0344] In another device embodiment of the present application, the search center determination unit may be configured to determine the target optical flow tracking point as the center of the search range when the three-dimensional position of the first target feature point is not initialized.

[0345] In another device embodiment of the present application, for the case where the first target feature point has no corresponding target optical flow tracking point in the current image frame;

[0346] See also Fig. 20 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.18 Based on the device embodiment shown, the search center determination unit 3221 may include:

[0347] The position initialization determination subunit 301 is configured to determine whether the three-dimensional position of the first target feature point has been initialized;

[0348] The projection position determination subunit 302 is configured to determine, when the three-dimensional position of the first target feature point has been initialized, a projection prediction position of the initialized three-dimensional point in the current image frame;

[0349] The third search center determination subunit 306 is configured to determine the projection prediction position as the search range center.

[0350] See also Fig.21 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.18 Based on the device embodiment shown, the search center determination unit 3221 may include:

[0351] The position initialization determination subunit 301 is configured to determine whether the three-dimensional position of the first target feature point has been initialized;

[0352] A homography matrix determination subunit 307, configured to determine a homography matrix representing a mapping relationship between the current image frame and the previous image frame when the three-dimensional position of the first target feature point is not initialized;

[0353] The fourth search center determination subunit 308 is configured to determine the search range center of the first target feature point according to the homography matrix.

[0354] See also Fig. 22 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.21 Based on the embodiment of the device shown, the homography matrix determination subunit 307 may include:

[0355] A quantity determination unit 3071 is configured to determine the quantity of the optical flow tracking points in the feature point set;

[0356] The pixel coordinate determining unit 3072 is configured to determine the pixel coordinate information of each optical flow matching point pair according to the optical flow tracking information of the optical flow tracking points if the number of the optical flow tracking points is greater than a preset number threshold;

[0357] A matrix determination unit 3073 is configured to determine the homography matrix according to the pixel coordinate information of each of the optical flow matching point pairs;

[0358] The optical flow matching point pair is a feature point pair matched in the previous image frame and the current image frame using an optical flow tracking method.

[0359] In an embodiment of a device of the present application, the tracking information of the descriptor tracking point obtained by the tracking information determination module 32 may include: the tracking point position and related tracking information, wherein the related tracking information is used to indicate the determination method of the search range center in the process of determining the descriptor tracking point;

[0360] See also Fig.23 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.15 Based on the device embodiment shown, the feature tracking module 33 may include:

[0361] An initial tracking result determination submodule 331 is configured to determine an initial feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point;

[0362] The target tracking result determination submodule 332 is configured to remove feature points from the initial feature tracking result according to relevant tracking information of each tracking point to obtain a target feature tracking result.

[0363] In an apparatus embodiment of the present application, if a first target feature point in a previous image frame is matched to an optical flow tracking point and a descriptor tracking point in the current image frame respectively;

[0364] See also Fig.24 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.23 Based on the embodiment of the device shown, the initial tracking result determination submodule 331 may include:

[0365] The second distance determining unit 3311 is configured to determine a second distance between the optical flow tracking point and the descriptor tracking point for the same first target feature point;

[0366] The first target determining unit 3312 is configured to determine the optical flow tracking point as a target tracking point of the first target feature point when the second distance is less than or equal to a preset second distance threshold;

[0367] The second target determination unit 3313 is configured to determine the descriptor tracking point as the target tracking point of the first target feature point when the second distance is greater than the preset second distance threshold.

[0368] See also Fig.25 A block diagram of another image feature tracking device according to an exemplary embodiment is shown. Fig.23 or Fig.24 Based on the device embodiment shown, the target tracking result determination submodule 332 may include:

[0369] The score determination unit 3321 is configured to determine the score of each tracking point in the initial feature tracking result according to the relevant tracking information of each tracking point and a preset weight scoring rule;

[0370] The sorting unit 3322 is configured to sort the tracking points in descending order of the tracking point scores to obtain a tracking point sorting list, wherein the tracking point sorting list includes: a correspondence between tracking point indexes and tracking point scores;

[0371] The tracking result determination unit 3323 is configured to eliminate the tracking points with similar distances in the current image frame according to the tracking point sorting list to obtain the target feature tracking result.

[0372] In another device embodiment of the present application, the sorting unit 3322 may include:

[0373] A score updating subunit is configured to update the scores of the tracking points using the number of times each of the tracking points is tracked in a previous image frame if the scores of the multiple tracking points are equal;

[0374] The sorting subunit is configured to sort the multiple tracking points based on the updated tracking point scores.

[0375] This application also provides a scene creation device, see Fig.26 According to a block diagram of a scene creation device shown in an exemplary embodiment, the scene creation device may include:

[0376] An image acquisition module 41 is configured to acquire an image frame sequence;

[0377] The feature tracking module 42 is configured to track image feature points of each current image frame in the image frame sequence using any of the above-mentioned image feature tracking devices to obtain image feature tracking results;

[0378] An information estimation module 43, configured to estimate the position and posture information and three-dimensional point cloud information of the terminal device according to the image feature tracking result of the image frame sequence;

[0379] The scene creation module 44 is configured to create an augmented reality scene based on the posture information and three-dimensional point cloud information of the terminal device.

[0380] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0381] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0382] Corresponding to the above-mentioned image feature tracking method, the present application embodiment also provides Fig. 27 Schematic diagram of the electronic device according to an exemplary embodiment of the present application. Similarly, corresponding to the above-mentioned scene creation method, the embodiment of the present application also provides Fig.28 The schematic structural diagram of an electronic device according to an exemplary embodiment of the present application is shown in FIG. Fig. 27 or Fig.28 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an intelligent driving control device at the logical level. Of course, in addition to software implementation methods, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0383] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0384] The present specification also provides a computer-readable storage medium, which may store a computer program, which implements the present specification when the program is executed by a processor. Figures 1 to 14 The steps of the image feature tracking method or scene creation method provided in any embodiment.

[0385] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0386] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0387] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0388] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0389] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0390] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0391] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0392] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. An image feature tracking method, characterized in that: include: For the acquired image frame sequence, determining a feature point set of each current image frame, wherein the feature point set includes: an optical flow tracking point; A descriptor is calculated for each of the feature points in the feature point set to obtain a descriptor of a target feature point, and feature point matching is performed based on the descriptor of the target feature point to determine tracking information of a descriptor tracking point; the target feature point refers to a feature point in the feature point set of the current image frame for which a descriptor can be calculated; the descriptor tracking point refers to a feature point in the feature point set that is tracked using a descriptor method based on a target feature point in a previous image frame; the tracking information includes: a tracking point position and related tracking information; the related tracking information is used to indicate a method for determining a search range center during a descriptor tracking point determination process; Determining the feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point includes: Determining an initial feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point; Feature points are removed from the initial feature tracking result according to relevant tracking information of each tracking point to obtain a target feature tracking result.

2. The method according to claim 1, characterized in that: The step of determining a feature point set of each current image frame includes: Acquire optical flow tracking information of a current image frame, and determine an optical flow tracking point in the current image frame based on the optical flow tracking information; Extracting feature points from the current image frame to obtain feature extraction points in the current image frame; Determining a feature point set of the current image frame according to the optical flow tracking points and the feature extraction points; The feature point set of the current image frame includes two types of feature points: the optical flow tracking points and the feature extraction points.

3. The method according to claim 1, characterized in that The performing feature point matching based on the descriptor of the target feature point to determine the tracking information of the descriptor tracking point includes: For each first target feature point in the previous image frame, determining a search range center of the first target feature point in the current image frame, wherein the first target feature point is a feature point having a descriptor in the previous image frame; Determine a descriptor of each second target feature point within a target search range, wherein the target search range is a range determined based on the center of the search range; Feature point matching is performed based on the descriptor of the first target feature point and the descriptors of each of the second target feature points to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

4. The method according to claim 3, characterized in that The first target feature point has a corresponding target optical flow tracking point in the current image frame; The step of determining the search range center of the first target feature point in the current image frame includes: Determining whether the three-dimensional position of the first target feature point has been initialized; If the three-dimensional position of the first target feature point has been initialized, determining a predicted projection position of the initialized three-dimensional point in the current image frame; Determining a first distance between the target optical flow tracking point and the projected predicted position; If the first distance is less than or equal to a preset first distance threshold, determining the target optical flow tracking point as the center of the search range; If the first distance is greater than a first preset distance threshold, the projected predicted position is determined as the center of the search range.

5. The method according to claim 4, characterized in that The step of determining the search range center of the first target feature point in the current image frame further includes: If the three-dimensional position of the first target feature point is not initialized, the target optical flow tracking point is determined as the center of the search range.

6. The method according to claim 3, characterized in that The first target feature point has no corresponding target optical flow tracking point in the current image frame; The step of determining the search range center of the first target feature point in the current image frame includes: Determining whether the three-dimensional position of the first target feature point has been initialized; If the three-dimensional position of the first target feature point has been initialized, determining a predicted projection position of the initialized three-dimensional point in the current image frame; The projected predicted position is determined as the center of the search range.

7. The method according to claim 6, characterized in that The step of determining the search range center of the first target feature point in the current image frame further includes: If the three-dimensional position of the first target feature point is not initialized, determining a homography matrix representing a mapping relationship between the current image frame and the previous image frame; The center of the search range of the first target feature point is determined according to the homography matrix.

8. The method according to claim 7, characterized in that The determining of a homography matrix representing a mapping relationship between the current image frame and the previous image frame comprises: Determining the number of the optical flow tracking points in the feature point set; If the number of the optical flow tracking points is greater than a preset number threshold, determining pixel coordinate information of each optical flow matching point pair according to the optical flow tracking information of the optical flow tracking points; Determine the homography matrix according to the pixel coordinate information of each of the optical flow matching point pairs; The optical flow matching point pair is a feature point pair matched in the previous image frame and the current image frame using an optical flow tracking method.

9. The method according to claim 1, characterized in that: If a first target feature point in the previous image frame is matched to an optical flow tracking point and a descriptor tracking point in the current image frame respectively; The step of determining the initial feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point comprises: For the same first target feature point, determining a second distance between the optical flow tracking point and the descriptor tracking point; If the second distance is less than or equal to a preset second distance threshold, determining the optical flow tracking point as a target tracking point of the first target feature point; If the second distance is greater than the preset second distance threshold, the descriptor tracking point is determined as the target tracking point of the first target feature point.

10. The method according to claim 1 or 9, characterized in that: The step of removing feature points from the initial feature tracking result according to the relevant tracking information of each tracking point to obtain the target feature tracking result includes: Determining the score of each tracking point in the initial feature tracking result according to the relevant tracking information of each tracking point and a preset weight scoring rule; Sorting the tracking points in descending order according to the tracking point scores to obtain a tracking point sorting list, wherein the tracking point sorting list includes: a corresponding relationship between tracking point indexes and tracking point scores; According to the tracking point sorting list, tracking points with similar distances in the current image frame are eliminated to obtain the target feature tracking result.

11. The method according to claim 10, characterized in that The step of sorting the scores of the tracking points in descending order of the tracking point scores includes: If the scores of multiple tracking points are equal, the scores of the tracking points are updated using the number of times each of the tracking points is tracked in the previous image frame; The plurality of tracking points are sorted based on the updated tracking point scores.

12. A scene creation method, characterized in that: include: Get image frame sequence; Tracking image feature points of each current image frame in the image frame sequence using the method described in any one of claims 1 to 11 to obtain an image feature tracking result; estimating the position information and three-dimensional point cloud information of the terminal device according to the image feature tracking results of the image frame sequence; An augmented reality scene is created based on the posture information and three-dimensional point cloud information of the terminal device.

13. An image feature tracking device, characterized in that: include: The feature point determination module is configured to determine a feature point set of each current image frame for the acquired image frame sequence, wherein the feature point set includes: an optical flow tracking point; A tracking information determination module is configured to perform descriptor matching on each feature point in the feature point set to obtain tracking information of the descriptor tracking point; the tracking information of the descriptor tracking point includes: a tracking point position and related tracking information, and the related tracking information is used to indicate a method for determining a search range center during the descriptor tracking point determination process; A feature tracking module, configured to determine a feature tracking result of the current image frame according to tracking information of the optical flow tracking point and the descriptor tracking point; Wherein, the tracking information determination module includes: A descriptor determination submodule is configured to calculate a descriptor for each of the feature points in the feature point set to obtain a descriptor for a target feature point; the target feature point refers to a feature point in the feature point set of the current image frame for which a descriptor can be calculated; A descriptor tracking submodule is configured to perform feature point matching based on the descriptor of the target feature point to determine tracking information of the descriptor tracking point; the descriptor tracking point refers to a feature point in the feature point set that is tracked based on the target feature point in the previous image frame using a descriptor method; The feature tracking module comprises: an initial tracking result determination submodule, configured to determine an initial feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point; The target tracking result determination submodule is configured to remove feature points from the initial feature tracking result according to relevant tracking information of each tracking point to obtain a target feature tracking result.

14. The device according to claim 13, characterized in that The feature point determination module comprises: an optical flow tracking submodule, configured to obtain optical flow tracking information of a current image frame, and determine an optical flow tracking point in the current image frame based on the optical flow tracking information; A feature point extraction submodule is configured to extract feature points from the current image frame to obtain feature extraction points in the current image frame; A feature point set determination submodule, configured to determine a feature point set of the current image frame according to the optical flow tracking points and the feature extraction points; The feature point set of the current image frame includes two types of feature points: the optical flow tracking points and the feature extraction points.

15. The device according to claim 13, characterized in that The descriptor tracking submodule includes: A search center determination unit is configured to determine, for each first target feature point in a previous image frame, a search range center of the first target feature point in the current image frame, wherein the first target feature point is a feature point having a descriptor in the previous image frame; A descriptor determining unit configured to determine a descriptor of each second target feature point within a target search range, wherein the target search range is a range determined based on a center of the search range; The matching unit is configured to perform feature point matching based on the descriptor of the first target feature point and the descriptor of each of the second target feature points to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

16. The device according to claim 15, characterized in that The first target feature point has a corresponding target optical flow tracking point in the current image frame; The search center determination unit includes: a position initialization determination subunit, configured to determine whether the three-dimensional position of the first target feature point has been initialized; a projection position determination subunit, configured to determine a projection prediction position of the initialized three-dimensional point in the current image frame when the three-dimensional position of the first target feature point has been initialized; A first distance determination subunit is configured to determine a first distance between the target optical flow tracking point and the projection prediction position; A first search center determination subunit is configured to determine the target optical flow tracking point as the center of the search range when the first distance is less than or equal to a preset first distance threshold; The second search center determination subunit is configured to determine the projection prediction position as the search range center when the first distance is greater than a first preset distance threshold.

17. The device according to claim 16, characterized in that The search center determination unit is configured to determine the target optical flow tracking point as the search range center when the three-dimensional position of the first target feature point is not initialized.

18. The device according to claim 15, characterized in that The first target feature point has no corresponding target optical flow tracking point in the current image frame; The search center determination unit includes: a position initialization determination subunit, configured to determine whether the three-dimensional position of the first target feature point has been initialized; a projection position determination subunit, configured to determine a projection prediction position of the initialized three-dimensional point in the current image frame when the three-dimensional position of the first target feature point has been initialized; The third search center determination subunit is configured to determine the projection prediction position as the search range center.

19. The device according to claim 18, characterized in that The search center determination unit further includes: a homography matrix determination subunit, configured to determine a homography matrix representing a mapping relationship between the current image frame and the previous image frame when the three-dimensional position of the first target feature point is not initialized; The fourth search center determination subunit is configured to determine the search range center of the first target feature point according to the homography matrix.

20. The device according to claim 19, characterized in that The homography matrix determination subunit includes: a quantity determination unit, configured to determine the quantity of the optical flow tracking points in the feature point set; a pixel coordinate determining unit configured to determine pixel coordinate information of each optical flow matching point pair according to optical flow tracking information of the optical flow tracking points if the number of the optical flow tracking points is greater than a preset number threshold; A matrix determination unit, configured to determine the homography matrix according to the pixel coordinate information of each of the optical flow matching point pairs; The optical flow matching point pair is a feature point pair matched in the previous image frame and the current image frame using an optical flow tracking method.

21. The device according to claim 13, characterized in that If a first target feature point in the previous image frame is matched to an optical flow tracking point and a descriptor tracking point in the current image frame respectively; The initial tracking result determination submodule includes: A second distance determination unit is configured to determine a second distance between the optical flow tracking point and the descriptor tracking point for the same first target feature point; A first target determination unit is configured to determine the optical flow tracking point as a target tracking point of the first target feature point when the second distance is less than or equal to a preset second distance threshold; The second target determination unit is configured to determine the descriptor tracking point as a target tracking point of the first target feature point when the second distance is greater than the preset second distance threshold.

22. The device according to claim 13 or 21, characterized in that The target tracking result determination submodule includes: A score determination unit, configured to determine the score of each tracking point in the initial feature tracking result according to relevant tracking information of each tracking point and a preset weight scoring rule; A sorting unit is configured to sort the tracking points in descending order of the tracking point scores to obtain a tracking point sorting list, wherein the tracking point sorting list includes: a corresponding relationship between the tracking point index and the tracking point score; The tracking result determination unit is configured to eliminate the tracking points with similar distances in the current image frame according to the tracking point sorting list to obtain the target feature tracking result.

23. The device according to claim 22, characterized in that The sorting unit comprises: A score updating subunit is configured to update the scores of the tracking points using the number of times each of the tracking points is tracked in a previous image frame if the scores of the multiple tracking points are equal; The sorting subunit is configured to sort the multiple tracking points based on the updated tracking point scores.

24. A scene creation device, characterized in that: include: An image acquisition module is configured to acquire a sequence of image frames; A feature tracking module, configured to track image feature points of each current image frame in the image frame sequence using the apparatus described in any one of claims 13 to 23 to obtain an image feature tracking result; An information estimation module, configured to estimate the position and posture information and three-dimensional point cloud information of the terminal device according to the image feature tracking results of the image frame sequence; The scene creation module is configured to create an augmented reality scene based on the posture information and three-dimensional point cloud information of the terminal device.

25. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

26. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 12 is implemented.

Images