Image feature tracking methods, devices, and computer program products

Abstract

This disclosure provides an image feature tracking method, apparatus, and computer program product. The image feature tracking method includes: acquiring a current image frame and its current features, and recording the current features in a tracking list; acquiring the next image frame as a target image frame; acquiring target features of the same type as the current features from the target image frame and recording them in a target feature list; projecting the current features from the current image frame to the target image frame using an inter-frame pixel motion model to obtain a feature projection result; calculating the feature matching degree between the feature projection result and the target features recorded in the target feature list; determining whether to record the target features in the tracking list based on the feature matching degree; and outputting the tracking list based on preset conditions. The inter-frame pixel motion model enables more accurate projection of features extracted from the current image frame onto neighboring image frames, thereby significantly reducing mismatches.

Description

Technical Field

[0001] This disclosure relates to the field of visual technology, specifically to image feature tracking methods, apparatus, and computer program products. Background Technology

[0002] Cameras are crucial sensors in the autonomous driving and high-precision mapping industries. Compared to LiDAR, cameras offer advantages such as lower cost and higher resolution, making them suitable for large-scale deployment. Furthermore, image-based perception algorithms are more mature than point cloud algorithms and have already seen numerous practical applications. In high-precision map production, continuous image frames captured by cameras can be used for visual 3D reconstruction of semantic features such as signs, light poles, and lane lines. Visual 3D reconstruction algorithms are based on multi-view geometry principles, requiring the identification of corresponding features across different image frames, and then reconstructing the 3D information of the features based on the camera's position and orientation at the moment of exposure. In high-precision positioning and static obstacle recognition scenarios for autonomous vehicles, matching and tracking features from multiple image frames is also necessary to obtain continuous and stable positioning features and obstacle information. Several image feature tracking solutions already exist in related technologies.

[0003] Whether it's traditional vision algorithms or deep learning networks, the features extracted from images can generally include descriptive information, such as SIFT (Scale-Invariant Feature Transform) descriptors, BRIEF (Binary Robust Independent Elementary Features) descriptors, and even semantic information like "sign corner points" or "lamp pole centerlines." Through this descriptive information, the matching degree between any two features can be calculated.

[0004] The following is an example illustrating an implementation scheme for image feature tracking.

[0005] (1) Feature descriptor-based matching scheme

[0006] This approach extracts traditional feature points and descriptors such as SIFT, SURF (Speed-up robust features), and ORB (Oriented Fast and Rotated BRIEF) from images. Then, it determines the best matching and tracking result for features from previous image frames by calculating the matching degree of the descriptors. The advantage of this approach is that it directly measures the similarity of feature points through descriptors. However, its disadvantages include high computational cost, significant susceptibility to lighting conditions, and a lack of structural and semantic information in the extracted feature points, with many even being non-semantic points. This makes it difficult to apply to tasks requiring structural and semantic mapping and static obstacle recognition.

[0007] (2) Direct projection calculation matching degree scheme

[0008] This scheme assumes that the camera movement is relatively smooth. It uses rule-based or deep learning methods to extract semantic feature elements such as signs, light poles, lane lines, and traffic lights in each frame. The feature elements of the current image frame are directly projected into the next frame. The matching degree between similar feature elements (such as line distance, bounding box overlap, etc.) is calculated, and the feature with the highest matching degree is selected as the tracking result.

[0009] The advantages of this method are its simple logic and high algorithm efficiency. The disadvantages are that the tracking results are greatly affected by inter-frame motion (especially attitude angle): when the vehicle (camera) turns, the same feature can have significant positional differences in adjacent image frames; even when traveling straight, features near the field of view move rapidly, resulting in significant changes in position and shape in adjacent images, which can easily lead to tracking errors. Furthermore, this method has low fault tolerance. During tracking, if feature extraction fails in the current image (a common phenomenon due to factors such as lighting and occlusion), even if it can be re-extracted in subsequent frames, the tracking usually ends due to excessive inter-frame motion.

[0010] (3) Manual annotation + deep neural network learning scheme

[0011] This approach employs an end-to-end deep learning strategy. First, extensive manual annotation generates inter-frame tracking annotation data with different feature types. Then, the recognition input is replaced with consecutive video frames instead of a single image, and the tracking results are directly output during feature extraction. This end-to-end method is fast and robust with a large amount of training data. However, it requires additional data annotation work and adjustment of the perceptual model's input. Furthermore, training data for different features cannot be directly reused, resulting in weak generalization ability for new features. Summary of the Invention

[0012] To address the problems in related technologies, embodiments of this disclosure provide an image feature tracking method, apparatus, and computer program product that are applicable to structural and semantic features. The solution can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition. Through the feature frame inter-pixel motion model, the features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing mismatches. Moreover, it does not require a large amount of manual annotation or modification of the recognition model.

[0013] In a first aspect, this disclosure provides an image feature tracking method, wherein the method includes:

[0014] Obtain the current image frame and the current features of the current image frame, and record the current features in the tracking list;

[0015] Obtain the next image frame of the current image frame as the target image frame;

[0016] From the target image frame, obtain target features of the same type as the current feature and record them in the target feature list;

[0017] The current feature is projected from the current image frame to the target image frame using a feature inter-frame pixel motion model to obtain the feature projection result;

[0018] Calculate the feature matching degree between the feature projection result and the target features recorded in the target feature list;

[0019] Whether to record the target feature in the tracking list is determined based on the feature matching degree;

[0020] The tracking list is output based on preset conditions.

[0021] In conjunction with the first aspect, in a first implementation of the first aspect of this disclosure, the step of outputting the tracking list based on preset conditions includes:

[0022] For a target image frame whose target features are recorded in the tracking list, determine whether the target image frame is the last image frame. If so, output the tracking list.

[0023] In conjunction with the first aspect or the first implementation of the first aspect, in the second implementation of the first aspect of this disclosure, the step of calculating the feature matching degree between the feature projection result and the target features recorded in the target feature list includes:

[0024] Calculate the feature matching degree between the feature projection results and the target features recorded in the target feature list, and determine the target feature with the best feature matching degree as the best matching feature.

[0025] The step of determining whether to record the target feature in the tracking list based on feature matching degree includes:

[0026] The best matching degree is compared with a preset matching degree threshold. If the best matching degree is greater than or equal to the preset matching degree threshold, the best matching feature is recorded in the tracking list.

[0027] In conjunction with the second implementation of the first aspect, in the third implementation of the first aspect of this disclosure, the method further includes:

[0028] If the optimal matching degree is less than the preset matching degree threshold, then the intermittent image frame count is incremented by one;

[0029] The intermittent image frame count is compared with the intermittent image frame count threshold. If the intermittent image frame count is less than the intermittent image frame count threshold, it is determined whether the target image frame is the last image frame.

[0030] If the number of discontinuous image frames is greater than or equal to the threshold number of discontinuous image frames, then the tracking list is output.

[0031] In conjunction with the third implementation of the first aspect, the method in the fourth implementation of the first aspect further includes:

[0032] Based on the determination that the target image frame is not the last image frame, the target image frame is obtained and assigned to the current image frame, the best matching feature is assigned to the current feature, and the intermittent image frame count is set to 0.

[0033] In conjunction with the first aspect and any one of the first to fourth implementations of the first aspect, in the fifth implementation of the first aspect of this disclosure, the step of projecting the current feature from the current image frame to the target image frame using a feature inter-frame pixel motion model to obtain a feature projection result includes:

[0034] Based on the intrinsic parameter matrix of the camera that captures the image frame, or based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera that captures the image frame, the pixel offset caused by the pure rotation between the current image frame and the target image frame is calculated using the pixel coordinate information of the current image frame.

[0035] Based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame, calculate the pixel offset caused by the pure translation between the current image frame and the target image frame;

[0036] The pixel offset caused by pure rotation and the pixel offset caused by pure translation between the current image frame and the target image frame are summed to obtain the total pixel offset between the current image frame and the target image frame as the feature projection result.

[0037] In conjunction with the fifth implementation of the first aspect, in the sixth implementation of the first aspect, the step of calculating the pixel offset caused by the pure rotation between the current image frame and the target image frame based on the intrinsic parameter matrix of the camera capturing the image frame, or based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera capturing the image frame, using the pixel coordinate information of the current image frame, includes:

[0038] Based on the camera's intrinsic parameter matrix of the captured image frame, the world coordinates of the feature points of the current image frame, the camera's pose before the pure rotation, and the camera's pose after the pure rotation, calculate the pixel offset caused by the pure rotation between the current image frame and the target image frame.

[0039] Based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera that captured the image frame, the pixel offset caused by the pure rotation between the current image frame and the target image frame is calculated using the normalized phase plane coordinates of the feature pixels of the current image frame before the pure rotation.

[0040] In conjunction with the fifth implementation of the first aspect, in the seventh implementation of the first aspect of this disclosure, the step of calculating the pixel offset caused by the pure translation between the current image frame and the target image frame based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame, includes:

[0041] Based on the inter-frame interval between the current image frame and the target image frame, and the pixel offset between the current image frame and the target image frame that meets the preset interval condition, the pixel offset caused by the pure translation between the current image frame and the target image frame is calculated.

[0042] Secondly, this disclosure provides an image feature tracking device, wherein the device includes:

[0043] The current image frame acquisition module is configured to acquire the current image frame and the current features of the current image frame, and record the current features in the tracking list;

[0044] The target image frame acquisition module is configured to acquire the next image frame of the current image frame as the target image frame, and acquire target features of the same type as the current feature from the target image frame and record them in the target feature list;

[0045] The feature projection calculation module is configured to use a feature inter-frame pixel motion model to project the current feature from the current image frame to the target image frame to calculate the feature projection result.

[0046] The tracking list update module is configured to calculate the feature matching degree between the feature projection result and the target feature in the target feature list, and to determine whether to record the target feature in the tracking list based on the calculated feature matching degree.

[0047] The tracking list output module is configured to output the tracking list based on preset conditions.

[0048] Thirdly, this disclosure provides a computer program product including computer instructions that, when executed by a processor, implement the method described in the first aspect and the first to seventh implementations of the first aspect.

[0049] The technical solutions provided in this disclosure may have the following beneficial effects:

[0050] According to the technical solution provided in this disclosure, the current image frame and its current features are obtained, and the current features are recorded in a tracking list; the next image frame of the current image frame is obtained as the target image frame; from the target image frame, target features of the same type as the current features are obtained and recorded in a target feature list; the current features are projected from the current image frame to the target image frame using a feature frame inter-pixel motion model to obtain a feature projection result; the feature matching degree between the feature projection result and the target features recorded in the target feature list is calculated respectively; based on the feature matching degree, it is determined whether to record the target features in the tracking list; the tracking list is output based on preset conditions, making the solution applicable to structural and semantic features, and applicable to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition; through the feature frame inter-pixel motion model, the features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing erroneous matching; and no large amount of manual annotation or modification of the recognition model is required.

[0051] According to the technical solution provided in this disclosure, outputting the tracking list based on preset conditions includes: for a target image frame whose target features are recorded in the tracking list, determining whether the target image frame is the last image frame; if so, outputting the tracking list. This solution can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition. Through the pixel motion model between feature frames, the features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing mismatches. Moreover, it does not require a large amount of manual annotation or modification of the recognition model.

[0052] According to the technical solution provided in this disclosure, the method of calculating the feature matching degree between the feature projection result and the target features recorded in the target feature list includes: calculating the feature matching degree between the feature projection result and the target features recorded in the target feature list, and determining the target feature with the best feature matching degree as the best matching feature. The method of determining whether to record the target feature in the tracking list based on the feature matching degree includes: comparing the best matching degree with a preset matching degree threshold; if the best matching degree is greater than or equal to the preset matching degree threshold, then the best matching feature is recorded in the tracking list. This solution can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition. Through the feature frame inter-pixel motion model, features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing erroneous matching. Moreover, it eliminates the need for extensive manual annotation and modification of the recognition model.

[0053] According to the technical solution provided in this disclosure, if the optimal matching degree is less than the preset matching degree threshold, the intermittent image frame count is incremented by one; the intermittent image frame count is compared with the intermittent image frame count threshold; if the intermittent image frame count is less than the intermittent image frame count threshold, it is determined whether the target image frame is the last image frame; if the intermittent image frame count is greater than or equal to the intermittent image frame count threshold, the tracking list is output. By introducing an intermittent image frame judgment mechanism, it has good fault tolerance for local image frame segmentation failure (or error). When the number of consecutively segmented failed (or erroneous) image frames is lower than a given threshold, breakpoint continuation can be achieved, which greatly improves the robustness of the solution in this disclosure.

[0054] According to the technical solution provided in the embodiments of this disclosure, by determining that the target image frame is not the last image frame, obtaining the target image frame and assigning it to the current image frame, assigning the best matching feature to the current feature, and setting the intermittent image frame count to 0, the scheme of the embodiments of this disclosure can be executed cyclically to achieve image feature tracking.

[0055] According to the technical solution provided in this disclosure, the feature projection result is obtained by projecting the current feature from the current image frame to the target image frame using a feature inter-frame pixel motion model. This includes: calculating the pixel offset caused by pure rotation between the current image frame and the target image frame based on the intrinsic parameter matrix of the camera capturing the image frame, or based on a combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera capturing the image frame, using the pixel coordinate information of the current image frame; and calculating the pixel offset caused by pure rotation between the current image frame and the target image frame based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame. The system calculates the pixel offset caused by pure translation between the current image frame and the target image frame. It then sums the pixel offset caused by pure rotation and pure translation between the current image frame and the target image frame to obtain the total pixel offset between them, which is used as the feature projection result. This allows for the implementation of the inter-frame pixel motion model in this embodiment using existing camera pose information. By decomposing inter-frame motion into pure rotation and pure translation, and modeling the pixel offsets caused by each, it achieves inter-frame pixel motion estimation for features with unknown depth information, significantly improving the accuracy of feature tracking.

[0056] According to the technical solution provided in this disclosure, the pixel offset caused by pure rotation between the current image frame and the target image frame is calculated based on the rotation matrix between the current image frame and the target image frame and the feature pixel coordinates of the current image frame. This includes: calculating the pixel offset caused by pure rotation between the current image frame and the target image frame based on the intrinsic parameter matrix of the camera capturing the image frame, the world coordinates of the feature points of the current image frame, the pose of the camera before the pure rotation, and the pose of the camera after the pure rotation; or, based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera capturing the image frame, using the normalized phase plane coordinates of the feature pixels of the current image frame before the pure rotation, the pixel offset caused by pure rotation between the current image frame and the target image frame can be calculated using existing camera pose information. This allows for the implementation of the feature frame inter-pixel motion model in this disclosure embodiment. By decomposing inter-frame motion into pure rotation and pure translation, and modeling the pixel offset caused by each separately, the estimation of pixel motion between the same feature frames under the premise of unknown depth information is achieved, significantly improving the accuracy of feature tracking.

[0057] According to the technical solution provided in this disclosure, the pixel offset caused by pure translation between the current image frame and the target image frame is calculated based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame. This includes: calculating the pixel offset caused by pure translation between the current image frame and the target image frame based on the inter-frame interval between the current image frame and the target image frame, and the pixel offset of the current image frame and the target image frame meeting a preset interval condition. This can utilize existing camera pose information to realize the feature inter-frame pixel motion model in this disclosure. By decomposing the inter-frame motion into pure rotation and pure translation, and modeling the pixel offset caused by each, the estimation of inter-frame pixel motion of the same feature under the premise of unknown depth information is realized, which greatly improves the accuracy of feature tracking.

[0058] According to the technical solution provided in this embodiment, the current image frame acquisition module is configured to acquire the current image frame and its current features, and record the current features in a tracking list; the target image frame acquisition module is configured to acquire the next image frame of the current image frame as the target image frame; the target feature recording module is configured to acquire target features of the same type as the current features from the target image frame and record them in a target feature list; the feature projection calculation module is configured to project the current features from the current image frame to the target image frame using an inter-frame pixel motion model to obtain a feature projection result; and a feature matching degree meter is configured to... The calculation module is configured to calculate the feature matching degree between the feature projection result and the target features recorded in the target feature list; the tracking list update module is configured to determine whether to record the target feature in the tracking list based on the feature matching degree; the tracking list output module is configured to output the tracking list based on preset conditions. The solution can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition. Through the feature frame inter-pixel motion model, the features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing erroneous matching; moreover, it does not require a large amount of manual annotation or modification of the recognition model.

[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0060] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0061] Figure 1 A flowchart of an image feature tracking method according to an embodiment of the present disclosure is shown;

[0062] Figure 2 A flowchart illustrating an exemplary execution process of an image feature tracking method according to an embodiment of the present disclosure;

[0063] Figure 3 A schematic diagram showing the offset caused by pure translation when calculating the total pixel offset of features between image frames in an image feature tracking method according to an embodiment of the present disclosure;

[0064] Figure 4 A structural block diagram of an image feature tracking device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0065] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of the exemplary embodiments have been omitted from the drawings.

[0066] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of labels, numbers, steps, actions, components, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of one or more other labels, numbers, steps, actions, components, parts or combinations thereof being present or added.

[0067] It should also be noted that, unless otherwise specified, the embodiments and labels in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0068] Based on the above discussion of image feature matching and tracking schemes in related technologies, it can be seen that the feature descriptor matching schemes in related technologies have the following drawbacks: large computational load, great influence from lighting, and the extracted feature points lack structural and semantic information, and even contain a large number of non-semantic points, making them difficult to use for structural and semantic mapping and static obstacle recognition.

[0069] The direct projection matching degree calculation scheme of related technologies has drawbacks: the tracking results are greatly affected by inter-frame motion (especially attitude angles). When the vehicle (camera) turns, the same feature has significant positional differences in adjacent image frames; even when traveling straight, features near the field of view move rapidly, resulting in significant changes in position and shape in adjacent images, which can easily lead to erroneous tracking. Furthermore, this method has low fault tolerance. During tracking, if feature extraction fails in the current image (a common phenomenon due to factors such as lighting and occlusion), even if it can be re-extracted in subsequent frames, the tracking usually ends due to excessive inter-frame motion.

[0070] The manual annotation + deep neural network learning approach has drawbacks: it requires additional data annotation work, adjustments to the input of the perception model, and training data for different features cannot be directly reused, resulting in weak generalization ability for new features.

[0071] In view of this, the present disclosure provides an image feature tracking method and apparatus by inputting a specified image frame and a specified feature to be tracked on the image frame, thereby outputting a tracking list.

[0072] According to the technical solution provided in this disclosure, the current image frame and its current features are obtained, and the current features are recorded in a tracking list; the next image frame of the current image frame is obtained as the target image frame; from the target image frame, target features of the same type as the current features are obtained and recorded in a target feature list; the current features are projected from the current image frame to the target image frame using an inter-frame pixel motion model to obtain a feature projection result; the feature matching degree between the feature projection result and the target features recorded in the target feature list is calculated respectively; based on the feature matching degree, it is determined whether to record the target features in the tracking list; the tracking list is output based on preset conditions, so that the solution can be applied to structural and semantic features, and can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition; through the inter-frame pixel motion model, the features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing erroneous matching; and no large amount of manual annotation or modification of the recognition model is required.

[0073] Figure 1 A flowchart illustrating an image feature tracking method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the image feature tracking method includes steps S101, S102, S103, S104, S105, S106, and S107.

[0074] In step S101, the current image frame and the current features of the current image frame are obtained, and the current features are recorded in the tracking list.

[0075] In step S102, the next image frame of the current image frame is obtained as the target image frame.

[0076] In step S103, target features of the same type as the current feature are obtained from the target image frame and recorded in the target feature list.

[0077] In step S104, the current feature is projected from the current image frame to the target image frame using the feature inter-frame pixel motion model to obtain the feature projection result.

[0078] In step S105, the feature matching degree between the feature projection result and the target features recorded in the target feature list is calculated respectively.

[0079] In step S106, it is determined whether to record the target feature into the tracking list based on the feature matching degree.

[0080] In step S107, the tracking list is output based on preset conditions.

[0081] In one embodiment of this disclosure, the current image frame and the next image frame can be adjacent image frames in an image stream captured by a camera. An image stream refers to multiple images captured consecutively by the same camera; an image is also called a frame or an image frame. In embodiments of this disclosure, an image frame can also be simply referred to as a frame. In one embodiment of this disclosure, the next image frame may not be the next image frame immediately adjacent to the current image frame, but rather an image frame with a gap between it and the current image frame. That is, if the obtained next image frame of the current image frame is used as the target image frame, and the calculated feature projection result matches the feature of the target feature list less than a preset matching threshold, then the target image frame will be used as a gap frame, and the next image frame of the target image frame can be used as the next image frame of the current image frame, thereby re-executing the image feature tracking method of this disclosure embodiment.

[0082] In one embodiment of this disclosure, a feature refers to a relatively stable geometric (structural) object or semantic object, such as a point, line, or surface, extracted from each image frame by a certain means (traditional visual algorithms or deep learning).

[0083] In one embodiment of this disclosure, tracking refers to the process of identifying features with the same name in two or more neighboring frames of images. For example, in scenarios such as high-precision positioning and static obstacle recognition of intelligent driving vehicles, it is necessary to match and track features from multiple frames of images in order to obtain continuous and stable positioning features and obstacle information.

[0084] In one embodiment of this disclosure, the current features of the current image frame and the target features of the target image frame can be structural features and / or semantic features. Therefore, the solution of this disclosure can be used for structural and semantic mapping and static obstacle recognition.

[0085] In one embodiment of this disclosure, the tracking list records one or more features used for tracking. During tracking, the feature with the best matching degree can be recorded in the tracking list. When tracking ends, the tracking list is output as the image feature tracking result.

[0086] In one embodiment of this disclosure, a feature-frame inter-pixel motion model is proposed. Given the camera pose, features extracted from an image frame can be projected onto a neighboring target frame. The matching degree between the extracted features and all similar features on the target frame is then calculated. The feature with the highest matching degree exceeding a matching degree threshold is selected to achieve feature tracking between two frames. This feature-frame inter-pixel motion model enables more accurate projection of features extracted from the current image frame onto neighboring image frames, significantly reducing false matching and improving the effectiveness of image feature tracking.

[0087] Moreover, the image feature tracking scheme of this disclosure does not require a large amount of manual annotation or modification of the recognition model, thus saving a significant amount of manpower.

[0088] Furthermore, in one embodiment of this disclosure, a fault-tolerant mechanism is added to address the problem of tracking breaks easily caused by local image frame segmentation failure. By introducing intermittent frames, a certain degree of breakpoint connection can be achieved.

[0089] The following is for reference Figure 2 This describes a specific implementation of an image feature tracking method.

[0090] Figure 2 A flowchart illustrating an exemplary execution process of an image feature tracking method according to an embodiment of the present disclosure is provided.

[0091] like Figure 2 As shown, at the start of the image feature tracking method, a specified frame is input, and a specified feature of the specified frame is obtained. Next, initialization is performed, that is, the specified frame is assigned to the current frame, the specified feature is assigned to the current feature, and the current feature is added to the tracking list, and the number of discontinuous frames is set to zero (step indicated by marker 2010). In one embodiment of this disclosure, the next image frame of the current image frame may not be the next image frame immediately adjacent to the current image frame, but rather an image frame with a discontinuity between it and the current image frame, i.e., a discontinuous frame. This embodiment of the disclosure introduces a discontinuous frame count, which can achieve breakpoint connection when the number of frames in which the best matching feature extraction fails is lower than the discontinuous frame threshold, greatly improving the robustness of the image feature tracking scheme of this embodiment. Step 2010 can obtain the current frame 2021 and the current feature 2022 of the current frame. When the image feature tracking method of this embodiment of the disclosure begins to execute cyclically, it can be considered that the current frame 2021 and the current feature 2022 of the current frame are obtained cyclically in step 2020.

[0092] Next, the next frame after the current frame 2021, i.e., the target frame 2031, is obtained. Features of the same type as the current feature 2022 can be obtained from the target frame 2031 and recorded in the target feature list 2041. Moreover, through the inter-frame pixel motion model, the current feature 2022 is projected from the current frame 2021 onto the target frame 2031 to obtain the feature projection result 2042. Subsequently, the feature matching degree between the feature projection result 2042 and the target feature list 2041 is calculated, that is, the feature matching degree between the feature projection result 2042 and the target features recorded in the target feature list 2041 is calculated (step 2050) to find the target feature with the best matching degree with the feature projection result 2042 as the best matching feature.

[0093] After step 2050, the relationship between the optimal matching degree and the matching degree threshold is determined, that is, whether the optimal matching degree is greater than or equal to the matching degree threshold (step 2060).

[0094] If the best matching degree is greater than or equal to the matching degree threshold (the result of step 2060 is yes), that is, the best matching degree of the best matching feature is greater than or equal to the matching degree threshold, then the best matching feature is added to the tracking list (step 2070).

[0095] If the best matching degree is less than the matching degree threshold (the result of step 2060 is negative), that is, the best matching degree of the best matching feature is less than the matching degree threshold, then the interrupted frame count is incremented by 1 (step 2080). After step 2080, the relationship between the number of interrupted frames and the interrupted frame count threshold is determined, that is, whether the number of interrupted frames is greater than or equal to (>=) the interrupted frame count threshold (step 2090).

[0096] If the number of interrupted frames is greater than or equal to the interrupted frame count threshold (if the result of step 2090 is yes), the image feature tracking process ends and a tracking list is output. For example, if the interrupted frame count threshold is 5, then if the number of interrupted frames is 5 or greater than 5, the image feature tracking process ends and a tracking list is output.

[0097] If the number of interrupted frames is less than the interrupted frame count threshold (the result of step 2090 is no), then it is determined whether the target frame 2031 is the last frame (step 2100). For example, if the interrupted frame count threshold is 5, then if the number of interrupted frames is 4 or less than 4, then it is determined whether the target frame 2031 is the last frame.

[0098] Additionally, after adding the best matching feature to the tracking list in step 2070, it is determined whether the target frame 2031 is the last frame (step 2100).

[0099] If it is determined that target frame 2031 is the last frame (the result of step 2100 is yes), the image feature tracking process ends and the tracking list is output.

[0100] If it is determined that the target frame 2031 is not the last frame (the result of step 2100 is no), then the target frame is assigned to the current frame, the best matching feature is assigned to the current feature, and the number of discontinuous frames is reset to zero (step 2110). Then, the process jumps to step 2020 and starts the loop, that is, the current frame 2021 and the current feature 2022 of the current frame are obtained.

[0101] In one embodiment of this disclosure, step S107 includes:

[0102] For a target image frame whose target features are recorded in the tracking list, determine whether the target image frame is the last image frame. If so, output the tracking list.

[0103] According to the technical solution provided in this disclosure, outputting the tracking list based on preset conditions includes: for a target image frame whose target features are recorded in the tracking list, determining whether the target image frame is the last image frame; if so, outputting the tracking list. This solution can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition. Through the pixel motion model between feature frames, the features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing mismatches. Moreover, it does not require a large amount of manual annotation or modification of the recognition model.

[0104] In one embodiment of this disclosure, step S105 includes:

[0105] Calculate the feature matching degree between the feature projection results and the target features recorded in the target feature list, and determine the target feature with the best feature matching degree as the best matching feature.

[0106] The step of determining whether to record the target feature in the tracking list based on feature matching degree includes:

[0107] The best matching degree is compared with a preset matching degree threshold. If the best matching degree is greater than or equal to the preset matching degree threshold, the best matching feature is recorded in the tracking list.

[0108] According to the technical solution provided in this disclosure, the method of calculating the feature matching degree between the feature projection result and the target features recorded in the target feature list includes: calculating the feature matching degree between the feature projection result and the target features recorded in the target feature list, and determining the target feature with the best feature matching degree as the best matching feature. The method of determining whether to record the target feature in the tracking list based on the feature matching degree includes: comparing the best matching degree with a preset matching degree threshold; if the best matching degree is greater than or equal to the preset matching degree threshold, then the best matching feature is recorded in the tracking list. This solution can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition. Through the feature frame inter-pixel motion model, features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing erroneous matching. Moreover, it eliminates the need for extensive manual annotation and modification of the recognition model.

[0109] In one embodiment of this disclosure, the image feature tracking method further includes:

[0110] If the optimal matching degree is less than the preset matching degree threshold, then the intermittent image frame count is incremented by one;

[0111] The intermittent image frame count is compared with the intermittent image frame count threshold. If the intermittent image frame count is less than the intermittent image frame count threshold, it is determined whether the target image frame is the last image frame.

[0112] If the number of discontinuous image frames is greater than or equal to the threshold number of discontinuous image frames, then the tracking list is output.

[0113] In this embodiment, the target image frame is an intermittent frame, so the intermittent image frame count is incremented by one.

[0114] According to the technical solution provided in this disclosure, if the optimal matching degree is less than the preset matching degree threshold, the intermittent image frame count is incremented by one; the intermittent image frame count is compared with the intermittent image frame count threshold; if the intermittent image frame count is less than the intermittent image frame count threshold, it is determined whether the target image frame is the last image frame; if the intermittent image frame count is greater than or equal to the intermittent image frame count threshold, the tracking list is output. By introducing an intermittent image frame judgment mechanism, it has good fault tolerance for local image frame segmentation failure (or error). When the number of consecutively segmented failed (or erroneous) image frames is lower than a given threshold, breakpoint continuation can be achieved, which greatly improves the robustness of the solution in this disclosure.

[0115] In one embodiment of this disclosure, the image feature tracking method further includes:

[0116] Based on the determination that the target image frame is not the last image frame, the target image frame is obtained and assigned to the current image frame, the best matching feature is assigned to the current feature, and the intermittent image frame count is set to 0.

[0117] In this embodiment, the target image frame is not a discontinuous frame, so the discontinuous image frame count is set to 0.

[0118] According to the technical solution provided in the embodiments of this disclosure, by determining that the target image frame is not the last image frame, obtaining the target image frame and assigning it to the current image frame, assigning the best matching feature to the current feature, and setting the intermittent image frame count to 0, the scheme of the embodiments of this disclosure can be executed cyclically to achieve image feature tracking.

[0119] In one embodiment of this disclosure, step S104 includes:

[0120] Based on the intrinsic parameter matrix of the camera that captures the image frame, or based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera that captures the image frame, the pixel offset caused by the pure rotation between the current image frame and the target image frame is calculated using the pixel coordinate information of the current image frame.

[0121] Based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame, calculate the pixel offset caused by the pure translation between the current image frame and the target image frame;

[0122] The pixel offset caused by pure rotation and the pixel offset caused by pure translation between the current image frame and the target image frame are summed to obtain the total pixel offset between the current image frame and the target image frame as the feature projection result.

[0123] According to the technical solution provided in this disclosure, the feature projection result is obtained by projecting the current feature from the current image frame to the target image frame using a feature inter-frame pixel motion model. This includes: calculating the pixel offset caused by pure rotation between the current image frame and the target image frame based on the intrinsic parameter matrix of the camera capturing the image frame, or based on a combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera capturing the image frame, using the pixel coordinate information of the current image frame; and calculating the pixel offset caused by pure rotation between the current image frame and the target image frame based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame. The system calculates the pixel offset caused by pure translation between the current image frame and the target image frame. It then sums the pixel offset caused by pure rotation and pure translation between the current image frame and the target image frame to obtain the total pixel offset between them, which is used as the feature projection result. This allows for the implementation of the inter-frame pixel motion model in this embodiment using existing camera pose information. By decomposing inter-frame motion into pure rotation and pure translation, and modeling the pixel offsets caused by each, it achieves inter-frame pixel motion estimation for features with unknown depth information, significantly improving the accuracy of feature tracking.

[0124] In one embodiment of this disclosure, it is necessary to project the features of the current image frame onto the target image frame to calculate the feature matching degree between the feature projection result of the current feature and the target feature. However, since the image frame lacks depth information, even if the poses (positions and orientations) of the two image frames are known, it is impossible to directly calculate the corresponding pixels of the feature pixels of one image frame on the other image frame. In response, the solution of this disclosure proposes a feature frame pixel motion model (also known as the feature pixel frame motion model), which decomposes the inter-frame motion of the feature pixels into two parts: pure translation and pure rotation, and calculates the pixel offset caused by each part respectively; then, the two are summed to obtain the total offset, as shown in equation (1).

[0125] diff(R,t,p)=f(t,p)+f(R,p) (1)

[0126] In equation (1), R represents the rotation matrix between the current image frame and the target image frame, t is the inter-frame interval (or translation vector) between the current image frame and the target image frame, p is the feature pixel coordinates on the current image frame, diff(R,t,p) is the total pixel offset of the feature between the two image frames, f(t,p) represents the pixel offset caused by pure position translation between the two image frames, and f(R,p) represents the pixel offset caused by pure rotation between the two image frames. For features containing multiple shape points (such as line features, polygon features, etc.), each shape point can be projected according to equation (1) to obtain the projected feature, i.e., the feature projection result is obtained.

[0127] In one embodiment of this disclosure, the pixel coordinate information of an image frame may include various information related to the pixel coordinates of the image frame, such as the feature pixel coordinates, pixel offset, and normalized phase plane coordinates of the feature pixels.

[0128] In one embodiment of this disclosure, the calculation of the pixel offset caused by pure rotation between the current image frame and the target image frame, based on the intrinsic parameter matrix of the camera capturing the image frame, or based on a combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera capturing the image frame, using the pixel coordinate information of the current image frame, includes:

[0129] Based on the camera's intrinsic parameter matrix of the captured image frame, the world coordinates of the feature points of the current image frame, the camera's pose before the pure rotation, and the camera's pose after the pure rotation, calculate the pixel offset caused by the pure rotation between the current image frame and the target image frame.

[0130] Based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera that captured the image frame, the pixel offset caused by the pure rotation between the current image frame and the target image frame is calculated using the normalized phase plane coordinates of the feature pixels of the current image frame before the pure rotation.

[0131] According to the technical solution provided in the embodiments of this disclosure, the pixel offset caused by the pure rotation between the current image frame and the target image frame is calculated using the pixel coordinate information of the current image frame, based on the intrinsic parameter matrix of the camera capturing the image frame, or based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera capturing the image frame. This includes: calculating the pixel offset caused by the pure rotation between the current image frame and the target image frame based on the intrinsic parameter matrix of the camera capturing the image frame, the world coordinates of the feature points of the current image frame, the pose of the camera before the pure rotation, and the pose of the camera after the pure rotation. The pixel offset caused by the pure rotation between the current image frame and the target image frame is calculated by combining the rotation matrix between the current image frame and the intrinsic parameter matrix of the camera that captured the image frame, using the normalized phase plane coordinates of the feature pixels of the current image frame before the pure rotation. This can be achieved by using existing camera pose information to realize the feature frame pixel motion model in this embodiment. By decomposing the inter-frame motion into pure rotation and pure translation, and modeling the pixel offset caused by each, the pixel motion estimation of the same feature frame under the premise of unknown depth information is realized, which greatly improves the accuracy of feature tracking.

[0132] In one embodiment of this disclosure, if camera translation is not considered—that is, the camera capturing the image does not translate but only rotates around the optical center—the pixel coordinates of the feature point before rotation can be directly calculated if the pixel coordinates after rotation are known. The pixel coordinate offset is:

[0133]

[0134] In equation (2), K is the camera intrinsic parameter matrix for capturing image frames, and P w The world coordinates of the feature points of the current image frame. This is the camera's pose before rotation. Let R represent the pose of the camera after rotation, R represent the rotation matrix between the current image frame and the target image frame, and p represent the feature pixel coordinates on the current image frame before rotation. Let be the normalized phase plane coordinates of the feature pixel p before rotation.

[0135] In one embodiment of this disclosure, the normalized phase plane refers to a plane located directly in front of the camera, parallel to the imaging plane, and at a preset distance (e.g., 1 meter) from the optical center.

[0136] In one embodiment of this disclosure, calculating the pixel offset caused by a pure translation between the current image frame and the target image frame based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame, includes:

[0137] Based on the inter-frame interval between the current image frame and the target image frame, and the pixel offset between the current image frame and the target image frame that meets the preset interval condition, the pixel offset caused by the pure translation between the current image frame and the target image frame is calculated.

[0138] According to the technical solution provided in this disclosure, the pixel offset caused by pure translation between the current image frame and the target image frame is calculated based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame. This includes: calculating the pixel offset caused by pure translation between the current image frame and the target image frame based on the inter-frame interval between the current image frame and the target image frame, and the pixel offset of the current image frame and the target image frame meeting a preset interval condition. This can utilize existing camera pose information to realize the feature inter-frame pixel motion model in this disclosure. By decomposing the inter-frame motion into pure rotation and pure translation, and modeling the pixel offset caused by each, the estimation of inter-frame pixel motion of the same feature under the premise of unknown depth information is realized, which greatly improves the accuracy of feature tracking.

[0139] In one embodiment of this disclosure, the camera will move along a straight line if rotation is not considered. References are made below. Figure 3 Describe the case where the camera performs a pure translation. Figure 3 This diagram illustrates the offset caused by pure translation when calculating the total pixel offset of a feature between image frames in an image feature tracking method according to an embodiment of the present disclosure.

[0140] To illustrate, such as Figure 3 As shown, the camera moves forward and captures three image frames at three equidistant positions t0, t1, and t2. The normalized phase plane coordinates of the feature point P in the real world in the three image frames are p0, p1, and p2. According to the definition of the normalized phase plane, the coordinate values ​​of p0, p1, and p2 are the tangent values ​​of the angle between the observation line of sight and the direction of motion, as shown in equation (3):

[0141]

[0142] Equation (3) yields the following equation (4):

[0143]

[0144] When considering a vehicle-mounted camera, the observation distance of a vehicle-mounted camera for features such as signs, light poles, traffic lights, and lane lines is generally quite far (mostly exceeding 10 meters, and tens of meters are also common). When the distance between two adjacent positions t0, t1, and t2 is d = 1 meter, the following formula (5) can be obtained:

[0145]

[0146] Equation (5) shows that when the interval between image frames is 1 meter, the motion of the same feature on the normalized phase plane of adjacent image frames is approximately an arithmetic sequence. Since the normalized phase plane coordinates and pixel coordinates differ by only one scale factor, the pixel coordinate motion of the feature can also be approximated as an arithmetic sequence. Therefore, the pixel offset caused by pure translation is modeled as Equation (6):

[0147]

[0148] In equation (6), ||t|| represents the interval between image frames. This refers to a pixel offset of 1 meter, which can be understood as the pixel movement speed over distance. Initialize to After each successful match, update as shown in equation (7):

[0149]

[0150] In equation (7), i represents the i-th match.

[0151] The aforementioned inter-frame pixel motion model requires knowledge of the camera's pose when capturing the image. Since high-precision map acquisition equipment and (autonomous driving) vehicles are equipped with position sensors (e.g., satellite positioning systems and inertial measurement units), the camera's pose can be obtained based on the camera's installation parameters. Therefore, the preconditions of the solutions in this disclosure embodiment can be met.

[0152] It should be understood that the above discussion of pixel offset caused by pure translation considering an interval of 1 meter between image frames is merely an example, and this disclosure is not limited thereto. According to the teachings of the embodiments of this disclosure, the interval between image frames can be other values, as long as they meet the calculation requirements of the feature frame pixel motion model.

[0153] The solution of this disclosure proposes an inter-frame pixel motion model by utilizing existing camera pose information. By decomposing inter-frame motion into pure rotation and pure translation, and modeling the pixel offset caused by each, it realizes inter-frame pixel motion estimation of the same feature under the premise of unknown depth information, which greatly improves the accuracy of feature tracking.

[0154] In one embodiment of this disclosure, it is necessary to calculate the feature matching degree between the feature projection result and the target features in the target feature list. The following describes the basis for calculating the feature matching degree based on the different structures of the features.

[0155] For point features, the feature matching degree can be calculated using the planar distance between two points. Of course, other metrics can also be used to calculate the feature matching degree for point features.

[0156] For linear features, the feature matching degree can be calculated using the average distance between lines, i.e., the average projected distance from a sampling point on one line to another. Of course, other metrics can also be used to calculate the feature matching degree for linear features.

[0157] For areal features, the feature matching degree can be calculated by combining the area difference and the area intersection-union ratio (i.e., the ratio of the area of ​​the intersection of two feature faces to the area of ​​the union). Of course, other indicators can also be used to calculate the feature matching degree for areal features.

[0158] As can be seen from the above description, compared with the feature descriptor matching-based schemes of related technologies, the scheme of this disclosure is applicable to structural and semantic features and can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning and static obstacle recognition.

[0159] Compared to the direct projection calculation matching degree scheme of related technologies, the scheme of this disclosure proposes a feature frame inter-pixel motion model, which can more accurately project the features extracted from the current frame onto neighboring frames, thereby greatly reducing false matching.

[0160] Compared to related technologies that combine manual annotation with deep neural network learning, the solution in this disclosure requires no extensive manual annotation and no modification to the recognition model. Furthermore, it incorporates a fault-tolerance mechanism to address the issue of tracking breaks caused by local frame segmentation failures. By introducing intermittent frames, a certain degree of breakpoint reconnection can be achieved. This solution introduces an intermittent frame judgment mechanism, which provides good fault tolerance for local communication frame segmentation failures (or errors). When the number of consecutive failed (or erroneous) segments falls below a given threshold, breakpoint reconnection can be achieved, significantly improving the robustness of the method.

[0161] The following is for reference Figure 4 This invention describes an image feature tracking apparatus according to an embodiment of the present disclosure. Figure 4 A structural block diagram of an image feature tracking device 400 according to an embodiment of the present disclosure is shown. Figure 4 As shown, the image feature tracking device 400 includes:

[0162] The current image frame acquisition module 401 is configured to acquire the current image frame and the current features of the current image frame, and record the current features in the tracking list;

[0163] The target image frame acquisition module 402 is configured to acquire the next image frame of the current image frame as the target image frame;

[0164] The target feature recording module 403 is configured to obtain target features of the same type as the current feature from the target image frame and record them in the target feature list;

[0165] The feature projection calculation module 404 is configured to project the current feature from the current image frame to the target image frame using a feature inter-frame pixel motion model to obtain the feature projection result;

[0166] The feature matching degree calculation module 405 is configured to calculate the feature matching degree between the feature projection result and the target features recorded in the target feature list, respectively.

[0167] The tracking list update module 406 is configured to determine whether to record the target feature into the tracking list based on the feature matching degree;

[0168] The tracking list output module 407 is configured to output the tracking list based on preset conditions.

[0169] According to the technical solution provided in this embodiment, the current image frame acquisition module is configured to acquire the current image frame and its current features, and record the current features in a tracking list; the target image frame acquisition module is configured to acquire the next image frame of the current image frame as the target image frame; the target feature recording module is configured to acquire target features of the same type as the current features from the target image frame and record them in a target feature list; the feature projection calculation module is configured to project the current features from the current image frame to the target image frame using an inter-frame pixel motion model to obtain a feature projection result; and a feature matching degree meter is configured to... The calculation module is configured to calculate the feature matching degree between the feature projection result and the target features recorded in the target feature list; the tracking list update module is configured to determine whether to record the target feature in the tracking list based on the feature matching degree; the tracking list output module is configured to output the tracking list based on preset conditions. The solution can be applied to common autonomous driving scenarios such as high-precision map building, high-precision positioning, and static obstacle recognition. Through the feature frame inter-pixel motion model, the features extracted from the current image frame can be projected more accurately onto neighboring image frames, thereby greatly reducing erroneous matching; moreover, it does not require a large amount of manual annotation or modification of the recognition model.

[0170] Those skilled in the art will understand that, with reference to Figure 4 The described technical solution can be compared with the reference. Figures 1 to 3 The described embodiments are combined to provide reference. Figures 1 to 3 The technical effects achieved by the described embodiments are explained above. For details, please refer to the above description. Figures 1 to 3 The specific details of the description will not be repeated here.

[0171] In particular, according to embodiments of this disclosure, the methods described above with reference to the accompanying drawings can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising computer instructions that, when executed by a processor, implement the methods as shown in the drawings. The computer program product may include a computer program tangibly contained on a readable medium thereof, the computer program containing program code for performing the methods in the drawings. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. For example, embodiments of this disclosure include a readable storage medium storing computer instructions that, when executed by a processor, implement program code for performing the methods in the drawings.

[0172] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0173] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0174] In another aspect, this disclosure also provides a computer-readable storage medium, which may be the computer-readable storage medium included in the nodes described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this disclosure.

[0175] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. An image feature tracking method, wherein, The method includes: Obtain the current image frame and the current features of the current image frame, and record the current features in the tracking list; Obtain the next image frame of the current image frame as the target image frame; From the target image frame, obtain target features of the same type as the current feature and record them in the target feature list; The current feature is projected from the current image frame to the target image frame using a feature inter-frame pixel motion model to obtain a feature projection result. The feature inter-frame pixel motion model decomposes the inter-frame motion of the feature pixel into two parts: pure translation and pure rotation. The pixel offset caused by each part is calculated separately, and the pixel offsets are summed to obtain the total offset as the feature projection result. Calculate the feature matching degree between the feature projection result and the target features recorded in the target feature list; Whether to record the target feature in the tracking list is determined based on the feature matching degree; The tracking list is output based on preset conditions.

2. The method according to claim 1, wherein, The step of outputting the tracking list based on preset conditions includes: For a target image frame whose target features are recorded in the tracking list, determine whether the target image frame is the last image frame. If so, output the tracking list.

3. The method according to claim 2, wherein, The step of calculating the feature matching degree between the feature projection result and the target features recorded in the target feature list includes: Calculate the feature matching degree between the feature projection results and the target features recorded in the target feature list, and determine the target feature with the best feature matching degree as the best matching feature. The step of determining whether to record the target feature in the tracking list based on feature matching degree includes: The optimal feature matching degree is compared with a preset matching degree threshold. If the optimal feature matching degree is greater than or equal to the preset matching degree threshold, the optimal matching feature is recorded in the tracking list.

4. The method according to claim 3, wherein, The method further includes: If the optimal feature matching degree is less than the preset matching degree threshold, then the intermittent image frame count is incremented by one; The intermittent image frame count is compared with the intermittent image frame count threshold. If the intermittent image frame count is less than the intermittent image frame count threshold, it is determined whether the target image frame is the last image frame. If the number of discontinuous image frames is greater than or equal to the threshold number of discontinuous image frames, then the tracking list is output.

5. The method according to claim 4, wherein, The method further includes: Based on the determination that the target image frame is not the last image frame, the target image frame is obtained and assigned to the current image frame, the best matching feature is assigned to the current feature, and the intermittent image frame count is set to 0.

6. The method according to any one of claims 1-5, wherein, The step of projecting the current feature from the current image frame to the target image frame using a feature inter-frame pixel motion model to obtain the feature projection result includes: Based on the intrinsic parameter matrix of the camera that captures the image frame, or based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera that captures the image frame, the pixel offset caused by the pure rotation between the current image frame and the target image frame is calculated using the pixel coordinate information of the current image frame. Based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame, calculate the pixel offset caused by the pure translation between the current image frame and the target image frame; The pixel offset caused by pure rotation and the pixel offset caused by pure translation between the current image frame and the target image frame are summed to obtain the total pixel offset between the current image frame and the target image frame as the feature projection result.

7. The method according to claim 6, wherein, The calculation of the pixel offset caused by pure rotation between the current image frame and the target image frame, based on the intrinsic parameter matrix of the camera capturing the image frame, or based on a combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera capturing the image frame, using the pixel coordinate information of the current image frame, includes: Based on the camera's intrinsic parameter matrix of the captured image frame, the world coordinates of the feature points of the current image frame, the camera's pose before the pure rotation, and the camera's pose after the pure rotation, calculate the pixel offset caused by the pure rotation between the current image frame and the target image frame. Based on the combination of the rotation matrix between the current image frame and the target image frame and the intrinsic parameter matrix of the camera that captured the image frame, the pixel offset caused by the pure rotation between the current image frame and the target image frame is calculated using the normalized phase plane coordinates of the feature pixels of the current image frame before the pure rotation.

8. The method according to claim 6, wherein, The step of calculating the pixel offset caused by the pure translation between the current image frame and the target image frame based on the inter-frame interval between the current image frame and the target image frame, and the pixel coordinate information of the current image frame and the target image frame, includes: Based on the inter-frame interval between the current image frame and the target image frame, and the pixel offset between the current image frame and the target image frame that meets the preset interval condition, the pixel offset caused by the pure translation between the current image frame and the target image frame is calculated.

9. An image feature tracking device, wherein, The device includes: The current image frame acquisition module is configured to acquire the current image frame and the current features of the current image frame, and record the current features in the tracking list; The target image frame acquisition module is configured to acquire the next image frame of the current image frame as the target image frame; The target feature recording module is configured to obtain target features of the same type as the current feature from the target image frame and record them in the target feature list; The feature projection calculation module is configured to project the current feature from the current image frame to the target image frame using a feature inter-frame pixel motion model to obtain a feature projection result; wherein, the feature inter-frame pixel motion model decomposes the inter-frame motion of the feature pixel into two parts: pure translation and pure rotation, calculates the pixel offset caused by each part respectively, and then sums the pixel offsets to obtain the overall offset as the feature projection result; The feature matching degree calculation module is configured to calculate the feature matching degree between the feature projection result and the target features recorded in the target feature list, respectively. The tracking list update module is configured to determine whether to record the target feature into the tracking list based on the feature matching degree; The tracking list output module is configured to output the tracking list based on preset conditions.

10. A computer program product comprising computer instructions that, when executed by a processor, implement the method as claimed in any one of claims 1-8.

Images