Visual tracking method, device and electronic device

Abstract

The present application discloses a visual tracking method, device and electronic device, belonging to the field of electronic technology. The method includes: when the first image frame collected is a descriptor extraction frame, obtaining N image feature points of the first image frame and the descriptor of each image feature point; performing descriptor tracking on the descriptor of each image feature point and M historical descriptor extraction frames, determining at least one first feature point and at least one second feature point, the M historical descriptor extraction frames include historical descriptor extraction frames within a period of a first preset time from the collection time of the first image frame; performing optical flow tracking on each second feature point and the second image frame; adding each first feature point pair and each second feature point pair to the tracking feature set corresponding to the first image frame, the first feature point pair includes the first feature point and the feature point tracked by the first feature point, and the second feature point pair includes the second feature point and the feature point tracked by the second feature point.

Description

Technical Field

[0001] The present application belongs to the field of electronic technology, and specifically relates to a visual tracking method, device and electronic equipment. Background Art

[0002] With the rapid development of electronic technology, electronic devices such as mobile phones and tablet computers have become more and more popular and have become an indispensable part of people's daily life. Among them, in order to improve the display effect of electronic devices in shooting moving targets, visual tracking technology has gradually been applied to electronic devices. It detects, extracts, identifies and tracks moving targets in image sequences to obtain the motion parameters of moving targets, such as position, speed, acceleration and motion trajectory, so as to locate the moving targets. However, in the process of visual tracking, current electronic devices usually consume a lot of time to ensure the accuracy of visual tracking. It can be seen that how to ensure the accuracy of visual tracking and reduce time consumption in the process of visual tracking of electronic devices has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a visual tracking method, device and electronic device, which can solve the problem of how to ensure the accuracy of visual tracking and reduce the time consumption during the visual tracking process of the electronic device.

[0004] In a first aspect, an embodiment of the present application provides a visual tracking method, comprising:

[0005] In the case where the first image frame collected is a descriptor extraction frame, obtaining N image feature points of the first image frame and a descriptor of each of the image feature points, where N is an integer greater than 1;

[0006] Descriptor tracking is performed on the descriptors of each of the image feature points and M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point, wherein the M historical descriptor extraction frames include historical descriptor extraction frames within a period of a first preset time length from the acquisition time of the first image frame; the first feature point is a feature point that is tracked among the N image feature points; the second feature point is a feature point that is not tracked among the N image feature points; and M is a positive integer;

[0007] Performing optical flow tracking on each of the second feature points and a second image frame, where the second image frame is an image frame previous to the first image frame;

[0008] Each first feature point pair and each second feature point pair are added to the tracking feature set corresponding to the first image frame, the first feature point pair includes the first feature point and the feature point tracked by the first feature point, and the second feature point pair includes the second feature point and the feature point tracked by the second feature point.

[0009] In a second aspect, an embodiment of the present application provides a visual tracking device, comprising:

[0010] A descriptor acquisition module, used for acquiring N image feature points of the first image frame and the descriptors of each of the image feature points when the first image frame acquired is a descriptor extraction frame, wherein N is an integer greater than 1;

[0011] A descriptor tracking module is used to perform descriptor tracking on the descriptors of each of the image feature points and M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point, wherein the M historical descriptor extraction frames include historical descriptor extraction frames within a period of a first preset time length from the acquisition time of the first image frame; the first feature point is a feature point that is tracked among the N image feature points; the second feature point is a feature point that is not tracked among the N image feature points; and M is a positive integer;

[0012] an optical flow tracking module, used for performing optical flow tracking on each of the second feature points and a second image frame, where the second image frame is an image frame previous to the first image frame;

[0013] A feature set construction module is used to add each first feature point pair and each second feature point pair to a tracking feature set corresponding to the first image frame, wherein the first feature point pair includes the first feature point and the feature point tracked by the first feature point, and the second feature point pair includes the second feature point and the feature point tracked by the second feature point.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0018] In an embodiment of the present application, when the first image frame collected is a descriptor extraction frame, N image feature points of the first image frame and the descriptors of each image feature point are first obtained, and then the descriptors of the image feature points are tracked with the M historical descriptor extraction frames within the first preset time period, and at least one first feature point that is successfully tracked and at least one second feature point that fails to be tracked are determined, and then the second feature point that fails to be tracked is tracked by optical flow, and finally the feature points tracked by each first feature point and each second feature point are formed into a tracking feature set corresponding to the first image frame. In this way, each first feature point in the first image frame can be tracked in M ​​historical descriptor extraction frames based on the descriptor method, and then each second feature point in the first image frame except the first feature point can be tracked in the previous image frame based on the optical flow method, which not only makes the visual tracking process have higher robustness, but also has faster speed, so that the accuracy of visual tracking can be guaranteed and the time consumption in visual tracking can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of an embodiment of the visual tracking method provided by the present application;

[0020] Figure 2 is a schematic diagram of continuous image frames in an embodiment of the visual tracking method provided by the present application;

[0021] Figure 3 is a schematic structural diagram of an embodiment of a visual tracking device provided by the present application;

[0022] Figure 4 is a schematic structural diagram of an embodiment of an electronic device provided by the present application;

[0023] Figure 5 It is a schematic structural diagram of another embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0026] The visual tracking method, device and electronic device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0027] See also Figure 1 , is a flow chart of an embodiment of the visual tracking method provided by the present application. The visual tracking method is applied to electronic devices, such as Figure 1 The visual tracking method at least includes the following steps 101 to 104.

[0028] Step 101: when the first image frame collected is a descriptor extraction frame, obtain N image feature points of the first image frame and a descriptor of each of the image feature points;

[0029] Step 102: perform descriptor tracking on the descriptors of the image feature points and M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point, wherein the M historical descriptor extraction frames include historical descriptor extraction frames within a period of a first preset time length from the acquisition time of the first image frame; the first feature point is a feature point that has been tracked among the N image feature points; the second feature point is a feature point that has not been tracked among the N image feature points; and M is a positive integer.

[0030] Step 103: perform optical flow tracking on each of the second feature points and a second image frame, where the second image frame is an image frame previous to the first image frame.

[0031] Step 104: Form a tracking feature set corresponding to the first image frame by combining the feature points tracked by the first feature points and the feature points tracked by the second feature points.

[0032] Based on this, when the first image frame collected is a descriptor extraction frame, first obtain N image feature points of the first image frame and the descriptors of each image feature point, then perform descriptor tracking on the descriptors of the image feature points and M historical descriptor extraction frames within the first preset time period, determine at least one first feature point that is successfully tracked and at least one second feature point that fails to be tracked, then perform optical flow tracking on the second feature point that fails to be tracked, and finally form a tracking feature set corresponding to the first image frame with the feature points tracked by each first feature point and each second feature point. In this way, each first feature point in the first image frame can be tracked in M ​​historical descriptor extraction frames based on the descriptor method, and then each second feature point in the first image frame except the first feature point can be tracked in the previous image frame based on the optical flow method, which not only makes the visual tracking process have higher robustness, but also has faster speed, thereby ensuring the accuracy of visual tracking and reducing the time consumption in visual tracking.

[0033] In the above step 101, when the electronic device acquires a first image frame, if it determines that the first image frame is a descriptor extraction frame, it obtains N image feature points of the first image frame and the descriptor of each image feature point.

[0034] In the embodiment of the present application, when the electronic device captures the first image frame, it can determine whether the first image frame is a descriptor extraction frame.

[0035] It should be noted that the electronic device determines whether the first image frame is a descriptor extraction frame by presetting a descriptor extraction frame judgment condition, and determines that the first image frame is a descriptor extraction frame when the first image frame meets the presetting descriptor extraction frame judgment condition. For example, it can be determined whether the first image frame is within a preset time period, and if so, the first image frame is determined to be a descriptor extraction frame; otherwise, it is a normal frame.

[0036] In some embodiments, before step 101, the method may further include: when a first image frame is captured, if the first image frame and a target historical descriptor extraction frame are separated by a preset number of image frames, determining that the first image frame is a descriptor extraction frame, and the target historical descriptor extraction frame is the historical descriptor extraction frame closest to the first image frame, thereby making the method of determining whether the first image frame is a descriptor extraction frame more flexible and accurate.

[0037] For example, it is assumed that the preset number of image frames is 3, and image frame 21 is a historical descriptor extraction frame (represented by a solid box). Figure 2 As shown, if the first image frame is image frame 22, since there are three image frames between image frame 22 and image frame 21, image frame 22 is also a descriptor extraction frame; if the first image frame is image frame 23, since there are only two image frames between image frame 23 and image frame 21, the image frame is a normal frame (indicated by a hollow square), and so on.

[0038] In addition, the above-mentioned acquisition of N image feature points of the first image frame may be that the electronic device extracts a feature point set of the first image frame through a preset feature point extraction model, and then uses part or all of the feature points in the feature point set as the above-mentioned N image feature points.

[0039] It should be noted that the feature point extraction model may be any model that can extract feature points from an image frame. Specifically, the feature point extraction model may include a fast feature detection (Features from Accelerated Segment Test, Fast) model.

[0040] In some implementations, the step of obtaining N image feature points of the first image frame includes:

[0041] Extracting a set of feature points of the first image frame;

[0042] Dividing the first image frame into a plurality of image blocks;

[0043] Based on the feature point set, N image feature points are determined in the multiple image blocks, wherein the number of image feature points distributed in each of the image blocks is less than or equal to a second preset number, and the second preset number is less than the N.

[0044] Based on this, by dividing the first image frame into multiple image blocks, and then extracting feature points less than or equal to a preset number from each image block in the extracted feature point set to form the above-mentioned N image feature points, the extracted N image feature points are evenly distributed on the first image frame, thereby improving the accuracy of visual tracking.

[0045] It should be noted that the determination of N image feature points in the plurality of image blocks based on the feature point set may be that the electronic device selects at least one feature point whose number is less than or equal to the second preset number as the image feature point in the image blocks where the feature points are distributed, according to the distribution of each feature point in the feature point set in the plurality of image blocks. Of course, if the feature points are not distributed in the image block, no feature point is selected in the image block.

[0046] In addition, the first image frame may be divided into a plurality of image blocks by presetting the number of image blocks to be divided, and then dividing the first image frame according to the set number of image blocks. For example, the first image frame may be divided into 1×3 or 2×2 image blocks.

[0047] In the embodiment of the present application, after the electronic device obtains the above-mentioned N image feature points, the electronic device can calculate the descriptor of each image feature point by reusing the descriptor calculation model.

[0048] It should be noted that the above descriptor calculation model can be any model that can realize the calculation of the descriptor of the feature point. Specifically, the above descriptor calculation model can include an Oriented Fast and Rotated Brief (ORB) model.

[0049] In the above step 102, after the electronic device obtains the above N image feature points and the descriptors of each image feature point, the electronic device can perform descriptor tracking on the descriptors of the image feature points and M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point.

[0050] In an embodiment of the present application, the descriptor tracking of the image feature points and the M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point may be performed by the electronic device based on a descriptor nearest distance search to match the feature points of the M historical descriptor extraction frames with the feature points of each image, and the image feature points that are successfully matched to the feature points in the M historical descriptor extraction frames are determined as the first feature points, while the image feature points that are not successfully matched to the feature points are determined as the second feature points.

[0051] It should be noted that the above-mentioned M historical descriptor extraction frames include historical descriptor extraction frames within a time period of a first preset time length from the acquisition time of the first image frame. The first preset time length can be a time length set according to actual needs, for example, it can be 1s or 2s, etc.

[0052] Exemplarily, when M is 1, the electronic device may obtain the descriptor distance between each image feature point and the feature point in the previous historical descriptor extraction frame, and then determine whether the minimum descriptor distance among the descriptor distances between each image feature point and the feature point in the previous historical descriptor extraction frame is less than or equal to a preset descriptor distance threshold. If so, the image feature point is determined as the first feature point; otherwise, it is determined as the second feature point.

[0053] Of course, when M is greater than 1, the electronic device can first obtain the descriptor distance between each image feature point and the feature point in the historical descriptor extraction frame of the previous frame of the first image frame according to the acquisition time of the M historical descriptor extraction frames. If the minimum descriptor distance between the image feature point and the feature point in the historical descriptor extraction frame of the previous frame is less than or equal to the preset descriptor distance threshold, the image feature point is determined to be the first feature point; otherwise, the descriptor distance between the image feature point and the feature point in the historical descriptor extraction frames of the previous two frames of the first image frame is continued to be obtained. If the minimum descriptor distance between the image feature point and the feature point in the historical descriptor extraction frames of the previous two frames is less than or equal to the preset descriptor distance threshold, the image feature point is determined to be the first feature point. If the descriptor distance is less than or equal to the above-mentioned preset descriptor distance threshold, the image feature point is determined to be the first feature point; otherwise, the descriptor distance between the image feature point and the feature point in the historical descriptor extraction frame of the three frames above the first image frame is continuously obtained, ..., and so on, until it is determined that the minimum descriptor distance between the image feature point and the feature point in a historical descriptor extraction frame is less than or equal to the preset descriptor distance threshold, and the image feature point is determined to be the first feature point; or, when the minimum descriptor distances between the image feature point and the feature points in M ​​historical descriptor extraction frames are all greater than the preset descriptor distance threshold, the image feature point is determined to be the second feature point.

[0054] In addition, in the process of performing the closest distance search based on the descriptor, the electronic device can perform feature projection through a preset camera model, estimate the pixel coordinates of each feature point in M ​​historical descriptor extraction frames on the first image frame, and search for the first feature point within a preset radius with the estimated pixel coordinates as the center on the first image frame, thereby narrowing the search range and speeding up the matching time.

[0055] In the above step 103, after the electronic device determines at least one first feature point and at least one second feature point among the above N image feature points, the electronic device can perform optical flow tracking on each second feature point among the above at least one second feature point and a second image frame, where the second image frame is an image frame previous to the first image frame.

[0056] It should be noted that the optical flow tracking of each second feature point and the second image frame may be performed by searching for feature points matching each second feature point in the second image frame through a preset optical flow feature point tracking model.

[0057] In addition, the optical flow tracking of each of the second feature points and the second image frame may be directly performing optical flow tracking on all feature points of the at least one second feature point.

[0058] In some implementations, before the above step 103, the following steps may also be included:

[0059] When the first image frame is a key frame, obtaining a bag-of-words vector of each of the second feature points;

[0060] The bag-of-words vectors of each of the second feature points are tracked with L historical key frames to determine at least one first sub-feature point and at least one second sub-feature point, wherein the L historical key frames include historical key frames within a period of a second preset time length from the acquisition time of the first image frame; the first sub-feature point is a feature point that is tracked among the at least one feature point; and the second sub-feature point is a feature point that is not tracked among the at least one feature point.

[0061] The step 103 may include: performing optical flow tracking on each of the second sub-feature points and the second image frame.

[0062] The step 104 may include:

[0063] Each first feature point pair, first sub-feature point pair, and second sub-feature point pair are added to a tracking feature set corresponding to the first image frame, each first sub-feature point pair includes the first sub-feature point and a feature point tracked by the first sub-feature, and each second sub-feature point pair includes the second sub-feature point and a feature point tracked by each second sub-feature point.

[0064] Based on this, when the above-mentioned first image frame is a key frame, it is also possible to track the feature points matching each second feature point in L historical key frames through visual tracking based on the bag-of-words method, and visually track the second sub-feature points that have not been tracked to the feature points through the optical flow method. In this way, the descriptor-based, bag-of-words-based and optical flow-based visual tracking can be combined to further improve the accuracy of visual tracking and reduce the time consumption.

[0065] It should be noted that after the above-mentioned electronic device determines the above-mentioned at least one second feature point, and before obtaining the bag-of-words vector of each second feature point when the above-mentioned first image frame is a key frame, the above-mentioned method may also include: when it is determined that the first image frame meets a preset condition, determining that the first image frame is a key frame.

[0066] The above-mentioned preset condition may be any condition that can be used to determine whether the captured image frame is a key frame. Specifically, the above-mentioned preset condition includes at least one of the following:

[0067] The time interval between the first image frame and the most recent key frame is greater than or equal to a preset time threshold;

[0068] Whether the rotation angle between the first image frame and the most recent key frame is greater than or equal to a preset angle threshold;

[0069] Whether the moving distance between the first image frame and the nearest key frame is greater than or equal to a preset distance threshold;

[0070] Whether the number of the first feature points is less than or equal to a preset number.

[0071] Based on this, through at least one of the above preset conditions, it can be achieved to determine whether the above first image frame is a key frame, so that the method of determining the key frame is flexible and diverse.

[0072] In addition, the second preset time length may be the same as the first preset time length, or may be a different time length, which is not limited here.

[0073] In the embodiment of the present application, the bag-of-words vectors of each of the second feature points are tracked with L historical key frames to determine at least one first sub-feature point and at least one second sub-feature point. This can be done by matching the feature points of the L historical key frames with each of the second feature points through a bag-of-words search.

[0074] Specifically, the above-mentioned bag-of-words tracking of the bag-of-words vectors of each of the second feature points with L historical key frames to determine at least one first sub-feature point and at least one second sub-feature point may include: determining, among the at least one second feature point, at least one candidate feature point whose bag-of-words vector matches the bag-of-words vectors of the feature points in the L historical key frames; performing descriptor tracking on the descriptors of each of the candidate feature points with the L historical key frames to determine at least one first sub-feature point, the first sub-feature point being a feature point tracked among the at least one candidate feature point; and the at least one second sub-feature point being a feature point of the at least one candidate feature point other than the at least one first sub-feature point.

[0075] Exemplarily, assuming that the above L is 1, the electronic device can first determine, among each second feature point, at least one candidate feature point whose bag-of-words vector matches the bag-of-words vector of the feature point in the previous historical key frame; perform descriptor tracking on the descriptor of each candidate feature point and the previous historical key frame to determine the above at least one first sub-feature point and the above at least one second sub-feature point.

[0076] Of course, when L is greater than 1, the electronic device can first match the feature points of the historical key frames whose acquisition time is close to the first image frame with the second feature points according to the order of the acquisition time of the L historical key frames. If no feature points are matched, then match the feature points of the historical key frames whose acquisition time is far from the first image frame with the second feature points. The implementation principle is similar to the principle of matching image feature points with feature points of M historical descriptor extraction frames, which will not be elaborated here.

[0077] It should be noted that, when the electronic device determines that the first image frame is not a descriptor extraction frame (ie, a common frame), the electronic device may track the first image frame based on an optical flow method.

[0078] In the above step 104, after the above electronic device performs optical flow tracking on each second feature point and the second image frame, the electronic device can obtain each first feature point pair and each second feature point pair, and form a tracking feature set through each first feature point pair and each second feature point pair, so as to locate the object in the first image frame through the tracking feature set.

[0079] In the embodiment of the present application, the first feature point pair includes: the first feature point and the feature point tracked by the first feature point; the second feature point pair includes: the second feature point and the feature point tracked by the second feature point.

[0080] It should be noted that the feature points tracked by the second feature points may all be image frames matched in the previous image frame based on the optical flow method; or, in the process of combining the bag-of-words and optical flow-based feature point matching for the second feature points, the feature points tracked by the second feature points may include feature points matched with the first sub-features tracked in the L-frame historical key frames based on the bag-of-words, and feature points matched with the second sub-features tracked in the previous image frame based on the optical flow method, that is, the tracking feature set includes a first feature point pair, a first sub-feature point pair, and a second sub-feature point pair, each of the first sub-feature point pairs includes the first sub-feature point and the feature points tracked by the first sub-feature, and each of the second sub-feature point pairs includes the second sub-feature point and the feature points tracked by each of the second sub-feature points.

[0081] Of course, when the above-mentioned first image frame is not a descriptor extraction frame and the first image frame is tracked based on the optical flow method, the feature point pairs in the above-mentioned tracking feature set may include: each image feature point, and feature points that are tracked in the above-mentioned previous image frame based on the optical flow method and match the image feature points.

[0082] The visual tracking method provided in the embodiment of the present application can be executed by a visual tracking device. In the embodiment of the present application, the visual tracking device provided in the embodiment of the present application is described by taking the visual tracking device executing the visual tracking method as an example.

[0083] like Figure 3 As shown in FIG. 1 , it is a schematic diagram of the structure of an embodiment of the visual tracking device provided by the present application. Figure 3 As shown, the visual tracking device 300 includes:

[0084] A descriptor acquisition module, used for acquiring N image feature points of the first image frame and the descriptors of each of the image feature points when the first image frame acquired is a descriptor extraction frame, wherein N is an integer greater than 1;

[0085] A descriptor tracking module is used to perform descriptor tracking on the descriptors of each of the image feature points and M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point, wherein the M historical descriptor extraction frames include historical descriptor extraction frames within a period of a first preset time length from the acquisition time of the first image frame; the first feature point is a feature point that is tracked among the N image feature points; the second feature point is a feature point that is not tracked among the N image feature points; and M is a positive integer;

[0086] an optical flow tracking module, used for performing optical flow tracking on each of the second feature points and a second image frame, where the second image frame is an image frame previous to the first image frame;

[0087] A feature set construction module is used to add each first feature point pair and each second feature point pair to a tracking feature set corresponding to the first image frame, wherein the first feature point pair includes the first feature point and the feature point tracked by the first feature point, and the second feature point pair includes the second feature point and the feature point tracked by the second feature point.

[0088] In some embodiments, the apparatus 300 further includes:

[0089] A bag-of-words vector acquisition module, used for acquiring a bag-of-words vector of each of the second feature points when the first image frame is a key frame;

[0090] A bag-of-words tracking module is used to perform bag-of-words tracking on the bag-of-words vectors of each of the second feature points and L historical key frames to determine at least one first sub-feature point and at least one second sub-feature point, wherein the L historical key frames include historical key frames within a period of a second preset time length from the acquisition time of the first image frame; the first sub-feature point is a feature point that is tracked among the at least one feature point; the second sub-feature point is a feature point that is not tracked among the at least one feature point, and L is a positive integer.

[0091] The optical flow tracking module 303 can be specifically used for:

[0092] Perform optical flow tracking on each of the second sub-feature points and the second image frame.

[0093] The feature set construction module 304 may be specifically used for:

[0094] Each first feature point pair, first sub-feature point pair, and second sub-feature point pair are added to a tracking feature set corresponding to the first image frame, each first sub-feature point pair includes the first sub-feature point and a feature point tracked by the first sub-feature, and each second sub-feature point pair includes the second sub-feature point and a feature point tracked by each second sub-feature point.

[0095] In some embodiments, the apparatus 300 further includes:

[0096] A key frame determination module is used to determine that the first image frame is a key frame when it is determined that the first image frame meets a preset condition, wherein the preset condition includes at least one of the following:

[0097] The time interval between the first image frame and the most recent key frame is greater than or equal to a preset time threshold;

[0098] Whether the rotation angle between the first image frame and the most recent key frame is greater than or equal to a preset angle threshold;

[0099] Whether the moving distance between the first image frame and the nearest key frame is greater than or equal to a preset distance threshold;

[0100] Whether the number of the first feature points is less than or equal to a preset number.

[0101] In some implementations, the descriptor acquisition module 301 includes:

[0102] A feature point set extraction unit, used to extract a feature point set of the first image frame;

[0103] An image block division unit, configured to divide the first image frame into a plurality of image blocks;

[0104] An image block determination unit is used to determine N image feature points in the multiple image blocks based on the feature point set, wherein the number of image feature points distributed in each of the image blocks is less than or equal to a second preset number, and the second preset number is less than the N.

[0105] In some embodiments, the apparatus 300 further includes:

[0106] When a first image frame is captured, if the first image frame is separated from a target historical descriptor extraction frame by a preset number of image frames, the first image frame is determined to be a descriptor extraction frame, and the target historical descriptor extraction frame is the historical descriptor extraction frame closest to the first image frame.

[0107] The visual tracking device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0108] The visual tracking device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0109] The visual tracking device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0110] Alternatively, if Figure 4 As shown, an embodiment of the present application further provides an electronic device 400, including a processor 401 and a memory 402, wherein the memory 402 stores programs or instructions that can be executed on the processor 401, and when the program or instructions are executed by the processor 401, the various steps of the above-mentioned visual tracking method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0111] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0112] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0113] The electronic device 500 includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, and a processor 510.

[0114] Those skilled in the art will appreciate that the electronic device 500 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0115] The processor 510 is used for:

[0116] In the case where the first image frame collected is a descriptor extraction frame, obtaining N image feature points of the first image frame and a descriptor of each of the image feature points, where N is an integer greater than 1;

[0117] Descriptor tracking is performed on the descriptors of each of the image feature points and M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point, wherein the M historical descriptor extraction frames include historical descriptor extraction frames within a period of a first preset time length from the acquisition time of the first image frame; the first feature point is a feature point that is tracked among the N image feature points; the second feature point is a feature point that is not tracked among the N image feature points; and M is a positive integer;

[0118] Performing optical flow tracking on each of the second feature points and a second image frame, where the second image frame is an image frame previous to the first image frame;

[0119] Each first feature point pair and each second feature point pair are added to the tracking feature set corresponding to the first image frame, the first feature point pair includes the first feature point and the feature point tracked by the first feature point, and the second feature point pair includes the second feature point and the feature point tracked by the second feature point.

[0120] In some implementations, the processor 510 is further configured to:

[0121] When the first image frame is a key frame, obtaining a bag-of-words vector of each of the second feature points;

[0122] Performing bag-of-words tracking on the bag-of-words vectors of each of the second feature points and L historical key frames to determine at least one first sub-feature point and at least one second sub-feature point, wherein the L historical key frames include historical key frames within a period of a second preset time length from the acquisition time of the first image frame; the first sub-feature point is a feature point that is tracked among the at least one feature point; the second sub-feature point is a feature point that is not tracked among the at least one feature point, and L is a positive integer;

[0123] Performing optical flow tracking on each of the second sub-feature points and the second image frame;

[0124] Each first feature point pair, first sub-feature point pair, and second sub-feature point pair are added to a tracking feature set corresponding to the first image frame, each first sub-feature point pair includes the first sub-feature point and a feature point tracked by the first sub-feature, and each second sub-feature point pair includes the second sub-feature point and a feature point tracked by each second sub-feature point.

[0125] In some implementations, the processor 510 is further configured to:

[0126] In the case where it is determined that the first image frame meets a preset condition, the first image frame is determined to be a key frame, wherein the preset condition includes at least one of the following:

[0127] The time interval between the first image frame and the most recent key frame is greater than or equal to a preset time threshold;

[0128] Whether the rotation angle between the first image frame and the most recent key frame is greater than or equal to a preset angle threshold;

[0129] Whether the moving distance between the first image frame and the nearest key frame is greater than or equal to a preset distance threshold;

[0130] Whether the number of the first feature points is less than or equal to a preset number.

[0131] In some implementations, the processor 510 is specifically configured to:

[0132] Extracting a set of feature points of the first image frame;

[0133] Dividing the first image frame into a plurality of image blocks;

[0134] Based on the feature point set, N image feature points are determined in the multiple image blocks, wherein the number of image feature points distributed in each of the image blocks is less than or equal to a second preset number, and the second preset number is less than the N.

[0135] In some implementations, the processor 510 is further configured to:

[0136] When a first image frame is captured, if the first image frame is separated from a target historical descriptor extraction frame by a preset number of image frames, the first image frame is determined to be a descriptor extraction frame, and the target historical descriptor extraction frame is the historical descriptor extraction frame closest to the first image frame.

[0137] It should be understood that in the embodiment of the present application, the input unit 504 may include a graphics processor (GPU) 5041 and a microphone 5042, and the graphics processor 5041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 506 may include a display panel 5061, and the display panel 5061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 507 includes a touch panel 5071 and at least one of other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include two parts: a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0138] The memory 509 can be used to store software programs and various data. The memory 509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 509 may include a volatile memory or a non-volatile memory, or the memory 509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0139] The processor 510 may include one or more processing units; optionally, the processor 510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 510.

[0140] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned visual tracking method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0141] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0142] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned visual tracking method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0143] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0144] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned visual tracking method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0145] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A visual tracking method, characterized in that: include: In the case where the first image frame collected is a descriptor extraction frame, obtaining N image feature points of the first image frame and a descriptor of each of the image feature points, where N is an integer greater than 1; Descriptor tracking is performed on the descriptors of each of the image feature points and M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point, wherein the M historical descriptor extraction frames include historical descriptor extraction frames within a period of a first preset time length from the acquisition time of the first image frame; the first feature point is a feature point that is tracked among the N image feature points; the second feature point is a feature point that is not tracked among the N image feature points; and M is a positive integer; Performing optical flow tracking on each of the second feature points and a second image frame, where the second image frame is an image frame previous to the first image frame; Each first feature point pair and each second feature point pair are added to the tracking feature set corresponding to the first image frame, the first feature point pair includes the first feature point and the feature point tracked by the first feature point, and the second feature point pair includes the second feature point and the feature point tracked by the second feature point.

2. The method according to claim 1, characterized in that: Before performing optical flow tracking on each of the second feature points and the second image frame, the method further includes: When the first image frame is a key frame, obtaining a bag-of-words vector of each of the second feature points; Performing bag-of-words tracking on the bag-of-words vectors of each of the second feature points and L historical key frames to determine at least one first sub-feature point and at least one second sub-feature point, wherein the L historical key frames include historical key frames within a period of a second preset time length from the acquisition time of the first image frame; the first sub-feature point is a feature point that is tracked among the at least one feature point; the second sub-feature point is a feature point that is not tracked among the at least one feature point, and L is a positive integer; The performing optical flow tracking on each of the second feature points and the second image frame includes: Performing optical flow tracking on each of the second sub-feature points and the second image frame; The step of adding each first feature point pair and each second feature point pair to a tracking feature set corresponding to the first image frame comprises: Each first feature point pair, first sub-feature point pair, and second sub-feature point pair are added to a tracking feature set corresponding to the first image frame, each first sub-feature point pair includes the first sub-feature point and a feature point tracked by the first sub-feature, and each second sub-feature point pair includes the second sub-feature point and a feature point tracked by each second sub-feature point.

3. The method according to claim 2, characterized in that Before obtaining the bag-of-words vectors of each of the second feature points when the first image frame is a key frame, the method further includes: In the case where it is determined that the first image frame meets a preset condition, the first image frame is determined to be a key frame, wherein the preset condition includes at least one of the following: The time interval between the first image frame and the most recent key frame is greater than or equal to a preset time threshold; Whether the rotation angle between the first image frame and the most recent key frame is greater than or equal to a preset angle threshold; Whether the moving distance between the first image frame and the nearest key frame is greater than or equal to a preset distance threshold; Whether the number of the first feature points is less than or equal to a preset number.

4. The method according to claim 1, characterized in that: The acquiring N image feature points of the first image frame includes: Extracting a set of feature points of the first image frame; Dividing the first image frame into a plurality of image blocks; Based on the feature point set, N image feature points are determined in the multiple image blocks, wherein the number of image feature points distributed in each of the image blocks is less than or equal to a second preset number, and the second preset number is less than the N.

5. The method according to claim 1, characterized in that When the collected first image frame is a descriptor extraction frame, before obtaining N image feature points of the first image frame and the descriptors of each of the image feature points, the method further includes: When a first image frame is captured, if the first image frame is separated from a target historical descriptor extraction frame by a preset number of image frames, the first image frame is determined to be a descriptor extraction frame, and the target historical descriptor extraction frame is the historical descriptor extraction frame closest to the first image frame.

6. A visual tracking device, characterized in that: include: A descriptor acquisition module, used for acquiring N image feature points of the first image frame and the descriptors of each of the image feature points when the first image frame acquired is a descriptor extraction frame, wherein N is an integer greater than 1; A descriptor tracking module is used to perform descriptor tracking on the descriptors of each of the image feature points and M historical descriptor extraction frames to determine at least one first feature point and at least one second feature point, wherein the M historical descriptor extraction frames include historical descriptor extraction frames within a period of a first preset time length from the acquisition time of the first image frame; the first feature point is a feature point that is tracked among the N image feature points; the second feature point is a feature point that is not tracked among the N image feature points; and M is a positive integer; an optical flow tracking module, used for performing optical flow tracking on each of the second feature points and a second image frame, where the second image frame is an image frame previous to the first image frame; A feature set construction module is used to add each first feature point pair and each second feature point pair to a tracking feature set corresponding to the first image frame, wherein the first feature point pair includes the first feature point and the feature point tracked by the first feature point, and the second feature point pair includes the second feature point and the feature point tracked by the second feature point.

7. The device according to claim 6, characterized in that Also includes: A bag-of-words vector acquisition module, used for acquiring a bag-of-words vector of each of the second feature points when the first image frame is a key frame; A bag-of-words tracking module is used to perform bag-of-words tracking on the bag-of-words vectors of each of the second feature points and L historical key frames to determine at least one first sub-feature point and at least one second sub-feature point, wherein the L historical key frames include historical key frames within a period of a second preset time from the acquisition time of the first image frame; the first sub-feature point is a feature point that is tracked among the at least one feature point; the second sub-feature point is a feature point that is not tracked among the at least one feature point, and L is a positive integer; The optical flow tracking module is specifically used for: Performing optical flow tracking on each of the second sub-feature points and the second image frame; The feature set construction module is specifically used for: Each first feature point pair, first sub-feature point pair, and second sub-feature point pair are added to a tracking feature set corresponding to the first image frame, each first sub-feature point pair includes the first sub-feature point and a feature point tracked by the first sub-feature, and each second sub-feature point pair includes the second sub-feature point and a feature point tracked by each second sub-feature point.

8. The device according to claim 7, characterized in that Also includes: A key frame determination module is used to determine that the first image frame is a key frame when it is determined that the first image frame meets a preset condition, wherein the preset condition includes at least one of the following: The time interval between the first image frame and the most recent key frame is greater than or equal to a preset time threshold; Whether the rotation angle between the first image frame and the most recent key frame is greater than or equal to a preset angle threshold; Whether the moving distance between the first image frame and the nearest key frame is greater than or equal to a preset distance threshold; Whether the number of the first feature points is less than or equal to a preset number.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the visual tracking method according to any one of claims 1 to 5 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the visual tracking method according to any one of claims 1 to 5 are implemented.

Images