A terminal, an image optical flow tracking method, device, chip and storage medium

Abstract

The application belongs to the technical field of image processing, and particularly relates to a terminal, an image optical flow tracking method and device, a chip and a storage medium. The application extracts edge features of a target region of a current frame image, so that when optical flow tracking of an image is performed, edge features of a region with less point features and line features can be tracked, that is, optical flow tracking of an image region with less rich texture and more blank regions can also be realized, and therefore, the visual positioning accuracy of the image can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a terminal, an image optical flow tracking method and device, a chip and a storage medium. BACKGROUND

[0002] Visual positioning is widely used in mobile robot positioning due to its low sensor cost and rich perception information. A visual positioning scheme usually calculates the robot motion and pose by using the correlation of features between different images.

[0003] However, the current visual positioning effect is poor for image regions that lack rich textures and have many blank regions, and there is a problem that the positioning accuracy of the robot cannot be met, which may even affect the robot motion. SUMMARY

[0004] The embodiments of the present application provide a terminal, an image optical flow tracking method and device, a chip and a storage medium, which are beneficial to improve the accuracy of visual positioning.

[0005] The first aspect of the embodiments of the present application provides a terminal, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0006] obtaining a current frame image that needs to be tracked by optical flow and a next frame image of the current frame image;

[0007] identifying point features and line features in the current frame image, and regarding a region of a preset area in which the number of the point features is less than or equal to a first number threshold and / or the number of the line features is less than or equal to a second number threshold as a target region;

[0008] extracting edge features from the target region of the current frame image to obtain to-be-tracked edge features of the current frame image;

[0009] tracking the to-be-tracked edge features of the current frame image by optical flow on the next frame image of the current frame image to obtain target edge features corresponding to the to-be-tracked edge features in the current frame image on the next frame image.

[0010] In a possible implementation manner of the present application, when the processor executes the computer program to implement the step of regarding a region of a preset area in which the number of the point features is less than or equal to a first number threshold and / or the number of the line features is less than or equal to a second number threshold as a target region, and the step of extracting edge features from the target region of the current frame image to obtain to-be-tracked edge features of the current frame image, the steps specifically include the following steps:

[0011] performing grid processing on the current frame image to obtain each grid image in the current frame image;

[0012] taking, as a target grid image, a grid image in which the number of point features is less than or equal to the first number threshold and / or the number of line features is less than or equal to the second number threshold;

[0013] performing edge feature extraction on the target grid image in the current frame image to obtain to-be-tracked edge features of the target grid image, and taking the to-be-tracked edge features of the target grid image as to-be-tracked edge features of the current frame image.

[0014] In a possible implementation of the present application, before performing edge feature extraction on the target grid image in the current frame image, the processor executing the computer program further implements the following steps:

[0015] performing edge detection on the current frame image to obtain edge features of the current frame image;

[0016] When the processor executing the computer program performs edge feature extraction on the target grid image in the current frame image to obtain to-be-tracked edge features of the current frame image, the following steps are implemented:

[0017] if the target grid image does not contain the edge features, taking N to-be-tracked pixel points in the target grid image as to-be-tracked edge features of the target grid image; the N to-be-tracked pixel points are the first N pixel points in the target grid image after sorting pixel points in the target grid image in descending order of gradients, and N is an integer greater than or equal to 1.

[0018] In a possible implementation of the present application, when the processor executes the computer program, the following steps are further implemented:

[0019] performing point feature extraction and line feature extraction on other regions in the current frame image except the target region to obtain to-be-tracked point features and to-be-tracked line features;

[0020] performing optical flow tracking on the to-be-tracked point features and the to-be-tracked line features of the current frame image on a next frame image of the current frame image to obtain target point features and target line features corresponding to the to-be-tracked point features and the to-be-tracked line features in the current frame image on the next frame image.

[0021] In a possible implementation manner of the present application, when the processor executes the computer program to perform optical flow tracking on the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature of the current frame image in the next frame image of the current frame image to obtain the target point feature, the target line feature and the target edge feature corresponding to the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature in the next frame image respectively, the method comprises the following steps:

[0022] constructing m-layer pyramid images corresponding to the next frame image of the current frame image, performing optical flow tracking on an i-layer image in the m-layer pyramid images in a non-photometric approximation direction of a to-be-tracked target feature, performing optical flow tracking on a j-layer image in the m-layer pyramid images in a photometric approximation direction of the to-be-tracked target feature to obtain a target feature corresponding to the to-be-tracked target feature in the next frame image, the to-be-tracked target feature comprising one or more of the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature; the target feature comprising one or more of the target point feature, the target line feature and the target edge feature; wherein m=i+j; i and j are both integers greater than or equal to 1.

[0023] In a possible implementation manner of the present application, when the processor executes the computer program to perform optical flow tracking on the to-be-tracked line feature of the current frame image in the next frame image of the current frame image to obtain the target line feature, the method comprises the following steps:

[0024] performing feature point sampling on the to-be-tracked line feature to obtain to-be-tracked sampling points of the to-be-tracked line feature;

[0025] performing optical flow tracking on the to-be-tracked sampling points in the next frame image of the current frame image to obtain target sampling points corresponding to the to-be-tracked sampling points in the next frame image, and obtaining the target line feature according to the target sampling points.

[0026] In a possible implementation manner of the present application, after the processor executes the computer program to obtain the target line feature, the following steps are further implemented:

[0027] performing straight line fitting on the target sampling points in the target line feature to obtain a straight line equation;

[0028] calculating a first distance from the target sampling points in the target line feature to the fitted straight line according to the straight line equation;

[0029] deleting the target sampling points with the first distance greater than a first distance threshold.

[0030] In a possible implementation manner of the present application, after the processor executes the computer program to obtain the target point feature, the target line feature and the target edge feature corresponding to the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature in the current frame image respectively, the following steps are further implemented:

[0031] calculating a fundamental matrix between the current frame image and the next frame image;

[0032] calculating an epipolar line of a first feature point on the current frame image on the next frame image based on the fundamental matrix; the first feature point includes the to-be-tracked point feature, a to-be-tracked sampling point on the to-be-tracked line feature and a to-be-tracked pixel point on the to-be-tracked edge feature;

[0033] calculating a second distance of a second feature point to an epipolar line corresponding to the second feature point; the second feature point is a feature point on the next frame image corresponding to the first feature point;

[0034] deleting the second feature point with the second distance greater than a second distance threshold.

[0035] In a possible implementation manner of the present application, when the processor executes the computer program, the following steps are further implemented:

[0036] determining a number of target sampling points corresponding to the target line feature;

[0037] deleting the target line feature with the number of target sampling points less than a number threshold.

[0038] In a possible implementation manner of the present application, after the processor executes the computer program to obtain the target line feature, the following steps are further implemented:

[0039] taking an intersection point between an epipolar line corresponding to a target sampling point and a fitting straight line corresponding to the target sampling point as a sampling point obtained by correcting the target sampling point.

[0040] In a possible implementation manner of the present application, when the processor executes the computer program, the following steps are further implemented:

[0041] taking the next frame image as a new current frame image, taking the target point feature, the target line feature and the target edge feature as a new to-be-tracked point feature, a new to-be-tracked line feature and a new to-be-tracked edge feature, and continuing to perform optical flow tracking until optical flow tracking of each frame image in the continuous multiple frames of to-be-tracked images is completed.

[0042] The second aspect of the embodiment of the present application provides an optical flow tracking method of an image, comprising:

[0043] acquire a current frame image which needs to be tracked by optical flow and a next frame image of the current frame image;

[0044] identify point features and line features in the current frame image, and take a region of a preset area in which a number of the point features in the current frame image is less than or equal to a first number threshold and / or a number of the line features is less than or equal to a second number threshold as a target region;

[0045] perform edge feature extraction on the target region of the current frame image to obtain to-be-tracked edge features of the current frame image;

[0046] perform optical flow tracking on the to-be-tracked edge features of the current frame image on the next frame image of the current frame image to obtain target edge features corresponding to the to-be-tracked edge features in the current frame image on the next frame image.

[0047] In a possible implementation manner of the present application, the taking of the region of the preset area in which the number of the point features in the current frame image is less than or equal to the first number threshold and / or the number of the line features is less than or equal to the second number threshold as the target region and the performing of the edge feature extraction on the target region of the current frame image to obtain the to-be-tracked edge features of the current frame image comprise:

[0048] perform grid processing on the current frame image to obtain each grid image in the current frame image;

[0049] take a grid image in which a number of the point features is less than or equal to the first number threshold and / or a number of the line features is less than or equal to the second number threshold as a target grid image;

[0050] perform edge feature extraction on the target grid image in the current frame image to obtain to-be-tracked edge features of the target grid image, and take the to-be-tracked edge features of the target grid image as the to-be-tracked edge features of the current frame image.

[0051] In a possible implementation manner of the present application, before the performing of the edge feature extraction on the target grid image in the current frame image, the method comprises:

[0052] perform edge detection on the current frame image to obtain edge features of the current frame image;

[0053] the performing of the edge feature extraction on the target grid image in the current frame image to obtain the to-be-tracked edge features of the current frame image comprises:

[0054] If the edge feature is not included in the target grid image, N to-be-tracked pixel points in the target grid image are taken as to-be-tracked edge features of the target grid image, the N to-be-tracked pixel points are the first N pixel points in the target grid image in descending order of gradients, and N is an integer greater than or equal to 1.

[0055] In a possible implementation manner of the present application, the optical flow tracking method of the image further includes:

[0056] Point feature extraction and line feature extraction are performed on a region other than the target region in the current frame image, to-be-tracked point features and to-be-tracked line features are obtained;

[0057] Optical flow tracking is performed on the to-be-tracked point features and the to-be-tracked line features of the current frame image on a next frame image of the current frame image, to obtain target point features and target line features corresponding to the to-be-tracked point features and the to-be-tracked line features in the current frame image on the next frame image.

[0058] In a possible implementation manner of the present application, the optical flow tracking is performed on the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features of the current frame image on the next frame image of the current frame image, to obtain target point features, target line features and target edge features corresponding to the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features in the current frame image on the next frame image, and the method includes:

[0059] An m-layer pyramid image corresponding to the next frame image of the current frame image is constructed, optical flow tracking is performed on an i-layer image in the m-layer pyramid image in a non-photometric approximation direction of to-be-tracked target features, optical flow tracking is performed on a j-layer image in the m-layer pyramid image in a photometric approximation direction of the to-be-tracked target features, to obtain target features corresponding to the to-be-tracked target features on the next frame image, the to-be-tracked target features include one or more of the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features, and the target features include one or more of the target point features, the target line features and the target edge features; wherein m=i+j, i and j are integers greater than or equal to 1.

[0060] In a possible implementation manner of the present application, the optical flow tracking is performed on the to-be-tracked line features of the current frame image on the next frame image of the current frame image, to obtain the target line features, and the method includes:

[0061] Feature point sampling is performed on the to-be-tracked line features, to obtain to-be-tracked sampling points of the to-be-tracked line features;

[0062] perform optical flow tracking on the next frame image of the current frame image on the to-be-tracked sampling points to obtain target sampling points corresponding to the to-be-tracked sampling points on the next frame image, and obtain the target line feature according to the target sampling points.

[0063] In a possible implementation manner of the present application, after the target line feature is obtained, the following steps are further included:

[0064] perform straight line fitting on the target sampling points in the target line feature to obtain a straight line equation;

[0065] calculate a first distance from the target sampling points in the target line feature to the fitted straight line according to the straight line equation;

[0066] delete the target sampling points with the first distance greater than a first distance threshold.

[0067] In a possible implementation manner of the present application, after the target point feature, the target line feature and the target edge feature corresponding to the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature in the current frame image respectively on the next frame image are obtained, the following steps are further included:

[0068] calculate a fundamental matrix between the current frame image and the next frame image;

[0069] calculate an epipolar line of a first feature point on the current frame image on the next frame image based on the fundamental matrix; the first feature point includes the to-be-tracked sampling points on the to-be-tracked line feature and the to-be-tracked pixel points on the to-be-tracked edge feature;

[0070] calculate a second distance from a second feature point to an epipolar line corresponding to the second feature point; the second feature point is a feature point on the next frame image corresponding to the first feature point;

[0071] delete the second feature point with the second distance greater than a second distance threshold.

[0072] In a possible implementation manner of the present application, the optical flow tracking method further includes:

[0073] determine a number of target sampling points corresponding to the target line feature;

[0074] delete the target line feature with the number of target sampling points less than a number threshold.

[0075] In a possible implementation manner of the present application, after the target line feature is obtained, the following steps are further included:

[0076] take an intersection point between an epipolar line corresponding to a target sampling point and a fitted straight line corresponding to the target sampling point as a sampling point obtained by correcting the target sampling point.

[0077] In a possible implementation manner of the present application, the current frame image is a first frame image in the continuous multiple frames of to-be-tracked images, and after target point features, target line features and target edge features corresponding to the to-be-tracked point features, to-be-tracked line features and to-be-tracked edge features in the current frame image are obtained on the next frame image, the method comprises:

[0078] taking the next frame image as a new current frame image, taking the target point features, target line features and target edge features as new to-be-tracked point features, new to-be-tracked line features and new to-be-tracked edge features, and continuing optical flow tracking until optical flow tracking of each frame image in the continuous multiple frames of to-be-tracked images is completed.

[0079] The third aspect of the embodiment of the present application provides an optical flow tracking device of images, comprising:

[0080] an acquisition unit configured to acquire a current frame image which needs to be subjected to optical flow tracking and a next frame image of the current frame image;

[0081] an identification unit configured to identify point features and line features in the current frame image, and take a region of a preset area in which a number of the point features is less than or equal to a first number threshold and / or a number of the line features is less than or equal to a second number threshold as a target region;

[0082] an extraction unit configured to perform edge feature extraction on the target region of the current frame image to obtain to-be-tracked edge features of the current frame image;

[0083] a tracking unit configured to perform optical flow tracking on the to-be-tracked edge features of the current frame image on the next frame image of the current frame image to obtain target edge features corresponding to the to-be-tracked edge features in the current frame image on the next frame image.

[0084] The fourth aspect of the embodiment of the present application provides a chip comprising a processor, wherein the processor is configured to read and execute a computer program stored in a memory to realize the steps of the method of the second aspect.

[0085] The fifth aspect of the embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the method of the second aspect.

[0086] In the embodiment of the present application, the edge features are extracted from the target region of the current frame image, so that when the optical flow tracking of the image is performed, the edge features of the region with less point features and line features can be tracked, that is, the optical flow tracking of the image can be realized for the image region with less rich texture and more blank regions, and thus the visual positioning accuracy of the image can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0087] In order to more clearly illustrate the method scheme of the embodiments of the present application, the drawings required in the embodiments will be briefly introduced as follows, and it should be understood that the following drawings only show some embodiments of the present application, and thus should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of the drawings.

[0088] Figure 1 is a schematic diagram of the optical flow provided by the embodiments of the present application;

[0089] Figure 2 is a structural schematic diagram of a terminal provided by the embodiments of the present application;

[0090] Figure 3 is a schematic diagram of the optical flow tracking direction provided by the embodiments of the present application;

[0091] Figure 4 is a schematic diagram of the implementation flow of the optical flow tracking method of the image provided by the embodiments of the present application;

[0092] Figure 5 is a structural schematic diagram of the optical flow tracking device of the image provided by the embodiments of the present application; DETAILED DESCRIPTION

[0093] In order to make the purpose, method scheme and advantages of the present application more clear and explicit, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0094] In this paper, "and / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean that there are three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two, and "at least one" or "one or more" means one, two or more than two.

[0095] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0096] In order to better understand the solution of this application, some technical terms used in this application are explained below.

[0097] First of all, optical flow refers to the displacement of the corresponding pixel points of the target object between two consecutive image frames due to the movement of the target object or the camera.

[0098] For example, Figure 1 As shown, when the moving object moves from position a to position b, there is a displacement (optical flow) between the pixel point corresponding to the moving object at position a and the pixel point corresponding to the moving object at position b as shown by the dotted line in the figure.

[0099] Optical flow tracking algorithms are used in various fields, such as motion detection of moving objects. For example, they are used in the field of mobile robots to calculate robot motion and posture.

[0100] Generally, optical flow tracking algorithms are divided into two categories: dense optical flow tracking algorithms and sparse optical flow tracking algorithms. The dense optical flow tracking algorithm performs optical flow tracking on all points in each frame image, while the sparse optical flow tracking algorithm performs optical flow tracking on a sparse feature point set in each frame image.

[0101] Specifically, the sparse optical flow tracking algorithm can obtain the sparse feature point set of the previous frame image through corner detection, and then use the previous frame image and the sparse feature point set of the previous frame image and the next frame image to obtain the point set with the same features in the next frame image, and obtain the optical flow between the sparse feature point sets of the two frames of images.

[0102] From this, we can see that point features are currently commonly used as image features. However, for image areas that lack rich textures and have more blank areas, the number of point features extracted is relatively small. Therefore, the visual positioning effect is poor, and there is a problem of being unable to meet the positioning accuracy of the robot, and it may even affect the robot's movement.

[0103] Based on this, the embodiment of the present application provides a terminal, an image optical flow tracking method and device, a chip and a storage medium, which are beneficial to improve the accuracy of visual positioning.

[0104] Specifically, the present application provides a terminal, an image optical flow tracking method and device, a chip and a storage medium, by performing edge feature extraction on the target region of the current frame image, so that when performing image optical flow tracking, edge features can be tracked in the region with less point features and line features, that is, image optical flow tracking can also be realized in the image region with less rich texture and more blank regions, so that the accuracy of visual positioning of the image can be improved.

[0105] In order to better understand the image optical flow tracking method provided by the embodiment of the present application, the specific implementation process is introduced from the implementation level in combination with the drawings.

[0106] Exemplarily, as shown in FIG. 1, the image optical flow tracking method provided by the embodiment of the present application can include the following steps. Figure 2 A structure schematic diagram of a terminal provided by an embodiment of the present application is shown. The terminal can be a robot, a mobile phone, a computer or the like. The terminal 2 can include but is not limited to a processor 20, a memory 21 and a computer program 22 stored in the memory 21 and executable on the processor 20.

[0107] The processor 10 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0108] The memory 21 can be an internal storage unit of the terminal 2, such as a hard disk or a memory of the terminal 2. The memory 21 can also be an external storage device of the terminal 2, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card or the like equipped on the terminal 2. Further, the memory 21 can include both the internal storage unit and the external storage device of the terminal 2. The memory 21 is used to store computer programs and other programs and data required by the electronic device. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0109] For example, the computer program 22 can be divided into one or more modules / units, one or more modules / units are stored in the memory 21 and executed by the processor 20 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 22 in the terminal 2.

[0110] It can be understood that the structure of the embodiments of the present application does not constitute a specific limitation of the terminal. In other embodiments of the present application, the device can include more or fewer components than the illustration, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0111] The following embodiments can be implemented on the terminal 2 with the above structure.

[0112] When the processor 20 executes the computer program 22, the optical flow tracking method of steps 201 to 204 can be implemented.

[0113] Step 201, obtaining a current frame image which needs to be tracked by optical flow and a next frame image of the current frame image.

[0114] In the embodiments of the present application, the current frame image which needs to be tracked by optical flow can be a certain frame of video image obtained by photographing a moving object, and the next frame image of the current frame image can be a next frame of video image in the video image, and the present application does not limit the source of the current frame image and the next frame image of the current frame image.

[0115] For example, in some embodiments of the present application, the current frame image which needs to be tracked by optical flow can also be a preview frame image obtained by photographing at the current time of the camera, and the next frame image of the current frame image is a preview frame image obtained by photographing at the next time of the camera.

[0116] Step 202, identifying point features and line features in the current frame image, and taking a region with a number of point features less than or equal to a first number threshold and / or a number of line features less than or equal to a second number threshold in a preset area as a target region.

[0117] In the embodiments of the present application, the to-be-tracked point features can be identified by using a feature extraction algorithm such as Harris corner point feature algorithm, Fast feature algorithm or Super Point feature algorithm, and the present application does not limit this.

[0118] The to-be-tracked line features can be recognized by using a line segment extraction algorithm such as Edline, LSD (Line Segment Detector), and LBD (Lind Band Descriptor), and the application does not limit this.

[0119] In the process of taking, as the target region, a region of a preset area in which the number of point features in the current frame image is less than or equal to a first number threshold and / or the number of line features is less than or equal to a second number threshold, the application can be implemented in any one of the following three ways:

[0120] First, taking, as the target region, a region of a preset area in which the number of point features in the current frame image is less than or equal to a first number threshold.

[0121] Second, taking, as the target region, a region of a preset area in which the number of line features in the current frame image is less than or equal to a second number threshold.

[0122] Third, taking, as the target region, a region of a preset area in which the number of point features in the current frame image is less than or equal to a first number threshold and the number of line features is less than or equal to a second number threshold.

[0123] When the number of point features in a region of a preset area of an image is less than or equal to a first number threshold, it indicates that the region is a region lacking point features. When the number of line features in a region of a preset area of an image is less than or equal to a second number threshold, it indicates that the region is a region lacking line features. When the number of point features in a region of an image is less than or equal to a first number threshold and the number of line features is less than or equal to a second number threshold, it indicates that the region is a region lacking rich textures or is a blank region.

[0124] It should be noted that the actual value of the preset area can be set according to the actual application scenario, and the preset area corresponding to different target regions and the corresponding first number threshold and second number threshold can be different or the same, and the application does not limit this, and for a certain region, the corresponding preset area and the corresponding first number threshold and second number threshold are determined.

[0125] In step 203, edge feature extraction is performed on the target region of the current frame image to obtain to-be-tracked edge features of the current frame image.

[0126] By performing edge feature extraction on the target region of the current frame image to obtain to-be-tracked edge features of the current frame image, the application can make the target region have features that can be tracked by optical flow, i.e., the to-be-tracked edge features.

[0127] Optionally, in some embodiments of the present application, the to-be-tracked edge feature can be a feature composed of one or more pixel points (to-be-tracked pixel points) with the maximum gradient in the target region.

[0128] In particular, in some embodiments of the present application, the to-be-tracked edge feature can be composed of only one pixel point with the maximum gradient in the target region, and thus, when performing optical flow tracking on the to-be-tracked edge feature, only the one pixel point with the maximum gradient needs to be tracked.

[0129] Specifically, in some embodiments of the present application, steps 202 to 203 can be implemented by steps 301 to 303.

[0130] Step 301: performing grid processing on the current frame image to obtain each grid image in the current frame image.

[0131] In the embodiments of the present application, the size of the grid image can be determined according to the image size and the accuracy of optical flow tracking, and the present application does not limit this.

[0132] Step 302: taking, as a target grid image, each grid image in which the number of point features is less than or equal to a first number threshold and / or the number of line features is less than or equal to a second number threshold.

[0133] Step 303: performing edge feature extraction on the target grid image in the current frame image to obtain a to-be-tracked edge feature of the target grid image, and taking the to-be-tracked edge feature of the target grid image as a to-be-tracked edge feature of the current frame image.

[0134] In the embodiments of the present application, when performing edge feature extraction on the target grid image in the current frame image, one or more pixel points with the maximum gradient in the target grid image can be directly taken as the to-be-tracked edge feature of the target grid image. Alternatively, edge detection can be performed on the current frame image to obtain edge features of the current frame image, and then N to-be-tracked pixel points in the target grid image that do not contain the edge features can be taken as the to-be-tracked edge feature of the target grid image; wherein the N to-be-tracked pixel points are the first N pixel points in the target grid image in descending order of gradient, and N is an integer greater than or equal to 1.

[0135] That is, when the edge feature is extracted, the edge feature of the image can be extracted first, and then it is judged whether the edge feature exists in the target grid image. If the edge feature exists, the edge feature is taken as the edge feature of the target grid image, or the pixel point with the maximum gradient in the edge feature is taken as the edge feature of the target grid image. If the edge feature does not exist, the first N pixel points of the target grid image sorted in descending order of the gradient are taken as the edge feature of the target grid image, instead of directly taking the first N pixel points of the target grid image sorted in descending order of the gradient as the edge feature of the target grid image.

[0136] In the embodiment of the present application, the current frame image is grid processed, and then the edge feature of the target grid image is extracted, so that each grid image in the current frame image contains one or more of the point feature, the line feature and the to-be-tracked edge feature.

[0137] In step 204, the optical flow tracking is performed on the to-be-tracked edge feature of the current frame image in the next frame image of the current frame image, so as to obtain the target edge feature corresponding to the to-be-tracked edge feature in the current frame image in the next frame image.

[0138] In the embodiment of the present application, the edge feature of the target region of the current frame image is extracted, so that when the optical flow tracking of the image is performed, the edge feature tracking of the region with less point feature and line feature can be realized, that is, the optical flow tracking of the image region with less rich texture and more blank regions can also be realized, thereby the visual positioning accuracy of the image can be improved.

[0139] Optionally, in order to reduce the calculation amount of the optical flow tracking algorithm, in some embodiments of the present application, during the optical flow tracking of the image, the edge feature extraction can be performed only in the target region, and the point feature extraction and the line feature extraction are not performed, and only the point feature extraction and the line feature extraction are performed in the region other than the target region in the current frame image, so as to obtain the to-be-tracked point feature and the to-be-tracked line feature.

[0140] In the embodiment of the present application, the to-be-tracked point feature can be extracted by using the Harris corner point feature algorithm, the Fast feature algorithm or the Super Point feature algorithm, which is not limited in the present application.

[0141] The to-be-tracked line feature can be extracted by using the Edline, the LSD (Line Segment Detector) and the LBD (Lind Band Descriptor) line segment extraction algorithm, which is not limited in the present application.

[0142] In some embodiments of the present application, after the point features and line features are extracted by the above algorithm, in order to reduce the amount of calculation, the interval distances between the extracted point features can be calculated, and part of the point features can be removed according to the interval distances, so that the interval distances between the point features are all greater than or equal to a first preset distance value, and the above-mentioned to-be-tracked point features are obtained.

[0143] In addition, part of the line features in the extracted line features, whose slope difference is less than a slope threshold value and whose interval distance is less than a second preset distance, can be removed, and only one line feature with the longest line length in the line features can be reserved as the above-mentioned to-be-tracked line feature, so as to reduce the amount of calculation of the optical flow tracking. Meanwhile, the number of to-be-tracked sampling points in the to-be-tracked line feature can be determined, and the to-be-tracked line feature with a number of to-be-tracked sampling points less than a number threshold value can be removed.

[0144] It should be noted that this is only an example, and it can be understood that in other embodiments of the present application, the target region can also be subjected to point feature extraction and line feature extraction at the same time to obtain to-be-tracked point features and to-be-tracked line features, and the present application does not limit this.

[0145] Correspondingly, the optical flow tracking method of the image provided by the present application also includes: respectively performing optical flow tracking on the to-be-tracked point features and the to-be-tracked line features of the current frame image on a next frame image of the current frame image, to obtain target point features and target line features corresponding to the to-be-tracked point features and the to-be-tracked line features in the current frame image on the next frame image.

[0146] In the embodiments of the present application, the edge features are extracted from the target region of the current frame image, and only the point features and the line features are extracted from other regions except the target region to obtain the to-be-tracked point features and the to-be-tracked line features, so that when the optical flow tracking of the image is performed, the point features, the line features and the edge features of the three different types of features can be used to perform the optical flow tracking of the image, so that each region of the image which lacks rich texture and has many blank regions contains features capable of optical flow tracking, and thus the visual positioning accuracy of the image can be improved.

[0147] Optionally, in some embodiments of the present application, in the process of performing optical flow tracking on the to-be-tracked line feature, the to-be-tracked line feature can be first subjected to feature point sampling to obtain to-be-tracked sampling points of the to-be-tracked line feature, and then the to-be-tracked sampling points can be subjected to optical flow tracking on the next frame image of the current frame image to obtain target sampling points corresponding to the to-be-tracked sampling points on the next frame image, and then the target line feature is obtained.

[0148] That is, in the process of performing optical flow tracking on the to-be-tracked line feature, the optical flow tracking is performed on the sampling points in the line feature.

[0149] Optionally, in some embodiments of the present application, in the process of implementing optical flow tracking on the to-be-tracked edge feature, optical flow tracking can be implemented on the to-be-tracked pixel points constituting the to-be-tracked edge feature.

[0150] Therefore, in some embodiments of the present application, in the process of implementing optical flow tracking on one or more features of the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature of the current frame image on the next frame image of the current frame image, it can be converted into a process of implementing optical flow tracking on one or more first feature points including the to-be-tracked sampling points on the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked pixel points on the to-be-tracked edge feature.

[0151] Specifically, in the process of implementing optical flow tracking on the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature respectively, Lucas-Kanade (LK), Gunnar Farneback and other optical flow tracking algorithms can be used to implement, and the present application does not make any limitation.

[0152] That is, Lucas-Kanade, Gunnar Farneback and other optical flow tracking algorithms can be used to implement optical flow tracking on the above-mentioned first feature points.

[0153] Wherein, when Lucas-Kanade optical flow tracking algorithm is used to implement optical flow tracking on the above-mentioned first feature points, first, the m-layer pyramid image corresponding to the next frame image of the current frame image can be constructed, then the coordinates of the first feature points of the current frame image on the current frame image can be taken as initial coordinates, the iterative tracking can be performed in the approximate direction of the light on the first layer pyramid image, and the tracking error can be calculated, when the tracking error is greater than the error threshold, the iterative tracking can be performed on the next layer pyramid image, and the tracking error can be recalculated until the tracking error is less than the error threshold, or after the iterative tracking of m layers is completed, the second feature points corresponding to the first feature points on the next frame image of the current frame image are obtained.

[0154] Optionally, after the second feature points are obtained, the second feature points on the next frame image of the current frame image whose tracking error is less than the error threshold can be tracked in the reverse direction on the current frame image, the reverse tracking error of the reverse optical flow tracking can be calculated, the second feature points whose reverse tracking error is greater than the reverse error threshold can be deleted, and the optical flow tracking on the first feature points can be completed.

[0155] Wherein, the calculation of the reverse tracking error of the reverse optical flow tracking can include the calculation of the coordinate difference between the feature points obtained by the reverse tracking and the first feature points, when the coordinate difference is greater than the reverse error threshold, the second feature points corresponding to the first feature points are deleted. Or, when it is determined that the feature points obtained by the reverse tracking are located outside the current frame image, the second feature points corresponding to the first feature points are deleted.

[0156] In the embodiments of the present application, the second feature point includes one or more of the target point feature, the target sampling point on the target line feature, and the target pixel point on the target edge feature.

[0157] That is, the optical flow tracking manner of the three features, i.e., the to-be-tracked point feature, the to-be-tracked line feature, and the to-be-tracked edge feature, can be the same or different, and the present application does not limit this.

[0158] However, since the gradient changes of the sampling points or pixel points on the to-be-tracked line feature and the to-be-tracked edge feature are similar, in the process of optical flow tracking, the problem of edge drift can occur, which reduces the accuracy of optical flow tracking. Similarly, when the to-be-tracked point feature is also located on a line or edge, the problem of edge drift can also occur.

[0159] For example, as shown in FIG. 1, the first feature point A is located on a line, and the to-be-tracked point feature is the first feature point A. In the process of optical flow tracking, the first feature point A is tracked on the next frame image. If the optical flow tracking is performed along the luminance approximate direction n2, the problem of edge drift can occur, and if the optical flow tracking is performed along the non-luminance approximate direction n1, the problem of edge drift can be avoided. Figure 3 The non-luminance approximate direction refers to a direction with an included angle of 0° with the luminance approximate direction and a direction with an included angle of 180°, such as a gradient change direction. In addition, the luminance approximate direction can refer to a direction in which the gray value change of the pixel block corresponding to the first feature point is less than a change threshold, which can be estimated based on the previous K frame images of the next frame image. K is an integer greater than or equal to 1.

[0160] Therefore, in order to avoid the problem of edge drift in the process of optical flow tracking of the to-be-tracked point feature, the to-be-tracked line feature, and the to-be-tracked edge feature, and to reduce the accuracy of optical flow tracking, in some embodiments of the present application, the optical flow tracking along the non-luminance approximate direction is added on the basis of the Lucas-Kanade optical flow tracking algorithm.

[0161] Specifically, the next frame of image can be down-sampled to construct m-layer pyramid images corresponding to the next frame of image of the current frame of image, optical flow tracking is performed on i-layer images in the m-layer pyramid images according to the non-photometric approximation direction of the to-be-tracked target feature, and optical flow tracking is performed on j-layer images in the m-layer pyramid images according to the photometric approximation direction of the to-be-tracked target feature, to obtain target features corresponding to the to-be-tracked target feature on the next frame of image, the to-be-tracked target feature including one or more of to-be-tracked point features, to-be-tracked line features and to-be-tracked edge features; the target feature including one or more of target point features, target line features and target edge features; wherein m = i + j; i and j are both integers greater than or equal to 1. That is, for any one or more of the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features, the optical flow tracking in the non-photometric approximation direction is added on the basis of the Lucas-Kanade optical flow tracking algorithm.

[0162] For example, when m = 4, the optical flow tracking can be performed on the first two layers of the 4-layer pyramid images according to the non-photometric approximation direction of the to-be-tracked target feature, and the optical flow tracking can be performed on the last two layers of the 4-layer pyramid images according to the photometric approximation direction of the to-be-tracked target feature; or the optical flow tracking can be performed on the first and third layers of the 4-layer pyramid images according to the non-photometric approximation direction of the to-be-tracked target feature, and the optical flow tracking can be performed on the second and fourth layers of the 4-layer pyramid images according to the photometric approximation direction of the to-be-tracked target feature; or the optical flow tracking can be performed on the first two layers of the 4-layer pyramid images according to the photometric approximation direction of the to-be-tracked target feature, and the optical flow tracking can be performed on the last two layers of the 4-layer pyramid images according to the non-photometric approximation direction of the to-be-tracked target feature, and the like. The application does not limit which layer or layers are selected to perform the optical flow tracking according to the non-photometric approximation direction of the to-be-tracked target feature and the optical flow tracking according to the photometric approximation direction of the to-be-tracked target feature, and the selection can be set according to the actual application scenario.

[0163] For example, i can be set to be greater than j or i can be set to be less than or equal to j according to the actual application scenario.

[0164] In the embodiments of the application, the optical flow tracking in the non-photometric approximation direction is added in the process of performing the optical flow tracking on the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features, so that the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features are subjected to the optical flow tracking in the non-photometric approximation direction and the optical flow tracking in the photometric approximation direction, the problem of edge drift is avoided, and the accuracy of the optical flow tracking is improved.

[0165] In actual applications, when performing image optical flow tracking, a video of a moving object is generally shot, and optical flow tracking is performed based on each continuous frame of image in the video.

[0166] That is, when performing image optical flow tracking, generally, optical flow tracking is performed on continuous multiple frames of images, and thus the current frame of image can be a first frame of image in the continuous multiple frames of images to be tracked. After completing optical flow tracking from the first frame of image to a second frame of image, optical flow tracking from the second frame of image to a third frame of image is performed, and so on, until optical flow tracking of each frame of image in the continuous multiple frames of images to be tracked is completed.

[0167] Therefore, in some embodiments of the present application, after step 204, the next frame of image can be taken as a new current frame of image, and the target point feature, the target line feature and the target edge feature can be taken as new to-be-tracked point feature, new to-be-tracked line feature and new to-be-tracked edge feature, and optical flow tracking is continued until optical flow tracking of each frame of image in the continuous multiple frames of images to be tracked is completed.

[0168] In addition, since each time optical flow tracking is performed, it cannot be guaranteed that each tracking result is accurate, and thus after obtaining the target point feature, the target line feature and the target edge feature corresponding to the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature in the current frame of image on the next frame of image, the target point feature, the target line feature and the target edge feature that do not meet the tracking error can be deleted, that is, the target point feature, the target line feature and the target edge feature that fail in tracking are deleted, so as to avoid entering optical flow tracking in the next round, thereby reducing invalid optical flow tracking.

[0169] In the process of deleting the target line feature and the target edge feature that fail in tracking, the target sampling point on the target line feature and the target pixel point on the target edge feature can be deleted.

[0170] Specifically, in some embodiments of the present application, after obtaining the target line feature, linear fitting is performed on the target sampling point in the target line feature to obtain a linear equation, and then a first distance from the target sampling point in the target line feature to the fitted straight line is calculated according to the linear equation. The target sampling point with a first distance greater than a first distance threshold is taken as the target sampling point that fails in tracking, and is deleted.

[0171] Since the number of target sampling points on the target line feature is less than the number threshold when the target line feature is tracked, it is likely that the target line feature is incorrect. Therefore, in an embodiment, before performing linear fitting on the target sampling points in the target line feature, the number of target sampling points corresponding to the target line feature is determined, and the target line feature with the number of target sampling points less than the number threshold is deleted, i.e., the target line feature with the number of target sampling points less than the number threshold is regarded as a tracking failed line feature and is deleted. After the target sampling points with the first distance greater than the first distance threshold are deleted, the number of target sampling points corresponding to the target line feature can also be determined again, and the target line feature with the number of target sampling points less than the number threshold is deleted again.

[0172] The above describes the implementation of deleting the tracking failed target line feature. The following describes the deletion methods of the three types of feature points, i.e., the target sampling points of the tracking failed target point feature, the target line feature, and the target pixel points of the target edge feature.

[0173] In some embodiments of the present application, after obtaining the target point feature, the target line feature, and the target edge feature corresponding to the to-be-tracked point feature, the to-be-tracked line feature, and the to-be-tracked edge feature in the current frame image respectively on the next frame image, the tracking failed target point feature, the target sampling points of the target line feature, and the target pixel points of the target edge feature can be deleted by using the following steps 501 to 504, which are described in detail as follows:

[0174] Step 501: Calculate the fundamental matrix between the current frame image and the next frame image.

[0175] In the embodiments of the present application, the fundamental matrix F is the mapping of the first feature point p on the current frame image to the epipolar line L on the next frame image. That is, L=F*p.

[0176] In the embodiments of the present application, the fundamental matrix between the current frame image and the next frame image can be calculated by the Ransac algorithm or other clustering algorithms, which are not limited in the present application.

[0177] Step 502: Calculate the epipolar line of the first feature point on the next frame image based on the fundamental matrix.

[0178] In the embodiments of the present application, the above-mentioned first feature point includes the to-be-tracked sampling point on the to-be-tracked line feature and the to-be-tracked pixel point on the to-be-tracked edge feature.

[0179] Step 503: Calculate the second distance of the second feature point to the epipolar line corresponding thereto.

[0180] In the embodiments of the present application, the second feature point described above is a feature point corresponding to the first feature point on the next frame of image, which can include a target point feature, a target sampling point on a target line feature, and a target pixel point on a target edge feature.

[0181] In step 504, the second feature point with the second distance greater than the second distance threshold is deleted.

[0182] In the embodiments of the present application, since the second feature point should be located on the epipolar line corresponding thereto, when the second distance corresponding to the second feature point is greater than the second distance threshold, it indicates that the second feature point is a feature point of tracking failure, and thus can be deleted.

[0183] It should be noted that the first distance threshold and the second distance threshold described above can be obtained according to actual application scenarios or practical experience, and the present application does not make any limitation in this regard. For example, the second distance threshold described above can be a distance of three pixels. That is, if the interval between the second feature point and the epipolar line corresponding thereto exceeds three pixel points, the second feature point is deleted.

[0184] Optionally, in some embodiments of the present application, when the second feature point described above is a target sampling point on a target line feature and a target pixel point on a target edge feature, before step 503 described above, it can be determined whether the angle between the direction of the epipolar line corresponding to the second feature point and the edge direction of the second feature point is less than an angle threshold. If the angle between the direction of the epipolar line corresponding to the second feature point and the edge direction of the second feature point is less than the angle threshold, the second feature point is retained.

[0185] Specifically, since the angle between the direction of the epipolar line corresponding to the second feature point and the edge direction of the second feature point is less than the angle threshold, it indicates that the second feature point is located on the epipolar line corresponding thereto. Therefore, the second distance of the second feature point to the epipolar line corresponding thereto does not need to be calculated, and it is also not necessary to determine whether the second feature point needs to be deleted according to the second distance.

[0186] The edge direction of the second feature point refers to a direction perpendicular to the gradient direction of the second feature point. The angle threshold described above can be obtained according to actual application scenarios or practical experience, and the present application does not make any limitation in this regard. For example, the angle threshold described above can range from 0° to 10°.

[0187] In the embodiments of the present application, after step 504, the number of target sampling points corresponding to the target line feature can be re-determined; and the target line feature with the number of target sampling points less than a number threshold is determined as a line feature of tracking failure, and thus is deleted.

[0188] After obtaining the final remaining target line feature, in some embodiments of the present application, the intersection between the epipolar line corresponding to the target sampling point and the fitting straight line corresponding to the target sampling point can be taken as the sampling point obtained by correcting the target sampling point.

[0189] Specifically, since the target sampling point should be located on the epipolar line corresponding to the target sampling point and the fitting straight line corresponding to the target sampling point, the coordinates of the target sampling point can be corrected by calculating the intersection between the epipolar line corresponding to the target sampling point and the fitting straight line corresponding to the target sampling point, so as to improve the optical flow tracking accuracy of the line feature.

[0190] In the embodiments of the present application, edge feature extraction is performed on the target region of the current frame image, and only point feature extraction and line feature extraction are performed on other regions except the target region, so as to obtain the to-be-tracked point feature and the to-be-tracked line feature. When performing optical flow tracking of the image, the point feature, the line feature and the edge feature of the three different types of features can be used for optical flow tracking of the image. Therefore, each region of the image which lacks rich texture and has many blank regions contains features that can be tracked by optical flow, so that the visual positioning accuracy of the image can be improved. In addition, the optical flow tracking along the non-photometric approximation direction is added in the process of tracking the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature, so as to improve the optical flow tracking algorithm, avoid the problem of edge drift, improve the accuracy of optical flow tracking, and calculate the epipolar line to remove the second feature point that fails to track and correct the coordinates of the target sampling point, so as to greatly improve the optical flow tracking accuracy of the image and improve the visual positioning accuracy of the terminal device such as a robot.

[0191] Figure 4 A schematic flowchart of an optical flow tracking method of an image provided by the embodiments of the present application is shown. The optical flow tracking method can be performed by the optical flow tracking device of the terminal such as a robot, a mobile phone, a computer and the like, and specifically includes the following steps 41 to 44.

[0192] Step 41, obtaining a current frame image which needs to be tracked by optical flow and a next frame image of the current frame image;

[0193] Step 42, identifying the point feature and the line feature in the current frame image, and taking a region with a preset area and a number of point features less than or equal to a first number threshold and / or a number of line features less than or equal to a second number threshold as a target region;

[0194] Step 43, performing edge feature extraction on the target region of the current frame image to obtain the to-be-tracked edge feature of the current frame image;

[0195] Step 44, performing optical flow tracking on the next frame image of the current frame image on the to-be-tracked edge features of the current frame image to obtain target edge features corresponding to the to-be-tracked edge features in the current frame image on the next frame image.

[0196] Optionally, in some embodiments of the present application, a preset area with a number of point features less than or equal to a first number threshold and / or a number of line features less than or equal to a second number threshold in the current frame image is taken as a target area, and edge feature extraction is performed on the target area of the current frame image to obtain the to-be-tracked edge features of the current frame image, including:

[0197] performing grid processing on the current frame image to obtain each grid image in the current frame image;

[0198] taking a grid image with a number of point features less than or equal to a first number threshold and / or a number of line features less than or equal to a second number threshold in each grid image as a target grid image;

[0199] performing edge feature extraction on the target grid image in the current frame image to obtain to-be-tracked edge features of the target grid image, and taking the to-be-tracked edge features of the target grid image as the to-be-tracked edge features of the current frame image.

[0200] Optionally, in some embodiments of the present application, before performing edge feature extraction on the target grid image in the current frame image, including:

[0201] performing edge detection on the current frame image to obtain edge features of the current frame image;

[0202] performing edge feature extraction on the target grid image in the current frame image to obtain the to-be-tracked edge features of the current frame image, including:

[0203] if the target grid image does not contain edge features, taking N to-be-tracked pixel points in the target grid image as the to-be-tracked edge features of the target grid image; the N to-be-tracked pixel points are the first N pixel points in the target grid image after sorting the pixel points in the target grid image in descending order of gradient, and N is an integer greater than or equal to 1.

[0204] Optionally, in some embodiments of the present application, the optical flow tracking method of the image further includes:

[0205] performing point feature extraction and line feature extraction on other areas of the current frame image except the target area to obtain to-be-tracked point features and to-be-tracked line features;

[0206] Respectively performing optical flow tracking on the to-be-tracked point features and the to-be-tracked line features of the current frame image on a next frame image of the current frame image, to obtain target point features and target line features corresponding to the to-be-tracked point features and the to-be-tracked line features in the current frame image respectively on the next frame image.

[0207] Optionally, in some embodiments of the present application, respectively performing optical flow tracking on the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features of the current frame image on a next frame image of the current frame image, to obtain target point features, target line features and target edge features corresponding to the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features in the current frame image respectively on the next frame image, comprises:

[0208] Constructing m-layer pyramid images corresponding to the next frame image of the current frame image, performing optical flow tracking on i-layer images in the m-layer pyramid images according to non-photometric approximation directions of to-be-tracked target features, and performing optical flow tracking on j-layer images in the m-layer pyramid images according to photometric approximation directions of the to-be-tracked target features, to obtain target features corresponding to the to-be-tracked target features on the next frame image, wherein the to-be-tracked target features comprise one or more of the to-be-tracked point features, the to-be-tracked line features and the to-be-tracked edge features; the target features comprise one or more of the target point features, the target line features and the target edge features; wherein m = i + j; i and j are both integers greater than or equal to 1.

[0209] Optionally, in some embodiments of the present application, performing optical flow tracking on the to-be-tracked line features of the current frame image on a next frame image of the current frame image to obtain target line features, comprises:

[0210] Performing feature point sampling on the to-be-tracked line features to obtain to-be-tracked sampling points of the to-be-tracked line features;

[0211] Performing optical flow tracking on the to-be-tracked sampling points on a next frame image of the current frame image to obtain target sampling points corresponding to the to-be-tracked sampling points on the next frame image, and obtaining the target line features according to the target sampling points.

[0212] Optionally, in some embodiments of the present application, after obtaining the target line features, comprising:

[0213] Performing straight line fitting on the target sampling points in the target line features to obtain a straight line equation;

[0214] Calculating first distances from the target sampling points in the target line features to the fitted straight line according to the straight line equation;

[0215] Deleting the target sampling points with the first distances greater than a first distance threshold.

[0216] Optionally, in some embodiments of the present application, after obtaining the target point feature, the target line feature and the target edge feature corresponding to the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature in the current frame image respectively on the next frame image, the method comprises:

[0217] calculating a fundamental matrix between the current frame image and the next frame image;

[0218] calculating an epipolar line of a first feature point on the next frame image based on the fundamental matrix; the first feature point comprises the to-be-tracked point feature, a to-be-tracked sampling point on the to-be-tracked line feature and a to-be-tracked pixel point on the to-be-tracked edge feature;

[0219] calculating a second distance of a second feature point to the epipolar line corresponding to the second feature point; the second feature point is a feature point on the next frame image corresponding to the first feature point;

[0220] deleting the second feature point with the second distance greater than a second distance threshold.

[0221] Optionally, in some embodiments of the present application, the optical flow tracking method further comprises:

[0222] determining a number of target sampling points corresponding to the target line feature;

[0223] deleting the target line feature with the number of target sampling points less than a number threshold.

[0224] Optionally, in some embodiments of the present application, after obtaining the target line feature, the method further comprises:

[0225] taking an intersection between an epipolar line corresponding to the target sampling point and a fitting straight line corresponding to the target sampling point as a sampling point obtained by correcting the target sampling point.

[0226] Optionally, in some embodiments of the present application, the current frame image is a first frame image in a plurality of continuous to-be-tracked images, and after obtaining the target point feature, the target line feature and the target edge feature corresponding to the to-be-tracked point feature, the to-be-tracked line feature and the to-be-tracked edge feature in the current frame image respectively on the next frame image, the method comprises:

[0227] taking the next frame image as a new current frame image, taking the target point feature, the target line feature and the target edge feature as a new to-be-tracked point feature, a new to-be-tracked line feature and a new to-be-tracked edge feature, and continuing the optical flow tracking until the optical flow tracking of each frame image in the plurality of continuous to-be-tracked images is completed.

[0228] It should be noted that, for the convenience and brevity of description, the specific working process of the image optical flow tracking method described above can refer to the description of the terminal above, and will not be repeated here. For the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions. In some embodiments of the present application, certain steps can be performed in other orders.

[0229] Figure 5 A structure diagram of an image optical flow tracking device 5 provided by an embodiment of the present application is shown, which includes an acquisition unit 51, an identification unit 52, an extraction unit 53, and a tracking unit 54.

[0230] The acquisition unit 51 is configured to acquire a current frame image and a next frame image of the current frame image which need to be tracked by optical flow.

[0231] The identification unit 52 is configured to identify point features and line features in the current frame image, and take a region of a preset area in which the number of point features is less than or equal to a first number threshold and / or the number of line features is less than or equal to a second number threshold as a target region.

[0232] The extraction unit 53 is configured to perform edge feature extraction on the target region of the current frame image to obtain to-be-tracked edge features of the current frame image.

[0233] The tracking unit 54 is configured to perform optical flow tracking on the to-be-tracked edge features of the current frame image on the next frame image of the current frame image to obtain target edge features corresponding to the to-be-tracked edge features in the current frame image on the next frame image.

[0234] It should be noted that, for the convenience and brevity of description, the specific working process of the image optical flow tracking device 5 described above can refer to the corresponding process of the image optical flow tracking method above, and will not be repeated here.

[0235] Exemplarily, the embodiment of the present application further provides a chip, which includes a processor, and the processor is configured to read and execute a computer program stored in a memory to implement the steps of the image optical flow tracking method shown above. Figure 4

[0236] Exemplarily, the embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the image optical flow tracking method shown above. Figure 4

[0237] Exemplarily, the embodiment of the present application further provides a computer product, which stores computer instructions, and the computer instructions are executed to implement the steps of the image optical flow tracking method shown above.​​Figure 4 Steps of the optical flow tracking method of the illustrated image.

[0238] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0239] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0240] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0241] In the embodiments provided in the present application, it should be understood that the disclosed terminal and method can be implemented in other ways. For example, the terminal embodiments described above are only schematic. For example, the division of modules or units is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0242] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0243] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0244] If the integrated module / unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be instructed by a computer program to related hardware to complete. The computer program can be stored in a computer readable storage medium, and the computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electric carrier wave signal and telecommunication signal.

[0245] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the following steps are implemented: Obtaining a current frame image for which optical flow tracking is required, and a next frame image of the current frame image; Identifying point features and line features in the current frame image, and taking an area of ​​a preset area in which the number of the point features in the current frame image is less than or equal to a first number threshold, and / or the number of the line features is less than or equal to a second number threshold as a target area; performing edge feature extraction on the target area of ​​the current frame image to obtain edge features to be tracked of the current frame image; Optical flow tracking is performed on the edge feature to be tracked in the current frame image on a next frame image of the current frame image to obtain a target edge feature on the next frame image corresponding to the edge feature to be tracked in the current frame image.

2. The terminal according to claim 1, wherein The processor executes the computer program to implement the steps of taking an area of ​​a preset area in which the number of point features in the current frame image is less than or equal to a first number threshold, and / or the number of line features is less than or equal to a second number threshold as a target area, and performing edge feature extraction on the target area of ​​the current frame image to obtain the edge features to be tracked of the current frame image, specifically including the following steps: Performing gridding processing on the current frame image to obtain each grid image in the current frame image; taking a grid image in which the number of the point features in each grid image is less than or equal to the first number threshold, and / or the number of the line features is less than or equal to the second number threshold as a target grid image; Edge features of the target grid image in the current frame image are extracted to obtain edge features to be tracked of the target grid image, and the edge features to be tracked of the target grid image are used as edge features to be tracked of the current frame image.

3. The terminal according to claim 2, characterized in that Before the processor executes the computer program to extract edge features from the target grid image in the current frame image, the processor further includes implementing the following steps: Performing edge detection on the current frame image to obtain edge features of the current frame image; The processor executes the computer program to extract edge features of the target grid image in the current frame image to obtain edge features to be tracked of the current frame image, including the following steps: If the target grid image does not contain the edge feature, the N pixel points to be tracked in the target grid image are used as the edge features to be tracked of the target grid image; the N pixel points to be tracked are the first N pixel points in the target grid image after the pixel points are sorted from large to small according to the gradient, and N is an integer greater than or equal to 1.

4. The terminal according to claim 3, wherein When the processor executes the computer program, the processor further includes implementing the following steps: Performing point feature extraction and line feature extraction on other areas of the current frame image except the target area to obtain point features and line features to be tracked; Optical flow tracking is performed on the point features to be tracked and the line features to be tracked of the current frame image on the next frame image of the current frame image, respectively, to obtain target point features and target line features on the next frame image that correspond to the point features to be tracked and the line features to be tracked in the current frame image, respectively.

5. The terminal according to claim 4, wherein: The processor executes the computer program to perform optical flow tracking on a frame image next to the current frame image for the point features to be tracked, the line features to be tracked, and the edge features to be tracked of the current frame image, respectively, and obtain target point features, target line features, and target edge features on the next frame image that correspond to the point features to be tracked, the line features to be tracked, and the edge features to be tracked in the current frame image, respectively, including the following steps: An m-layer pyramid image corresponding to a next frame image of the current frame image is constructed, and optical flow tracking is performed on an i-layer image in the m-layer pyramid image according to a non-photometric approximate direction of a target feature to be tracked. Optical flow tracking is performed on a j-layer image in the m-layer pyramid image according to a photometric approximate direction of the target feature to be tracked, thereby obtaining a target feature corresponding to the target feature to be tracked on the next frame image, wherein the target feature to be tracked includes one or more of the point feature to be tracked, the line feature to be tracked, and the edge feature to be tracked; and the target feature includes one or more of the target point feature, the target line feature, and the target edge feature. Wherein, m=i+j; and i and j are both integers greater than or equal to 1. The terminal according to claim 4 , wherein: The processor executes the computer program to perform optical flow tracking on the line feature to be tracked of the current frame image on a frame image next to the current frame image, and obtains the target line feature, including the following steps: Sampling feature points of the line feature to be tracked to obtain sampling points to be tracked of the line feature to be tracked; Optical flow tracking is performed on the sampling point to be tracked on a next frame image of the current frame image to obtain a target sampling point on the next frame image corresponding to the sampling point to be tracked, and the target line feature is obtained according to the target sampling point.

7. The terminal according to claim 6, wherein: After the processor executes the computer program to obtain the target line feature, the processor further implements the following steps: Performing straight line fitting on the target sampling points in the target line feature to obtain a straight line equation; Calculate a first distance from a target sampling point in the target line feature to a fitting straight line according to the straight line equation; Delete target sampling points whose first distance is greater than a first distance threshold.

8. The terminal according to claim 6, wherein: After the processor executes the computer program to obtain target point features, target line features, and target edge features on the next frame image that correspond to the point features to be tracked, the line features to be tracked, and the edge features to be tracked in the current frame image, the processor further implements the following steps: Calculating a basic matrix between the current frame image and the next frame image; Obtaining, based on the fundamental matrix, an epipolar line of a first feature point on the current frame image on the next frame image; the first feature point includes the point feature to be tracked, a sampling point to be tracked on the line feature to be tracked, and a pixel point to be tracked on the edge feature to be tracked; Calculating a second distance from a second feature point to its corresponding epipolar line; the second feature point is a feature point on the next frame image corresponding to the first feature point; The second feature point whose second distance is greater than the second distance threshold is deleted.

9. The terminal according to any one of claims 1 to 8, wherein: When the processor executes the computer program, the processor further implements the following steps: Determining the number of target sampling points corresponding to the target line feature; Delete the target line feature whose number of target sampling points is less than the number threshold.

10. The terminal according to claim 9, wherein: After the processor executes the computer program to obtain the target line feature, the processor further implements the following steps: The intersection point between the epipolar line corresponding to the target sampling point and the fitting straight line corresponding to the target sampling point is used as the sampling point obtained by correcting the target sampling point.

11. A method for optical flow tracing of an image, characterized in that: include: Obtaining a current frame image for which optical flow tracking is required, and a next frame image of the current frame image; Identifying point features and line features in the current frame image, and taking an area of ​​a preset area in which the number of the point features in the current frame image is less than or equal to a first number threshold, and / or the number of the line features is less than or equal to a second number threshold as a target area; performing edge feature extraction on the target area of ​​the current frame image to obtain edge features to be tracked of the current frame image; Optical flow tracking is performed on the edge feature to be tracked in the current frame image on a next frame image of the current frame image to obtain a target edge feature on the next frame image corresponding to the edge feature to be tracked in the current frame image.

12. An optical flow tracking device for an image, characterized in that: include: An acquisition unit, configured to acquire a current frame image on which optical flow tracking is required, and a next frame image of the current frame image; an identification unit, configured to identify point features and line features in the current frame image, and to select an area of ​​a preset area in which the number of the point features in the current frame image is less than or equal to a first number threshold, and / or the number of the line features is less than or equal to a second number threshold as a target area; an extraction unit, configured to extract edge features from the target area of ​​the current frame image to obtain edge features to be tracked of the current frame image; The tracking unit is used to perform optical flow tracking on the edge feature to be tracked in the current frame image on the next frame image of the current frame image, and obtain the target edge feature on the next frame image corresponding to the edge feature to be tracked in the current frame image.

13. A chip comprising a processor, wherein the processor is configured to read and execute a computer program stored in a memory to implement the steps of the method according to claim 11.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 11 are implemented.

Images