Tracking methods and movable platforms

Abstract

A tracking method and a movable platform, the method comprising: obtaining an estimated position of a preselected feature point in the first image according to the relative position of the first camera when shooting the first image and the second image; the first camera is the left eye or the right eye of a binocular camera (12, 401), and the second image is the previous frame image of the first image (S201, S301); using the estimated position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the first image to obtain a first tracking result (S202, S302); using the position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the third image; the third image is shot by the second camera of the binocular camera (12, 401), and the shooting time of the first image and the third image is the same (S203, S303). The tracking method can improve the accuracy of the tracking result.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a tracking method and a movable platform. Background Art

[0002] At present, object tracking is involved in military, civilian and other fields. For example, a camera device can be mounted on a mobile platform, and an object in an image can be tracked through multiple video frame images taken by the camera device.

[0003] The commonly used tracking algorithms include KLT algorithm, which is very dependent on the initial position of tracking. If the initial position is far away from the actual position of the tracking target, the tracking result will be inaccurate or even fail. At present, it is usually assumed that the change between frames is small and the position of the tracking target in the previous frame image is directly used as the initial position for tracking the position of the target in the current frame image. However, when the mobile platform equipped with the camera device moves at a high speed, the change between frames is relatively large. Therefore, when the position of the tracking target in the previous frame image is used as the initial position for tracking the position of the target in the current frame image, the tracking result will be inaccurate. Summary of the invention

[0004] The embodiments of the present application provide a tracking method and a movable platform to overcome the technical problem of inaccurate tracking results in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a tracking method, comprising: obtaining an estimated position of a preselected feature point in the first image according to the relative posture of the first camera when the first image and the second image are captured; the first camera is the left eye or the right eye of a binocular camera, and the second image is an image of the previous frame of the first image; taking the estimated position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the first image to obtain a first tracking result; taking the position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in a third image to obtain a second tracking result; the third image is captured by the second camera of the binocular camera, and the first image and the third image are captured at the same time.

[0006] In a possible design, the binocular camera is mounted on a mobile platform; and obtaining the estimated position of a preselected feature point in the first image based on the relative pose when the first camera takes the first image and the second image includes: obtaining the relative pose based on inertial measurement unit IMU data of the mobile platform and the wheel speed of the mobile platform; and obtaining the estimated position of the preselected feature point in the first image based on the relative pose.

[0007] In a possible design, obtaining the estimated position of the preselected feature point in the first image according to the relative posture includes: obtaining the first three-dimensional coordinates of the preselected feature point in the first camera coordinate system when the second image is captured according to the position of the preselected feature point tracked in the second image and in the fourth image; the fourth image is captured by the second camera, and the second image and the fourth image are captured at the same time; according to the relative posture, converting the first three-dimensional coordinates to the second three-dimensional coordinates of the preselected feature point in the first camera coordinate system when the first image is captured; according to the second three-dimensional coordinates and the intrinsic parameter matrix of the first camera, obtaining the estimated position of the preselected feature point in the first image.

[0008] In one possible design, the estimated position of the preselected feature point in the first image is used as the initial position, and the position of the preselected feature point in the first image is tracked to obtain a first tracking result, including: using the estimated position of the preselected feature point in the first image as the initial position, tracking the preselected feature point from a reference image to the first image, and obtaining the tracked position of the preselected feature point in the first image, wherein the reference image is an image in which the preselected feature point is first detected.

[0009] In a possible design, the position of the preselected feature point in the first image is taken as the initial position, and the position of the preselected feature point in the third image is tracked to obtain a second tracking result, including: taking the position of the preselected feature point in the first image as the initial position, tracking the preselected feature point from the first image to the third image, and obtaining the tracked position of the preselected feature point in the third image.

[0010] In a possible design, the position of the preselected feature point in the first image is taken as the initial position, and the position of the preselected feature point in the third image is tracked to obtain a second tracking result, including: taking the position of the preselected feature point in the first image as the initial position, tracking the preselected feature point from the first image to the third image, and obtaining a tracking result that the position of the preselected feature point in the third image is not tracked; obtaining the similarity between each position in the horizontal direction of the first side of the position of the preselected feature point tracked in the first image and the tracked position, and taking the position with the largest similarity as the starting position, tracking the preselected feature point from the first image to the third image to obtain a second tracking result, and the second tracking result is the position of the preselected feature point in the third image that is tracked or the position of the preselected feature point in the third image that is not tracked.

[0011] In a possible design, the pre-selected feature point is a feature point among a plurality of feature points to be tracked in the first image that is tracked to a position in the first image and satisfies at least one of the following conditions: (1) the Euclidean distance between the tracked position and the estimated position of the pre-selected feature point in the first image is less than or equal to a first preset distance; (2) when the perspective point PnP problem is solved using the random sampling consistent RANSAC algorithm, the pre-selected feature point satisfies the geometric constraint; (3) the Euclidean distance between the actual position of the pre-selected feature point in the reference image and the similarity of the first reverse tracking position is less than or equal to a second preset distance, and the first reverse tracking position is a first reference position of the pre-selected feature point in the reference image. The position is the starting position, and the pre-selected feature point is tracked from the first image to the reference image; the first reference position is the actual position and any position in the adjacent area of ​​the actual position; (4) the similarity between the first image block and the second image block is greater than or equal to a first preset value, the first image block is an image block composed of the actual position and the adjacent area of ​​the actual position, and the second image block is an image block composed of the position where the pre-selected feature point is tracked in the first image and the adjacent area of ​​the tracked position; (5) when the eigenmatrix between the first image and the second image is obtained according to the RANSAC algorithm, the pre-selected feature point satisfies the epipolar constraint.

[0012] In a possible design, the method further includes: after obtaining the second tracking result, determining a target feature point from a feature point set, wherein the feature point set includes all pre-selected feature points.

[0013] In a possible design, the target feature point is a feature point in the feature point set that is tracked to a position in the third image and satisfies at least one of the following conditions: (1) the Euclidean distance between the position of the target feature point tracked in the first image and the first back-projected position is less than or equal to a third preset distance, and the Euclidean distance between the position of the target feature point tracked in the third image and the second back-projected position is less than or equal to a fourth preset distance; wherein the first back-projected position is obtained by back-projecting the third three-dimensional coordinates into the first image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera after obtaining the third three-dimensional coordinates of the target feature point in the first camera coordinate system according to the position of the target feature point tracked in the first image and the third image; and the second back-projected position is obtained by back-projecting the fourth three-dimensional coordinates into the first image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera after obtaining the fourth three-dimensional coordinates of the target feature point in the second camera coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera. (2) the Euclidean distance between the position where the target feature point is tracked in the first image and the second reverse tracking position is less than or equal to a fifth preset distance, the second reverse tracking position is obtained by tracking the target feature point from the third image to the first image with the first reference position of the target feature point in the third image as the starting position; the first reference position is the position where the target feature point is tracked in the third image and any position in the adjacent area of ​​the tracked position; (3) the similarity between the third image block and the fourth image block is greater than or equal to a second preset value, the third image block is an image block composed of the position where the target feature point is tracked in the first image and the adjacent area of ​​the tracked position, and the fourth image block is an image block composed of the position where the target feature point is tracked in the third image and the adjacent area of ​​the tracked position; (4) when the eigenmatrix of the binocular camera is obtained according to the RANSAC algorithm, the target feature point satisfies the epipolar constraint.

[0014] In a possible design, it also includes: dividing the first image into multiple identical rectangular areas, uniformly detecting first newly added feature points from each rectangular area, and determining a first number of second newly added feature points from the first newly added feature points, so that the sum of the first number and the number of pre-selected feature points is greater than or equal to a first preset number; the feature point set also includes all of the second newly added feature points; saving the pixel blocks where the second newly added feature points in the first image are located and the pixel blocks in the adjacent areas of the pixel blocks.

[0015] In a possible design, the first image is divided into multiple identical rectangular areas, the first new feature points are evenly detected from each rectangular area, and before the second new feature points are determined from the first new feature points, it also includes: determining that the average disparity between the first image and the most recently determined key frame image is greater than or equal to a preset disparity; or, determining that the total number of the pre-selected feature points is less than or equal to a second preset number.

[0016] In a possible design, after determining the first number of second newly added feature points from the first newly added feature points, it also includes: taking the position of the second newly added feature points in the first image as the initial position, tracking the second newly added feature points from the first image to the third image to obtain a third tracking result.

[0017] In a second aspect, an embodiment of the present application provides a movable platform, comprising: a binocular camera, which is mounted on the movable platform; a processor, which is communicatively connected to the binocular camera and is used to perform the following operations: obtaining an estimated position of a preselected feature point in the first image according to the relative posture of the first camera when shooting a first image and a second image; the first camera is the left eye or the right eye of the binocular camera, and the second image is the previous frame image of the first image; taking the estimated position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the first image, and obtaining a first tracking result; taking the position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in a third image, and obtaining a second tracking result; the third image is shot by the second camera of the binocular camera, and the shooting time of the first image and the third image is the same.

[0018] In a possible design, when the processor is used to execute the operation of obtaining the estimated position of a preselected feature point in the first image according to the relative pose when the first image and the second image are taken by the first camera, the processor is specifically used to perform the following operations: obtaining the relative pose according to the inertial measurement unit IMU data of the mobile platform and the wheel speed of the mobile platform; obtaining the estimated position of the preselected feature point in the first image according to the relative pose.

[0019] In a possible design, when the processor is used to execute the operation of obtaining the estimated position of the preselected feature point in the first image according to the relative posture, it is specifically used to perform the following operations: according to the position of the preselected feature point in the second image and the tracked position in the fourth image, obtain the first three-dimensional coordinates of the preselected feature point in the first camera coordinate system when the second image is taken; the fourth image is taken by the second camera, and the shooting time of the second image and the fourth image is the same; according to the relative posture, convert the first three-dimensional coordinates to the second three-dimensional coordinates of the preselected feature point in the first camera coordinate system when the first image is taken; according to the second three-dimensional coordinates and the intrinsic parameter matrix of the first camera, obtain the estimated position of the preselected feature point in the first image.

[0020] In one possible design, when the processor is used to perform an operation of tracking the position of the preselected feature point in the first image with the estimated position of the preselected feature point in the first image as the initial position to obtain a first tracking result, the processor is specifically used to perform the following operation: using the estimated position of the preselected feature point in the first image as the initial position, tracking the preselected feature point from a reference image to the first image to obtain the tracked position of the preselected feature point in the first image, wherein the reference image is the image in which the preselected feature point is first detected.

[0021] In a possible design, when the processor is used to perform an operation of taking the position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the third image, and obtaining a second tracking result, the processor is specifically used to perform the following operation: taking the position of the preselected feature point in the first image as the initial position, tracking the preselected feature point from the first image to the third image, and obtaining the tracked position of the preselected feature point in the third image.

[0022] In a possible design, when the processor is used to perform an operation of taking the position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the third image, and obtaining a second tracking result, the processor is specifically used to perform the following operations: taking the position of the preselected feature point in the first image as the initial position, tracking the preselected feature point from the first image to the third image, and obtaining a tracking result that the position of the preselected feature point in the third image is not tracked; obtaining the similarity between each position in the horizontal direction of the first side of the position where the preselected feature point is tracked in the first image and the tracked position, and taking the position with the largest similarity as the starting position, tracking the preselected feature point from the first image to the third image to obtain a second tracking result, and the second tracking result is the position where the preselected feature point is tracked in the third image or the position where the preselected feature point is not tracked in the third image.

[0023] In a possible design, the pre-selected feature point is a feature point among a plurality of feature points to be tracked in the first image that is tracked to a position in the first image and satisfies at least one of the following conditions: (1) the Euclidean distance between the tracked position and the estimated position of the pre-selected feature point in the first image is less than or equal to a first preset distance; (2) when the perspective point PnP problem is solved using the random sampling consistent RANSAC algorithm, the pre-selected feature point satisfies the geometric constraint; (3) the Euclidean distance between the actual position of the pre-selected feature point in the reference image and the similarity of the first reverse tracking position is less than or equal to a second preset distance, and the first reverse tracking position is a first reference position of the pre-selected feature point in the reference image. The position is the starting position, and the pre-selected feature point is tracked from the first image to the reference image; the first reference position is the actual position and any position in the adjacent area of ​​the actual position; (4) the similarity between the first image block and the second image block is greater than or equal to a first preset value, the first image block is an image block composed of the actual position and the adjacent area of ​​the actual position, and the second image block is an image block composed of the position where the pre-selected feature point is tracked in the first image and the adjacent area of ​​the tracked position; (5) when the eigenmatrix between the first image and the second image is obtained according to the RANSAC algorithm, the pre-selected feature point satisfies the epipolar constraint.

[0024] In a possible design, after obtaining the second tracking result, the processor is further configured to perform the following operation: determining a target feature point from a feature point set, where the feature point set includes all pre-selected feature points.

[0025] In a possible design, the target feature point is a feature point in the feature point set that is tracked to a position in the third image and satisfies at least one of the following conditions: (1) the Euclidean distance between the position of the target feature point tracked in the first image and the first back-projected position is less than or equal to a third preset distance, and the Euclidean distance between the position of the target feature point tracked in the third image and the second back-projected position is less than or equal to a fourth preset distance; wherein the first back-projected position is obtained by back-projecting the third three-dimensional coordinates into the first image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera after obtaining the third three-dimensional coordinates of the target feature point in the first camera coordinate system according to the position of the target feature point tracked in the first image and the third image; and the second back-projected position is obtained by back-projecting the fourth three-dimensional coordinates into the first image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera after obtaining the fourth three-dimensional coordinates of the target feature point in the second camera coordinate system according to the position of the target feature point tracked in the first image and the third image. to the third image; (2) the Euclidean distance between the position where the target feature point is tracked in the first image and the second reverse tracking position is less than or equal to a fifth preset distance, the second reverse tracking position is obtained by tracking the target feature point from the third image to the first image with the first reference position of the target feature point in the third image as the starting position; the first reference position is the position where the target feature point is tracked in the third image and any position in the adjacent area of ​​the tracked position; (3) the similarity between the third image block and the fourth image block is greater than or equal to a second preset value, the third image block is an image block composed of the position where the target feature point is tracked in the first image and the adjacent area of ​​the tracked position, and the fourth image block is an image block composed of the position where the target feature point is tracked in the third image and the adjacent area of ​​the tracked position; (4) when the eigenmatrix of the binocular camera is obtained according to the RANSAC algorithm, the target feature point satisfies the epipolar constraint.

[0026] In a possible design, the processor is also used to perform the following operations: divide the first image into multiple identical rectangular areas, uniformly detect first newly added feature points from each rectangular area, and determine a first number of second newly added feature points from the first newly added feature points, so that the sum of the first number and the number of pre-selected feature points is greater than or equal to a first preset number; the feature point set also includes all of the second newly added feature points; save the pixel blocks in the first image where the second newly added feature points are located and the pixel blocks in the adjacent areas of the pixel blocks.

[0027] In one possible design, before the processor is used to perform the operation of dividing the first image into multiple identical rectangular areas, uniformly detecting first new feature points from each rectangular area, and determining second new feature points from the first new feature points, the processor is also used to perform the following operations: determining that the average disparity between the first image and the most recently determined key frame image is greater than or equal to a preset disparity; or, determining that the total number of the pre-selected feature points is less than or equal to a second preset number.

[0028] In one possible design, after the processor is used to perform an operation of determining a first number of second newly added feature points from the first newly added feature points, the processor is also used to perform the following operation: using the position of the second newly added feature point in the first image as the initial position, tracking the second newly added feature point from the first image to the third image to obtain a third tracking result.

[0029] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including a program or instructions. When the program or instructions are run on a computer, the method described in the first aspect and each possible design of the first aspect is executed.

[0030] In a fourth aspect, an embodiment of the present invention provides a computer program, which, when executed by a computer, is used to execute the method described in the above-mentioned first aspect and various possible designs.

[0031] The program may be stored in whole or in part on a storage medium packaged with the processor, or in whole or in part on a storage medium not packaged with the processor. The storage medium is, for example, a memory.

[0032] In the present application, the estimated position of the feature point in the first image (that is, the initial position of tracking) is obtained according to the relative posture between two adjacent images (the first image and the second image, the second image being the previous frame image of the first image) taken by the first camera in the binocular camera. That is, when obtaining the initial position of tracking, the relative posture when the first camera takes the first image and the second image is taken into account, and the tracking result of the feature point in the previous pair of images can be combined to obtain an estimated position that is closer to the actual position of the feature point in the first image. That is, the estimated position of the feature point in the first image obtained in the present application is closer to the actual position of the feature point in the first image relative to the current initial position of tracking (the position where the feature point is tracked in the second image), so that the tracking result can be more accurate when tracking the position of the feature point in the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0035] Figure 2 The process of the tracking method provided in the embodiment of the present application Figure 1 ;

[0036] Figure 3 The process of the tracking method provided in the embodiment of the present application Figure 2 ;

[0037] Figure 4 A schematic diagram of the structure of a movable platform provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] Figure 1 For the application scenario diagram provided by the embodiment of the present application, see Figure 1 , a binocular camera 12 is mounted on the mobile platform 11.

[0040] The mobile platform may be a vehicle. The binocular camera 12 is used to photograph the surrounding environment of the mobile platform to obtain an image of the surrounding environment of the mobile platform.

[0041] First, a specific embodiment is used to illustrate the tracking method of the present application. Figure 2 The process of the tracking method provided in the embodiment of the present application Figure 1 . See Figure 2 , the method of this embodiment includes:

[0042] Step S201, obtaining an estimated position of a preselected feature point in the first image in the first image according to the relative posture of the first camera when the first image and the second image are captured; the first camera is a left eye or a right eye of a binocular camera, and the second image is a previous frame image of the first image.

[0043] The binocular camera includes a left eye and a right eye, that is, the binocular camera includes two cameras: a first camera and a second camera. In this embodiment, if the first camera is the left eye, the second camera is the right eye; if the first camera is the right eye, the second camera is the left eye. Two images taken by the first camera and the second camera at the same time can be called an image pair.

[0044] In this embodiment, the first image is the image captured by the first camera in the current image pair currently being processed, and the second image is the image captured by the first camera in the previous pair of image pairs of the current image pair, that is, the first image and the second image are two adjacent images captured by the first camera, and the second image is the previous frame image of the first image.

[0045] The preselected feature point is a feature point among the multiple feature points to be tracked in the first image, and a method for determining at least one preselected feature point from the multiple feature points to be tracked is described later. The multiple feature points to be tracked in the first image are the feature points determined after tracking the feature points in the previous pair of images using the tracking method in this embodiment before tracking the feature points in the current pair of images.

[0046] It can be understood that for tracking feature points in the current image pair: in addition to obtaining the estimated position of the pre-selected feature points in the first image based on the relative posture when the first camera takes the first image and the second image, it is also necessary to obtain the estimated position of other feature points in the first image based on the relative posture when the first camera takes the first image and the second image, where the other feature points are feature points other than the pre-selected feature points among the multiple feature points to be tracked in the first image.

[0047] The method for obtaining the estimated position of each feature point in the first image among the multiple feature points to be tracked in the first image is the same, and the following is explained by taking obtaining the estimated position of the pre-selected feature point in the first image as an example.

[0048] In one embodiment, according to the relative posture of the first camera when taking the first image and the second image, obtaining the estimated position of the preselected feature point in the first image in the first image can be achieved by a1-a2:

[0049] a1. According to the inertial measurement unit (IMU) data of the mobile platform and the wheel speed of the mobile platform, the relative position and posture of the first camera when taking the first image and the second image are obtained.

[0050] Optionally, the relative posture when the first camera takes the first image and the second image can be obtained by using a Kalman filter method according to the IMU data and the wheel speed of the mobile platform. The relative posture when the first camera takes the first image and the second image is also called the relative posture between the first image and the second image, that is, the relative posture between the posture when the first camera takes the first image and the posture when the first camera takes the second image.

[0051] a2. According to the relative position and posture, obtain the estimated position of the pre-selected feature point in the first image.

[0052] The position of the feature point in the image is a two-dimensional position, which can be expressed as (u, v), indicating that the position of the feature point in the image is the vth pixel in the uth row of the image. That is, the position and the pixel correspond one to one.

[0053] In one embodiment, obtaining the estimated position of the preselected feature point in the first image according to the relative position and posture can be achieved by a21 to a23:

[0054] a21. According to the tracked positions of the preselected feature points in the second image and in the fourth image, obtain the first three-dimensional coordinates of the preselected feature points in the first camera coordinate system when shooting the second image; the fourth image is shot by the second camera, and the shooting time of the second image and the fourth image is the same.

[0055] The second image and the fourth image are an image pair, the second image is an image taken by the first camera, the fourth image is an image taken by the second camera, and the second image and the fourth image are taken at the same time, that is, the second image and the fourth image are a previous image pair of the current image pair.

[0056] The positions of the pre-selected feature points tracked in the second image and in the fourth image are the positions of the pre-selected feature points tracked in the second image and in the fourth image obtained according to the tracking method in this embodiment before tracking the feature points in the current image pair, that is, the tracking results of the pre-selected feature points in the previous pair of images.

[0057] When the first camera is the left eye of a binocular camera, the first camera coordinate system is the left eye coordinate system; when the first camera is the right eye of a binocular camera, the first camera coordinate system is the right eye coordinate system.

[0058] a22. According to the relative position and posture, the first three-dimensional coordinates are converted into second three-dimensional coordinates of the feature points preselected when the first image is captured in the first camera coordinate system.

[0059] a23. Obtain an estimated position of the preselected feature point in the first image according to the second three-dimensional coordinates and the intrinsic parameter matrix of the first camera.

[0060] The estimated position of the feature point in the first image (that is, the initial position of tracking) is obtained according to the relative posture between two adjacent images (the first image and the second image, the second image being the previous frame image of the first image) taken by the first camera in the binocular camera. That is, when obtaining the initial position of tracking, the relative posture when the first camera takes the first image and the second image is considered, and the tracking result of the feature point in the previous pair of images can be combined to obtain an estimated position that is closer to the actual position of the feature point in the first image. That is, the estimated position of the feature point in the first image obtained in this embodiment is closer to the actual position of the feature point in the first image relative to the current initial position of tracking (the position where the feature point is tracked in the second image), so that the tracking result can be more accurate when tracking the position of the feature point in the first image.

[0061] Step S202: Taking the estimated position of the pre-selected feature point in the first image as the initial position, track the position of the pre-selected feature point in the first image to obtain a first tracking result.

[0062] Among them, for the tracking of feature points in the current image pair: in addition to taking the estimated position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the first image, and obtaining a first tracking result, it is also necessary to take the estimated position of other feature points in the first image as the initial position, track the positions of other feature points in the first image, and obtain a third tracking result. The meaning of other feature points can be found in the above explanation.

[0063] For each pre-selected feature: In one method, the estimated position of the pre-selected feature point in the first image is used as the initial position, and the position of the pre-selected feature point in the first image is tracked to obtain a first tracking result, including: using the estimated position of the pre-selected feature point in the first image as the initial position, tracking the pre-selected feature point from the first reference image to the first image, and obtaining the position of the pre-selected feature point tracked in the first image, the first reference image being the image in which the pre-selected feature point is first detected. In other words, the first tracking result is the position of the pre-selected feature point tracked in the first image.

[0064] The KLT algorithm may be used to track the preselected feature points from the first reference image to the first image using the estimated positions of the preselected feature points in the first image as the initial positions, thereby obtaining the tracked positions of the preselected feature points in the first image.

[0065] For each other feature point: In one method, the estimated position of the other feature point in the first image is used as the initial position, and the position of the other feature point in the first image is tracked to obtain a third tracking result, including: the estimated position of the other feature point in the first image is used as the initial position, and the other feature point is tracked from the second reference image to the first image to obtain a third tracking result, and the second reference image is the image where the other feature point is first detected. The third tracking result can be the position of the other feature point tracked in the first image (that is, the position of the other feature point tracked in the first image), or the position of the other feature point not tracked in the first image.

[0066] Step S203, taking the position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the third image to obtain a second tracking result; the third image is captured by the second camera of the binocular camera, and the first image and the third image are captured at the same time.

[0067] After obtaining the position of the pre-selected feature point tracked in the first image, the position of the pre-selected feature point tracked in the third image is then obtained. The third image is an image captured by the second camera of the binocular camera in the current image pair, and the first image and the third image are captured at the same time. It can be understood that the fourth image is the previous frame of the third image.

[0068] In one embodiment, taking the position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the third image, and obtaining the second tracking result includes: taking the position of the pre-selected feature point in the first image as the initial position, tracking the pre-selected feature point from the first image to the third image, and obtaining the position of the pre-selected feature point tracked in the third image. The second tracking result is the position of the pre-selected feature point tracked in the third image.

[0069] This method is to use the position of the pre-selected feature point tracked in the first image as the initial position, and to be able to track the position of the pre-selected feature point in the third image.

[0070] In another embodiment, taking the position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the third image to obtain a second tracking result includes:

[0071] c1. Taking the position of the pre-selected feature point tracked in the first image as the initial position, the pre-selected feature point is tracked from the first image to the third image, and the tracking result is that the position of the pre-selected feature point in the third image is not tracked. That is, taking the position of the pre-selected feature point tracked in the first image as the initial position, tracking the position of the pre-selected feature point in the third image fails.

[0072] c2. Obtain the similarity between each position in the horizontal direction of the first side of the position where the pre-selected feature point is tracked in the first image and the tracked position; take the position with the greatest similarity as the starting position, track the pre-selected feature point from the first image to the third image, and obtain a second tracking result, which is the position where the pre-selected feature point is tracked in the third image or the position where the pre-selected feature point is not tracked in the third image.

[0073] The first side may be the left side. Accordingly, the Sum of Squared Differences (SSD) algorithm may be used to sequentially calculate the similarity between each position in the horizontal direction to the left of the position where the pre-selected feature point is tracked in the first image and the tracked position. That is, one-dimensional SSD is performed pixel by pixel along the horizontal direction of the third image from the position where the pre-selected feature point is tracked in the first image to the left.

[0074] This method uses the position of the pre-selected feature point tracked in the first image as the initial position, and cannot track the position of the pre-selected feature point in the third image.

[0075] Through the above method, a first tracking result of the pre-selected feature point in the first image and a second tracking result in the third image in the current image pair are obtained.

[0076] In the case where the average disparity between the first image and the most recently determined key frame image is less than the preset disparity or the total number of pre-selected feature points is greater than the second preset number, in order to remove the feature points whose tracking results are inaccurate (i.e., remove the external points) among the pre-selected feature points and the pre-selected feature points whose positions are not tracked in the third image, after obtaining the second tracking result of the pre-selected feature points in the third image, the target feature points can be determined from the pre-selected feature points. Specifically, the target feature points can be determined from the pre-selected feature points based on the first tracking result and the second tracking result. The method for determining the target feature points from the pre-selected feature points will be described in subsequent embodiments. The number of target feature points determined from the pre-selected feature points can be 0 or one or more.

[0077] In another solution, if the average disparity between the first image and the most recently determined key frame image is greater than or equal to the preset disparity or the total number of pre-selected feature points is less than or equal to the second preset number, after determining at least one pre-selected feature point from the plurality of points to be tracked in the first image, the following steps may also be included:

[0078] e1. Divide the first image into multiple identical rectangular areas, uniformly detect first new feature points from each rectangular area, and determine a first number of second new feature points from the first new feature points, so that the sum of the first number and the number of pre-selected feature points is greater than or equal to a first preset number.

[0079] Among them, an image that meets the following conditions may be called a key frame image: the average disparity between the image and the most recently determined key frame image is greater than or equal to a preset disparity, or the number of preselected feature points determined from a plurality of feature points to be tracked in the image is less than a third preset number. The second preset number may be greater than the third preset number, or may be equal to the third preset number.

[0080] Detecting the first newly added feature point and determining a second preset number of second newly added feature points from the first newly added feature point are the same as the current method of detecting feature points from an image and the method of selecting stable feature points from the detected feature points, which will not be repeated here.

[0081] In addition, in the present embodiment, the first newly added feature point newly detected from the first image may be a feature point of a static object in the first image. That is to say, the feature points tracked by the method of the embodiment of the present application are feature points of static objects. Among them, the moving objects in the image can be detected by deep learning (such as machine learning algorithms such as deep neural networks), and the detected moving objects can be annotated with a labeling box. When newly detecting feature points from the first image, the detection result of the moving object of the second image can be used. Optionally, the labeling box for annotating the moving object of the second image can be expanded in the first image so that the labeling box surrounds the moving object in the first image as much as possible.

[0082] e2. Save the pixel block where the second newly added feature point in the first image is located and the pixel blocks in the adjacent area of ​​the pixel block.

[0083] The pixel block where the second newly added feature point is located is the image block of the pixel where the second newly added feature point is located. The pixel block in the adjacent area of ​​the pixel block is the image block composed of the neighborhood of the pixel.

[0084] Those skilled in the art should understand that in order to implement the tracking method in this embodiment, when each feature point is detected for the first time, it is necessary to save the pixel block where the feature point is located and the pixel blocks in the adjacent area of ​​the pixel block.

[0085] e3. Taking the position of the second newly added feature point in the first image as the initial position, track the second newly added feature point from the first image to the third image to obtain a third tracking result.

[0086] e4. Determine all target feature points from the pre-selected feature points and the second newly added feature points.

[0087] Specifically, all target feature points may be determined from the pre-selected feature points and the second newly added feature points according to the first tracking result, the second tracking result, the position of the second newly added feature point in the first image, and the third tracking result.

[0088] Among them, according to the first tracking result, the second tracking result, the position of the second newly added feature point in the first image and the third tracking result, the method of determining all target feature points from the pre-selected feature points and the second newly added feature points is the same as the method of determining the target feature points from the pre-selected feature points when there is no second newly added feature point.

[0089] All target feature points determined after tracking the feature points of the current image pair are multiple feature points to be tracked in the image captured by the first camera in the next image pair. Similarly, all target feature points determined after tracking the feature points of the previous image pair of the current image pair according to the above method are multiple feature points to be tracked in the first image captured by the first camera in the current image pair.

[0090] It can be understood that the method for tracking the feature points in each image pair taken by the binocular camera is the same.

[0091] The tracking method of this embodiment can obtain a relatively accurate tracking result when tracking the position of the feature point in the first image.

[0092] The following uses a specific embodiment to explain in detail the pre-selected feature points and the determination of the pre-selected feature points.

[0093] In order to remove feature points whose tracking positions in the first image are inaccurate (i.e., remove outliers) and feature points that are not tracked to their positions in the first image from among the multiple feature points to be tracked in the first image, preselected feature points can be determined from the multiple feature points to be tracked in the first image.

[0094] The preselected feature point may be a feature point among a plurality of feature points to be tracked in the first image that is tracked to a position in the first image and satisfies at least one of the following conditions (11) to (15):

[0095] (11) The Euclidean distance between the tracked position and the estimated position of the pre-selected feature point in the first image is less than or equal to a first preset distance.

[0096] Optionally, the value of the first preset distance may be any value in the set [8, 15].

[0097] (12) When using the random sample consensus (RANSAC) algorithm to solve the perspective-n-point (PnP) problem, the pre-selected feature points satisfy the geometric constraints.

[0098] (13) The Euclidean distance between the first actual position of the pre-selected feature point in the first reference image and the first reverse tracking position is less than or equal to the second preset distance; the first reverse tracking position is obtained by tracking the pre-selected feature point from the first image to the first reference image with the first reference position of the pre-selected feature point in the first reference image as the starting position; the first reference position is the first actual position of the pre-selected feature point in the first reference image and any position in an adjacent area of ​​the first actual position.

[0099] The first reference position is a position randomly selected from the first actual position of the preselected feature point in the first reference image and the adjacent area of ​​the first actual position, which may be the first actual position or a position in the adjacent area of ​​the first actual position. The meanings of subsequent similar descriptions are the same and will not be repeated. The adjacent area of ​​a position may be an area covered by the neighborhood of the pixel point corresponding to the position.

[0100] Optionally, a KLT algorithm may be used to track the preselected feature point from the first image to the first reference image with the first reference position of the preselected feature point in the first reference image as the starting position to obtain a first reverse tracking position of the preselected feature point in the first reference image.

[0101] Optionally, the value of the second preset distance may be any value in the set [8, 15].

[0102] (14) The similarity between the first image block and the second image block is greater than a first preset value, the first image block is an image block consisting of a first actual position of a pre-selected feature point in the first reference image and an adjacent area to the first actual position, and the second image block is an image block consisting of a position of the pre-selected feature point tracked in the first image and an adjacent area to the tracked position.

[0103] The similarity between the first image block and the second image block may be tested by using the Normalized Cross Correlation (NCC) principle. Optionally, the first preset value may be any value in the set [80%, 90%].

[0104] (15) When acquiring the eigenmatrix between the first image and the second image according to the RANSAC algorithm, the preselected feature points satisfy the epipolar constraint.

[0105] It should be noted that each pre-selected feature point needs to satisfy at least one of the same conditions in (11) to (15) above. For example, the pre-selected feature point is a feature point among multiple feature points to be tracked in the first image that is tracked to its position in the first image and satisfies each of the conditions in (11) to (15) above. For another example, the pre-selected feature point is a feature point among multiple feature points to be tracked that is tracked to its position in the first image and satisfies each of the conditions in (12) to (14) above.

[0106] A specific implementation of determining at least one pre-selected feature point from a plurality of feature points to be tracked in the first image is described below.

[0107] b1. Remove the feature points that are not tracked in the first image from among the plurality of feature points to be tracked in the first image to obtain first remaining feature points.

[0108] b2. For each first remaining feature point: obtain the first Euclidean distance between the tracked position and the estimated position of the first remaining feature point in the first image, and determine whether the number of first remaining feature points whose first Euclidean distance is less than or equal to the first preset distance is greater than or equal to a preset multiple of the total number of first remaining feature points; if so, remove the first remaining feature points whose first Euclidean distance is greater than the first preset distance to obtain first-category feature points; if not, use the first remaining feature points as first-category feature points.

[0109] If the first Euclidean distance is greater than the first preset distance, it means that at least one of the tracked position and the estimated position of the first remaining feature point in the first image is inaccurate, and such first remaining feature point needs to be removed.

[0110] In addition, if the number of the first remaining feature points whose first preset distance is greater than the first preset distance is less than a preset multiple of the total number of the first remaining feature points, it means that the estimated position of the first remaining feature points may be inaccurate. At this time, no first remaining feature points are removed according to the first Euclidean distance, and the first remaining feature points are regarded as first-category feature points.

[0111] Optionally, the preset multiple may be any value in the set [0.5, 0.6].

[0112] b3. According to the three-dimensional coordinates of the first type of feature points in the first camera coordinate system when taking the second image and the positions of the first type of feature points tracked in the first image, the RANSAC algorithm is used to solve the PnP problem, and the first type of feature points that do not meet the geometric constraints are removed to obtain the second type of feature points.

[0113] b4. For each second-category feature point: taking the second reference position of the second-category feature point in the third reference image as the starting position, track the second-category feature point from the first image to the third reference image to obtain a third reverse tracking position of the second-category feature point in the third reference image; the second reference position is the second actual position of the second-category feature point in the third reference image and any position in an adjacent area of ​​the second actual position, and the third reference image is the image in which the second-category feature point is first detected; and obtain a second Euclidean distance between the third reverse tracking position and the second actual position; remove the second-category feature points whose second Euclidean distance is greater than the second preset distance to obtain the third-category feature points.

[0114] Among them, the second Euclidean distance between the third anti-tracking position and the second actual position is greater than the second preset distance, indicating that at least one of the third anti-tracking position and the position tracked in the first image is inaccurate, and such second-type feature points need to be removed.

[0115] b5. For each third-category feature point: obtain the similarity between the fifth image block and the sixth image block, the fifth image block being an image block composed of the third actual position of the third-category feature point in the fourth reference image and the adjacent area of ​​the third actual position, and the sixth image block being an image block composed of the position of the third-category feature point tracked in the first image and the adjacent area of ​​the tracked position; remove the third-category feature points whose similarity between the fifth image block and the sixth image block is less than the first preset value, and obtain the fourth-category feature points. The fourth reference image is the image in which the third-category feature point appears for the first time.

[0116] For example, the similarity between the fifth image block and the sixth image block may be obtained through the NCC principle.

[0117] Among them, the similarity between the fifth image block and the sixth image block is less than the first preset value, indicating that the position of the third type of feature point tracked in the first image is likely to be inaccurate and needs to be removed.

[0118] b6. According to the RANSAC algorithm, an eigenmatrix between the first image and the second image is obtained to remove feature points that do not satisfy the epipolar constraint in the fourth type of feature points, and obtain at least one pre-selected feature point.

[0119] In this specific implementation, since some feature points have been removed from b1 to b5, using b6 as the last step to determine at least one pre-selected feature point from multiple feature points to be tracked can reduce the time required for b6 and improve the effect of removing feature points.

[0120] Furthermore, in the RANSAC algorithm, the RANSAC threshold used for feature points in the fourth category that are closer to the binocular camera may be greater than the RANSAC threshold used for feature points in the fourth category that are farther from the binocular camera.

[0121] The target feature points and their determination are described in detail below using a specific embodiment.

[0122] The target feature point is a feature point in the feature point set that is tracked to a position in the third image and satisfies at least one of the following (21) to (24), and the feature point set consists of the pre-selected feature point or the pre-selected feature point and the second newly added feature point:

[0123] (21) The Euclidean distance between the position where the target feature point is tracked in the first image and the first back-projected position is less than or equal to the third preset distance, and the Euclidean distance between the position where the target feature point is tracked in the third image and the second back-projected position is less than or equal to the fourth preset distance; wherein the first back-projected position is obtained by back-projecting the third three-dimensional coordinates into the first image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera after obtaining the third three-dimensional coordinates of the target feature point in the first camera coordinate system according to the position where the target feature point is tracked in the first image and the third image; and the second back-projected position is obtained by back-projecting the fourth three-dimensional coordinates into the third image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera after obtaining the fourth three-dimensional coordinates of the target feature point in the second camera coordinate system according to the position where the target feature point is tracked in the first image and the third image.

[0124] It can be understood that when the target feature point is the second newly added feature point, the position of the target feature point tracked in the first image is the position of the target feature point in the first image, that is, the position where the target feature point is detected in the first image.

[0125] Optionally, the third preset distance may be any value in the set [8, 15], and the fourth preset distance may be any value in the set [8, 15].

[0126] (22) The Euclidean distance between the position where the target feature point is tracked in the first image and the second reverse tracking position is less than or equal to a fifth preset distance, and the second reverse tracking position is obtained by tracking the target feature point from the third image to the first image with the first reference position of the target feature point in the first image as the starting position; the first reference position is the position where the target feature point is tracked in the first image and any position in an adjacent area of ​​the tracked position.

[0127] Optionally, the fifth preset distance may be any value in the set [8, 15].

[0128] (23) The similarity between the third image block and the fourth image block is greater than or equal to a second preset value, the third image block is an image block consisting of the position where the target feature point is tracked in the first image and an adjacent area to the tracked position, and the fourth image block is an image block consisting of the position where the target feature point is tracked in the third image and an adjacent area to the tracked position.

[0129] The similarity between the first image block and the second image block may be checked by the NCC principle. Optionally, the second preset value may be any value in the set [80%, 90%].

[0130] (24) When obtaining the intrinsic matrix of the binocular camera according to the RANSAC algorithm, the target feature points satisfy the epipolar constraints.

[0131] Understandably, Figure 2 The newly added target feature point in the illustrated embodiment is a feature point among the second newly added feature points that is tracked to a position in the third image and satisfies at least one of the conditions in (21) to (24) above.

[0132] It should be noted that each target feature point needs to meet the same conditions, for example: the target feature point is a feature point in the feature point set that is tracked to the position in the third image and satisfies the above conditions (21) to (24); for another example, the target feature point is a feature point in the feature point set that is tracked to the position in the third image and satisfies the above conditions (21) to (23).

[0133] A specific implementation method for determining a target feature point from a feature point set is described below.

[0134] d1. Remove the feature points whose positions in the third image are not tracked in the feature point set to obtain second remaining feature points.

[0135] d2. According to the position of the second remaining feature point tracked in the first image and the third image, obtain the third three-dimensional coordinates of the second remaining feature point in the first camera coordinate system and the fourth three-dimensional coordinates in the second camera coordinate system when shooting the first image; and according to the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera, back-project the third three-dimensional coordinates to the first image to obtain the first back-projected position, and according to the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera, back-project the fourth three-dimensional coordinates to the third image to obtain the second back-projected position; obtain the third Euclidean distance between the position of the second remaining feature point tracked in the first image and the first back-projected position, and the fourth Euclidean distance between the position of the second remaining feature point tracked in the third image and the second back-projected position, remove the second remaining feature points whose third Euclidean distance is greater than the third preset distance and / or whose fourth Euclidean distance is greater than the fourth preset distance, to obtain the fifth type of feature points.

[0136] d3. Taking the first reference position of the fifth category feature point in the first image as the starting position, track the fifth category feature point from the third image to the first image to obtain the second reverse tracking position of the fifth category feature point in the first image; the first reference position is the tracked position of the fifth category feature point in the first image and any position in the adjacent area of ​​the tracked position; obtain the fifth Euclidean distance between the tracked position and the second reverse tracking position of the fifth category feature point in the first image, remove the fifth category feature points whose fifth Euclidean distance is greater than the fifth preset distance, and obtain the sixth category feature points.

[0137] Among them, the fifth Euclidean distance between the tracked position of the fifth type of feature point in the first image and the second anti-tracking position is greater than the fifth preset distance, indicating that at least one of the tracked position of the fifth type of feature point in the first image and the second anti-tracking position is inaccurate, and such fifth type of feature point needs to be removed.

[0138] d4. Obtain the similarity between the third image block and the fourth image block, where the third image block is an image block composed of the position where the sixth type of feature point is tracked in the first image and any position in an adjacent area of ​​the tracked position, and the fourth image block is an image block composed of the position where the sixth type of feature point is tracked in the third image and any position in an adjacent area of ​​the tracked position; remove the sixth type of feature points whose similarity between the third image block and the fourth image block is less than a second preset value, and obtain the seventh type of feature points.

[0139] For example, the third similarity between the third image block and the fourth image block may be obtained through the NCC principle.

[0140] Among them, the similarity between the third image block and the fourth image block is less than the first preset value, indicating that the position of the sixth type of feature point tracked in the first image and / or the position tracked in the third image is likely to be inaccurate and needs to be removed.

[0141] d5. According to the RANSAC algorithm, the intrinsic matrix of the binocular camera is obtained to remove the feature points that do not satisfy the epipolar constraint in the seventh category of feature points, and finally at least one target feature point is obtained.

[0142] In this specific implementation, since some feature points in the feature point set have been removed from d1 to d4, when d5 is used as the last step to determine the target feature point from the feature point set, the time required for d5 can be reduced and the effect of removing feature points can be improved.

[0143] Furthermore, in the RANSAC algorithm, the RANSAC threshold used for feature points in the seventh category that are closer to the binocular camera may be greater than the RANSAC threshold used for feature points in the seventh category that are farther from the binocular camera.

[0144] The tracking method provided by the present application is described through the above embodiments. Figure 3 The illustrated embodiment illustrates the process of the complete tracking method provided by the present application.

[0145] Figure 3 The process of the tracking method provided in the embodiment of the present application Figure 2 , see Figure 3 , the method of this implementation includes:

[0146] Step S301, obtaining an estimated position of a feature point to be tracked in the first image according to the relative posture of the first camera when the first image and the second image are taken; the first camera is a left eye or a right eye of a binocular camera, and the second image is a previous frame image of the first image.

[0147] Step S302: Taking the estimated position of the feature point to be tracked in the first image as the initial position, track the position of the feature point to be tracked in the first image to obtain a first tracking result.

[0148] Step S303: Determine pre-selected feature points from the feature points to be tracked in the first image according to the first tracking result.

[0149] If the average disparity between the first image and the most recently determined key frame image is greater than or equal to the preset disparity, or the total number of pre-selected feature points is less than or equal to the second preset number, then steps S304 to S308 are executed:

[0150] Step S304, divide the first image into multiple identical rectangular areas, evenly detect first new feature points from each rectangular area, and determine a first number of second new feature points from the first new feature points, so that the sum of the first number and the total number of pre-selected feature points is greater than or equal to a first preset number.

[0151] Step S305: Save the pixel block where the second newly added feature point in the first image is located and the pixel blocks in the adjacent area of ​​the pixel block.

[0152] Among them, steps S304 to S305 are optional.

[0153] Step S306, taking the position of the pre-selected feature point tracked in the first image as the initial position, tracking the position of the pre-selected feature point in the third image to obtain a second tracking result; the third image is an image taken by the second camera of the binocular camera, and the first image and the third image are taken at the same time.

[0154] Step S307: Taking the position of the second newly added feature point in the first image as the initial position, track the second newly added feature point from the first image to the third image to obtain a third tracking result.

[0155] Step S308: determine all target feature points from the pre-selected feature points and the second newly added feature points.

[0156] If the average disparity between the first image and the most recently determined key frame image is less than the preset disparity and the total number of pre-selected feature points is greater than the second preset number, step S309 to step S310 are executed:

[0157] Step S309, taking the position of the pre-selected feature point tracked in the first image as the initial position, tracking the position of the pre-selected feature point in the third image to obtain a second tracking result; the third image is an image taken by the second camera of the binocular camera, and the first image and the third image are taken at the same time.

[0158] Step S310: determine all target feature points from the pre-selected feature points.

[0159] The tracking method of this embodiment can obtain a relatively accurate tracking result when tracking the position of the feature point in the first image.

[0160] The tracking method according to the embodiment of the present application is described above. The tracking device according to the embodiment of the present application is described below.

[0161] Figure 4 A schematic diagram of a movable platform provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the movable platform includes: a binocular camera 401, which is mounted on the movable platform; a processor 402, which is in communication with the binocular camera 401 and is used to perform the following operations: obtaining an estimated position of a preselected feature point in the first image according to the relative posture of the first camera when shooting the first image and the second image; the first camera is the left eye or the right eye of the binocular camera 401, and the second image is the previous frame image of the first image; taking the estimated position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the first image, and obtaining a first tracking result; taking the position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the third image, and obtaining a second tracking result; the third image is shot by the second camera of the binocular camera 401, and the shooting time of the first image and the third image is the same.

[0162] Optionally, when the processor 402 is used to execute the operation of obtaining the estimated position of a preselected feature point in the first image according to the relative pose when the first image and the second image are taken by the first camera, the processor 402 is specifically used to perform the following operations: obtaining the relative pose according to the inertial measurement unit IMU data of the mobile platform and the wheel speed of the mobile platform; obtaining the estimated position of the preselected feature point in the first image according to the relative pose.

[0163] Optionally, when the processor 402 is used to execute the operation of obtaining the estimated position of the preselected feature point in the first image according to the relative posture, it is specifically used to perform the following operations: obtaining the first three-dimensional coordinates of the preselected feature point in the first camera coordinate system when the second image is taken according to the position of the preselected feature point tracked in the second image and in the fourth image; the fourth image is taken by the second camera, and the shooting time of the second image and the fourth image is the same; according to the relative posture, converting the first three-dimensional coordinates to the second three-dimensional coordinates of the preselected feature point in the first camera coordinate system when the first image is taken; according to the second three-dimensional coordinates and the intrinsic parameter matrix of the first camera, obtaining the estimated position of the preselected feature point in the first image.

[0164] Optionally, when the processor 402 is used to perform an operation of tracking the position of the preselected feature point in the first image with the estimated position of the preselected feature point in the first image as the initial position to obtain a first tracking result, the processor 402 is specifically used to perform the following operation: using the estimated position of the preselected feature point in the first image as the initial position, tracking the preselected feature point from a reference image to the first image to obtain the tracked position of the preselected feature point in the first image, wherein the reference image is the image in which the preselected feature point is first detected.

[0165] Optionally, when the processor 402 is used to perform an operation of taking the position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the third image, and obtaining a second tracking result, the processor 402 is specifically used to perform the following operation: taking the position of the pre-selected feature point in the first image as the initial position, tracking the pre-selected feature point from the first image to the third image, and obtaining the tracked position of the pre-selected feature point in the third image.

[0166] Optionally, when the processor 402 is used to perform an operation of taking the position of the preselected feature point in the first image as the initial position, tracking the position of the preselected feature point in the third image, and obtaining a second tracking result, the processor 402 is specifically used to perform the following operations: taking the position of the preselected feature point in the first image as the initial position, tracking the preselected feature point from the first image to the third image, and obtaining a tracking result that the position of the preselected feature point in the third image is not tracked; obtaining the similarity between each position in the horizontal direction of the first side of the position where the preselected feature point is tracked in the first image and the tracked position, and taking the position with the largest similarity as the starting position, tracking the preselected feature point from the first image to the third image to obtain a second tracking result, wherein the second tracking result is the position where the preselected feature point is tracked in the third image or the position where the preselected feature point is not tracked in the third image.

[0167] Optionally, the pre-selected feature point is a feature point among a plurality of feature points to be tracked in the first image that is tracked to a position in the first image and satisfies at least one of the following conditions: (1) the Euclidean distance between the tracked position and the estimated position of the pre-selected feature point in the first image is less than or equal to a first preset distance; (2) when the perspective point PnP problem is solved using the random sampling consistent RANSAC algorithm, the pre-selected feature point satisfies the geometric constraint; (3) the Euclidean distance between the actual position of the pre-selected feature point in the reference image and the similarity of the first reverse tracking position is less than or equal to a second preset distance, and the first reverse tracking position is based on the first reference position of the pre-selected feature point in the reference image. The starting position is obtained by tracking the pre-selected feature point from the first image to the reference image; the first reference position is the actual position and any position in the adjacent area of ​​the actual position; (4) the similarity between the first image block and the second image block is greater than or equal to a first preset value, the first image block is an image block composed of the actual position and the adjacent area of ​​the actual position, and the second image block is an image block composed of the position where the pre-selected feature point is tracked in the first image and the adjacent area of ​​the tracked position; (5) when obtaining the eigenmatrix between the first image and the second image according to the RANSAC algorithm, the pre-selected feature point satisfies the epipolar constraint.

[0168] Optionally, the processor 402 is further configured to perform the following operation: determining a target feature point from a feature point set, wherein the feature point set includes all pre-selected feature points.

[0169] Optionally, the target feature point is a feature point in the feature point set that is tracked to a position in the third image and satisfies at least one of the following conditions: (1) the Euclidean distance between the position of the target feature point tracked in the first image and the first back-projected position is less than or equal to a third preset distance, and the Euclidean distance between the position of the target feature point tracked in the third image and the second back-projected position is less than or equal to a fourth preset distance; wherein the first back-projected position is obtained by back-projecting the third three-dimensional coordinates into the first image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera after obtaining the third three-dimensional coordinates of the target feature point in the first camera coordinate system according to the position of the target feature point tracked in the first image and the third image; and the second back-projected position is obtained by back-projecting the fourth three-dimensional coordinates into the second camera coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera after obtaining the fourth three-dimensional coordinates of the target feature point in the second camera coordinate system according to the position of the target feature point tracked in the first image and the third image. The method comprises the following steps: (1) the first image block is a block where the target feature point is tracked and the second image block is a block where the target feature point is tracked. The first image block is a block where the target feature point is tracked and the first image block is a block where the target feature point is tracked. The first image block is a block where the target feature point is tracked and the first image block is a block where the target feature point is tracked. The first image block is a block where the target feature point is tracked and the first image block is a block where the target feature point is tracked.

[0170] Optionally, the processor 402 is also used to perform the following operations: divide the first image into multiple identical rectangular areas, uniformly detect first new feature points from each rectangular area, and determine a first number of second new feature points from the first new feature points, so that the sum of the first number and the number of pre-selected feature points is greater than or equal to a first preset number; save the pixel blocks in the first image where the second new feature points are located and the pixel blocks in the adjacent areas of the pixel blocks.

[0171] Optionally, before the processor 402 is used to perform the operation of dividing the first image into multiple identical rectangular areas, uniformly detecting first new feature points from each rectangular area, and determining second new feature points from the first new feature points, the processor 402 is also used to perform the following operations: determining that the average disparity between the first image and the most recently determined key frame image is greater than or equal to a preset disparity; or, determining that the total number of the pre-selected feature points is less than or equal to a second preset number.

[0172] Optionally, after the processor 402 is used to perform an operation of determining a first number of second newly added feature points from the first newly added feature points, the processor 402 is also used to perform the following operations: taking the position of the second newly added feature point in the first image as the initial position, tracking the second newly added feature point from the first image to the third image to obtain a third tracking result; determining a newly added target feature point from the second newly added feature point based on the position of the second newly added feature point in the first image and the third tracking result.

[0173] The movable platform of this embodiment can be used to execute the technical solutions in the above-mentioned method embodiments. The implementation principles and technical effects are similar and will not be repeated here.

[0174] An embodiment of the present application provides a computer-readable storage medium, including a program or an instruction. When the program or the instruction is executed on a computer, the method described in the above method embodiment is executed.

[0175] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A tracking method, characterized in that: include: Obtaining an estimated position of a preselected feature point in the first image according to a relative posture of the first camera when the first image and the second image are captured; The first camera is a left camera or a right camera of a binocular camera, and the second image is a previous frame image of the first image; Taking the estimated position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the first image to obtain a first tracking result; Taking the position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the third image to obtain a second tracking result; the third image is captured by the second camera of the binocular camera, and the first image and the third image are captured at the same time; The second tracking result is a position where the pre-selected feature point is tracked in the third image or a position where the pre-selected feature point is not tracked in the third image.

2. The method according to claim 1, characterized in that The binocular camera is mounted on a mobile platform; and obtaining an estimated position of a preselected feature point in the first image according to the relative posture of the first camera when the first image and the second image are captured, comprises: Acquire the relative position and posture according to the inertial measurement unit IMU data of the mobile platform and the wheel speed of the mobile platform; According to the relative position and posture, an estimated position of the preselected feature point in the first image is obtained.

3. The method according to claim 2, characterized in that The obtaining, according to the relative posture, an estimated position of the preselected feature point in the first image comprises: According to the position of the pre-selected feature point tracked in the second image and in the fourth image, obtaining the first three-dimensional coordinates of the pre-selected feature point in the first camera coordinate system when the second image is captured; the fourth image is captured by the second camera, and the second image and the fourth image are captured at the same time; According to the relative position and posture, converting the first three-dimensional coordinates to second three-dimensional coordinates of the preselected feature point in the first camera coordinate system when the first image is captured; According to the second three-dimensional coordinates and the intrinsic parameter matrix of the first camera, an estimated position of the preselected feature point in the first image is obtained.

4. The method according to claim 1, characterized in that The step of taking the estimated position of the pre-selected feature point in the first image as the initial position and tracking the position of the pre-selected feature point in the first image to obtain a first tracking result includes: Taking the estimated position of the preselected feature point in the first image as the initial position, the preselected feature point is tracked from the reference image to the first image to obtain the tracked position of the preselected feature point in the first image, and the reference image is the image in which the preselected feature point is first detected.

5. The method according to claim 4, characterized in that The step of taking the position of the pre-selected feature point in the first image as the initial position and tracking the position of the pre-selected feature point in the third image to obtain a second tracking result includes: Taking the position of the pre-selected feature point in the first image as the initial position, the pre-selected feature point is tracked from the first image to the third image to obtain the tracked position of the pre-selected feature point in the third image.

6. The method according to claim 4, characterized in that The step of taking the position of the pre-selected feature point in the first image as the initial position and tracking the position of the pre-selected feature point in the third image to obtain a second tracking result includes: Taking the position of the pre-selected feature point in the first image as the initial position, tracking the pre-selected feature point from the first image to the third image, and obtaining a tracking result that the position of the pre-selected feature point in the third image is not tracked; The similarity between each position in the horizontal direction of the first side of the position where the pre-selected feature point is tracked in the first image and the tracked position is obtained, and the position with the largest similarity is used as the starting position, and the pre-selected feature point is tracked from the first image to the third image to obtain a second tracking result.

7. The method according to claim 5 or 6, characterized in that: The preselected feature point is a feature point that is tracked to a position in the first image and satisfies at least one of the following conditions among a plurality of feature points to be tracked in the first image: The Euclidean distance between the tracked position and the estimated position of the preselected feature point in the first image is less than or equal to a first preset distance; When the perspective point PnP problem is solved by using the random sampling consistent RANSAC algorithm, the pre-selected feature points satisfy the geometric constraints; The Euclidean distance between the similarity between the actual position of the pre-selected feature point in the reference image and the first reverse tracking position is less than or equal to a second preset distance, the first reverse tracking position is obtained by tracking the pre-selected feature point from the first image to the reference image with the first reference position of the pre-selected feature point in the reference image as the starting position; the first reference position is the actual position and any position in an adjacent area of ​​the actual position; The similarity between the first image block and the second image block is greater than or equal to a first preset value, the first image block is an image block consisting of the actual position and an adjacent area of ​​the actual position, and the second image block is an image block consisting of a position where the preselected feature point is tracked in the first image and an adjacent area of ​​the tracked position; When acquiring the eigenmatrix between the first image and the second image according to the RANSAC algorithm, the preselected feature points satisfy epipolar constraints.

8. The method according to any one of claims 4 to 6, characterized in that: After obtaining the second tracking result, the method further includes: A target feature point is determined from a feature point set, wherein the feature point set includes all pre-selected feature points.

9. The method according to claim 8, characterized in that The target feature point is a feature point in the feature point set that is tracked to a position in the third image and satisfies at least one of the following conditions: The Euclidean distance between the position where the target feature point is tracked in the first image and the first back-projected position is less than or equal to the third preset distance, and the Euclidean distance between the position where the target feature point is tracked in the third image and the second back-projected position is less than or equal to the fourth preset distance; wherein, the first back-projected position is obtained by back-projecting the third three-dimensional coordinates of the target feature point in the first camera coordinate system according to the position where the target feature point is tracked in the first image and the third image into the first image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera to obtain the third three-dimensional coordinates; the second back-projected position is obtained by back-projecting the fourth three-dimensional coordinates into the third image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera after obtaining the fourth three-dimensional coordinates of the target feature point in the second camera coordinate system according to the position where the target feature point is tracked in the first image and the third image; The Euclidean distance between the position where the target feature point is tracked in the first image and the second reverse tracking position is less than or equal to a fifth preset distance, and the second reverse tracking position is obtained by tracking the target feature point from the third image to the first image with the first reference position of the target feature point in the first image as the starting position; the first reference position is the position where the target feature point is tracked in the first image and any position in an adjacent area of ​​the tracked position; The similarity between the third image block and the fourth image block is greater than or equal to a second preset value, the third image block is an image block consisting of a position where the target feature point is tracked in the first image and an adjacent area to the tracked position, and the fourth image block is an image block consisting of a position where the target feature point is tracked in the third image and an adjacent area to the tracked position; When the intrinsic matrix of the binocular camera is obtained according to the RANSAC algorithm, the target feature points satisfy the epipolar constraint.

10. The method according to claim 8, characterized in that Also includes: Dividing the first image into a plurality of identical rectangular areas, uniformly detecting first newly added feature points from each rectangular area, and determining a first number of second newly added feature points from the first newly added feature points, so that the sum of the first number and the number of the pre-selected feature points is greater than or equal to a first preset number; the feature point set also includes all of the second newly added feature points; The pixel block where the second newly added feature point is located in the first image and the pixel blocks in the adjacent area of ​​the pixel block are saved.

11. The method according to claim 10, characterized in that Before dividing the first image into a plurality of identical rectangular areas, uniformly detecting a first newly added feature point from each rectangular area, and determining a second newly added feature point from the first newly added feature point, the method further includes: Determine that the average disparity between the first image and the most recently determined key frame image is greater than or equal to a preset disparity; or, It is determined that the total number of the pre-selected feature points is less than or equal to a second preset number.

12. The method according to claim 10, characterized in that After determining a first number of second newly added feature points from the first newly added feature points, the method further includes: Taking the position of the second newly added feature point in the first image as the initial position, the second newly added feature point is tracked from the first image to the third image to obtain a third tracking result.

13. A movable platform, characterized in that: include: A binocular camera, wherein the binocular camera is mounted on the movable platform; A processor is connected to the binocular camera for performing the following operations: Obtaining an estimated position of a preselected feature point in the first image according to a relative posture of the first camera when the first image and the second image are captured; the first camera is a left eye or a right eye of the binocular camera, and the second image is a previous frame image of the first image; Taking the estimated position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the first image to obtain a first tracking result; Taking the position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the third image to obtain a second tracking result; the third image is captured by the second camera of the binocular camera, and the first image and the third image are captured at the same time; The second tracking result is a position where the pre-selected feature point is tracked in the third image or a position where the pre-selected feature point is not tracked in the third image.

14. The movable platform according to claim 13, characterized in that: When the processor is used to perform the operation of obtaining the estimated position of the preselected feature point in the first image according to the relative posture of the first image and the second image when the first camera takes the first image, it is specifically used to perform the following operations: Acquire the relative position and posture according to the inertial measurement unit IMU data of the mobile platform and the wheel speed of the mobile platform; According to the relative position and posture, an estimated position of the preselected feature point in the first image is obtained.

15. The movable platform according to claim 14, characterized in that: When the processor is used to perform the operation of obtaining the estimated position of the preselected feature point in the first image according to the relative posture, the processor is specifically used to perform the following operation: According to the position of the pre-selected feature point tracked in the second image and in the fourth image, obtaining the first three-dimensional coordinates of the pre-selected feature point in the first camera coordinate system when the second image is captured; the fourth image is captured by the second camera, and the second image and the fourth image are captured at the same time; According to the relative position and posture, converting the first three-dimensional coordinates to second three-dimensional coordinates of the preselected feature point in the first camera coordinate system when the first image is captured; According to the second three-dimensional coordinates and the intrinsic parameter matrix of the first camera, an estimated position of the preselected feature point in the first image is obtained.

16. The movable platform according to any one of claims 13 to 15, characterized in that: When the processor is used to perform an operation of taking the estimated position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the first image, and obtaining a first tracking result, the processor is specifically used to perform the following operations: Taking the estimated position of the preselected feature point in the first image as the initial position, the preselected feature point is tracked from the reference image to the first image to obtain the tracked position of the preselected feature point in the first image, and the reference image is the image in which the preselected feature point is first detected.

17. The movable platform according to claim 16, characterized in that: When the processor is used to perform an operation of taking the position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the third image, and obtaining a second tracking result, the processor is specifically used to perform the following operations: Taking the position of the pre-selected feature point in the first image as the initial position, the pre-selected feature point is tracked from the first image to the third image to obtain the tracked position of the pre-selected feature point in the third image.

18. The movable platform according to claim 16, characterized in that: When the processor is used to perform an operation of taking the position of the pre-selected feature point in the first image as the initial position, tracking the position of the pre-selected feature point in the third image, and obtaining a second tracking result, the processor is specifically used to perform the following operations: Taking the position of the pre-selected feature point in the first image as the initial position, tracking the pre-selected feature point from the first image to the third image, and obtaining a tracking result that the position of the pre-selected feature point in the third image is not tracked; The similarity between each position in the horizontal direction of the first side of the position where the pre-selected feature point is tracked in the first image and the tracked position is obtained, and the position with the largest similarity is used as the starting position, and the pre-selected feature point is tracked from the first image to the third image to obtain a second tracking result.

19. The movable platform according to claim 17, characterized in that: The preselected feature point is a feature point that is tracked to a position in the first image and satisfies at least one of the following conditions among a plurality of feature points to be tracked in the first image: The Euclidean distance between the tracked position and the estimated position of the preselected feature point in the first image is less than or equal to a first preset distance; When the perspective point PnP problem is solved by using the random sampling consistent RANSAC algorithm, the pre-selected feature points satisfy the geometric constraints; The Euclidean distance between the similarity between the actual position of the pre-selected feature point in the reference image and the first reverse tracking position is less than or equal to a second preset distance, the first reverse tracking position is obtained by tracking the pre-selected feature point from the first image to the reference image with the first reference position of the pre-selected feature point in the reference image as the starting position; the first reference position is the actual position and any position in an adjacent area of ​​the actual position; The similarity between the first image block and the second image block is greater than or equal to a first preset value, the first image block is an image block consisting of the actual position and an adjacent area of ​​the actual position, and the second image block is an image block consisting of a position where the preselected feature point is tracked in the first image and an adjacent area of ​​the tracked position; When acquiring the eigenmatrix between the first image and the second image according to the RANSAC algorithm, the preselected feature points satisfy epipolar constraints.

20. The movable platform according to claim 16, characterized in that: After obtaining the second tracking result, the processor is further configured to perform the following operation: determining a target feature point from a feature point set, wherein the feature point set includes all pre-selected feature points.

21. The movable platform according to claim 20, characterized in that: The target feature point is a feature point in the feature point set that is tracked to a position in the third image and satisfies at least one of the following conditions: The Euclidean distance between the position where the target feature point is tracked in the first image and the first back-projected position is less than or equal to the third preset distance, and the Euclidean distance between the position where the target feature point is tracked in the third image and the second back-projected position is less than or equal to the fourth preset distance; wherein, the first back-projected position is obtained by back-projecting the third three-dimensional coordinates of the target feature point in the first camera coordinate system according to the position where the target feature point is tracked in the first image and the third image into the first image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera to obtain the third three-dimensional coordinates; the second back-projected position is obtained by back-projecting the fourth three-dimensional coordinates into the third image according to the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera after obtaining the fourth three-dimensional coordinates of the target feature point in the second camera coordinate system according to the position where the target feature point is tracked in the first image and the third image; The Euclidean distance between the position where the target feature point is tracked in the first image and the second reverse tracking position is less than or equal to a fifth preset distance, and the second reverse tracking position is obtained by tracking the target feature point from the third image to the first image with the first reference position of the target feature point in the first image as the starting position; the first reference position is the position where the target feature point is tracked in the first image and any position in an adjacent area of ​​the tracked position; The similarity between the third image block and the fourth image block is greater than or equal to a second preset value, the third image block is an image block consisting of a position where the target feature point is tracked in the first image and an adjacent area to the tracked position, and the fourth image block is an image block consisting of a position where the target feature point is tracked in the third image and an adjacent area to the tracked position; When the intrinsic matrix of the binocular camera is obtained according to the RANSAC algorithm, the target feature points satisfy the epipolar constraint.

22. The movable platform according to claim 20, characterized in that: The processor is further configured to perform the following operations: Dividing the first image into a plurality of identical rectangular areas, uniformly detecting first newly added feature points from each rectangular area, and determining a first number of second newly added feature points from the first newly added feature points, so that the sum of the first number and the number of the pre-selected feature points is greater than or equal to a first preset number; the feature point set also includes all of the second newly added feature points; The pixel block where the second newly added feature point is located in the first image and the pixel blocks in the adjacent area of ​​the pixel block are saved.

23. The movable platform according to claim 22, characterized in that: Before the processor is used to perform the operations of dividing the first image into a plurality of identical rectangular regions, uniformly detecting a first newly added feature point from each rectangular region, and determining a second newly added feature point from the first newly added feature point, the processor is further used to perform the following operations: Determine that the average disparity between the first image and the most recently determined key frame image is greater than or equal to a preset disparity; or, It is determined that the total number of the pre-selected feature points is less than or equal to a second preset number.

24. The movable platform according to claim 22 or 23, characterized in that: After the processor is configured to perform an operation of determining a first number of second newly added feature points from the first newly added feature points, the processor is further configured to perform the following operation: Taking the position of the second newly added feature point in the first image as the initial position, the second newly added feature point is tracked from the first image to the third image to obtain a third tracking result.

25. A computer-readable storage medium, comprising a program or an instruction, characterized in that: When the program or instruction runs on a computer, the method according to any one of claims 1 to 12 is executed.

Images