Electronic device, optical flow tracking method, device and storage medium

Abstract

The present application is applicable to the field of image processing technology and provides an electronic device, an optical flow tracing method, an apparatus, and a storage medium, including: obtaining a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points; determining a predicted position of the electronic device corresponding to the second moment based on motion data from the first moment to the second moment; determining a predicted point position corresponding to the first feature point in the second image based on the predicted position; and determining a second feature point corresponding to the first feature point in the second image using a preset optical flow algorithm based on the first feature point and the predicted point position. Embodiments of the present application can improve the efficiency of optical flow tracing.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an electronic device, an optical flow tracking method, an apparatus, and a storage medium. Background Art

[0002] Currently, image processing often requires image registration based on the correlation between feature points in different images, enabling applications such as pose calculation and map construction. Currently, the correlation between feature points in different images can be constructed through optical flow tracking.

[0003] However, in existing optical flow tracking methods, the search range of the optical flow algorithm iteration is usually large or not accurate enough, resulting in low efficiency of existing optical flow tracking methods. Summary of the Invention

[0004] In view of this, embodiments of the present application provide an electronic device, an optical flow tracing method, an apparatus, and a storage medium to solve the problem of how to improve the efficiency of optical flow tracing in the prior art.

[0005] A first aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0006] Acquire a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points;

[0007] Determining a predicted position and posture of the electronic device at the second moment based on the motion data from the first moment to the second moment;

[0008] Determining, based on the predicted pose, a predicted point position of the first feature point corresponding to the second image;

[0009] According to the first feature point and the predicted point position, a second feature point corresponding to the first feature point in the second image is determined by a preset optical flow algorithm.

[0010] Optionally, the motion data includes sensor data and / or motion state information. When the processor executes the computer program, determining the predicted posture corresponding to the second moment based on the motion data from the first moment to the second moment includes:

[0011] Obtaining a first posture corresponding to the electronic device at the first moment;

[0012] Determining a target posture increment corresponding to the second moment based on sensor data and / or motion state information from the first moment to the second moment;

[0013] Determine a predicted pose corresponding to the second moment according to the target pose increment and the first pose.

[0014] Optionally, when the processor executes the computer program, determining, based on the predicted pose, a predicted point position of the first feature point corresponding to the second image includes:

[0015] According to the first three-dimensional feature points corresponding to the first three-dimensional feature points in the preset three-dimensional map, and the predicted posture, the projection position of each first three-dimensional feature point is projected to the second image as the corresponding predicted point position; wherein, the preset three-dimensional map is a three-dimensional map constructed based on each frame image collected at the first moment and before the first moment.

[0016] Optionally, when the processor executes the computer program, determining the second feature point in the second image by using a preset optical flow algorithm based on the first feature point and the predicted point position includes:

[0017] According to the first feature point and the predicted point position, a second feature point in the second image is determined by running a preset optical flow algorithm in a preset thread pool; wherein the preset thread pool includes at least two threads.

[0018] Optionally, when the processor executes the computer program, determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm includes:

[0019] Determine, based on the first feature point and the predicted point position, a first tracking point in the second image corresponding to the first feature point using an optical flow algorithm of a first pyramid model;

[0020] If a feature point to be tracked exists in the first image, determining a second tracking point in the second image using an optical flow algorithm of a second pyramid model based on the feature point to be tracked and a position of a predicted point corresponding to the feature point to be tracked; wherein the feature point to be tracked is a first feature point for which no corresponding first tracking point exists; and the number of levels of the second pyramid model is greater than the number of levels of the first pyramid model;

[0021] A second feature point in the second image is determined based on the first tracking point and the second tracking point.

[0022] Optionally, when the processor executes the computer program, determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm includes:

[0023] Determining a third tracking point tracked in the second image using a preset optical flow algorithm based on the first feature point and the predicted point position;

[0024] Determining, according to a reverse optical flow tracking algorithm, the third tracking point that can be reversely tracked to the corresponding first feature point as a target tracking point;

[0025] Determine a second feature point in the second image based on the target tracking point.

[0026] Optionally, when the processor executes the computer program, the following steps are further implemented:

[0027] If the number of the second feature points is less than a preset threshold, feature points of the second image are extracted using a preset feature point extraction algorithm to obtain new feature points; wherein the sum of the number of the second feature points and the new feature points is equal to the preset threshold.

[0028] A second aspect of the embodiments of the present application provides an optical flow tracing method, including:

[0029] Acquire a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points;

[0030] Determining a predicted position and posture of the electronic device at the second moment based on the motion data from the first moment to the second moment;

[0031] Determining, based on the predicted pose, a predicted point position of the first feature point corresponding to the second image;

[0032] According to the first feature point and the predicted point position, a second feature point corresponding to the first feature point in the second image is determined by a preset optical flow algorithm.

[0033] A third aspect of the embodiments of the present application provides an optical flow tracking device, comprising:

[0034] An image acquisition module, configured to acquire a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points;

[0035] a predicted posture determination module, configured to determine a predicted posture corresponding to the electronic device at the second moment based on the motion data from the first moment to the second moment;

[0036] a predicted point position determination module, configured to determine the predicted point position of the first feature point corresponding to the second image based on the predicted pose;

[0037] The second feature point determination module is used to determine a second feature point corresponding to the first feature point in the second image through a preset optical flow algorithm based on the first feature point and the predicted point position.

[0038] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the electronic device implements the steps of the optical flow tracing method.

[0039] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the steps of the optical flow tracing method described in the second aspect above.

[0040] Compared with the prior art, the embodiments of the present application have the following beneficial effects: in the embodiments of the present application, after obtaining a first image corresponding to a first moment and a second image corresponding to a second moment, the predicted position of the electronic device corresponding to the second moment is determined based on the motion data from the first moment to the second moment; based on the predicted position, the predicted point position of the first feature point corresponding to the second image can be preliminarily calculated; thereafter, based on the first feature point and the predicted point position, the second feature point in the second image is determined by a preset optical flow algorithm. Since the predicted point position of the first feature point in the second image can be preliminarily determined by the predicted position of the electronic device corresponding to the second moment, an accurate initial position can be provided for the subsequent preset optical flow algorithm, reducing the search range of the optical flow tracking iteration in the second image, quickly and accurately determining the second feature point corresponding to the first feature point, and improving the success rate and accuracy of optical flow tracking while reducing the optical flow tracking time, thereby improving the efficiency of optical flow tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art.

[0042] Figure 1 is a schematic diagram of an electronic device provided in an embodiment of the present application;

[0043] Figure 2 This is a schematic structural diagram of a robot provided in an embodiment of the present application;

[0044] Figure 3 This is a schematic diagram of an implementation flow of an optical flow tracing method provided in an embodiment of the present application;

[0045] Figure 4 is a schematic diagram of an optical flow tracking device provided in an embodiment of the present application; DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0047] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0048] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0049] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0050] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0051] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0052] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0053] Currently, in optical flow tracking methods, the search range of optical flow algorithm iteration is usually large or not accurate enough, resulting in low efficiency of existing optical flow tracking methods.

[0054] For example, in the field of robotics, due to the low cost of sensors, simultaneous localization and mapping (SLAM) technology has gradually been widely used. In SLAM positioning solutions, a multi-sensor fusion and vision-driven positioning method is usually adopted. In this positioning method, feature points are visually utilized to calculate the robot's moving posture based on the correlation between feature points in different images. In addition to constructing this correlation through descriptor matching, another commonly used method for constructing feature point correlation is the above-mentioned optical flow tracking method.

[0055] In the field of robotics, SLAM-based map construction or pose calculation typically involves capturing images while the robot is moving using visual sensors. Traditional optical flow tracking methods can cause significant offsets between the current frame and the previous image due to the robot's high speed or large turns. The initial tracking position determined by traditional optical flow tracking algorithms is inaccurate, meaning the search range of the optical flow algorithm is inaccurate. This can easily lead to feature point tracking failures and affect optical flow tracking efficiency.

[0056] In order to solve the above-mentioned technical problems, the embodiments of the present application provide an electronic device, an optical flow tracking method, an apparatus and a storage medium. After obtaining a first image corresponding to a first moment and a second image corresponding to a second moment, the predicted posture of the electronic device corresponding to the second moment is determined based on the motion data from the first moment to the second moment; based on the predicted posture, the predicted point position of the first feature point corresponding to the second image can be preliminarily calculated; thereafter, based on the first feature point and the predicted point position, the second feature point in the second image is determined through a preset optical flow algorithm.

[0057] Since the predicted position of the first feature point in the second image can be preliminarily determined through the predicted posture of the electronic device corresponding to the second moment, it is possible to provide an accurate initial position for the subsequent preset optical flow algorithm, reduce the search range of the optical flow tracking iteration, and improve the success rate and accuracy of the optical flow tracking while reducing the optical flow tracking time, thereby improving the efficiency of the optical flow tracking.

[0058] When a robot performs map construction or pose calculation based on SLAM, the optical flow tracking method of the embodiment of the present application can be used to enable the robot to maintain a robust optical flow tracking effect when moving at high speed or rotating rapidly, thereby improving the success rate of optical flow tracking and efficiently performing map construction or pose calculation.

[0059] Example 1:

[0060] Figure 1Schematic diagram of an electronic device provided in one embodiment of the present application. The electronic device in this embodiment of the present application is a movable device, including but not limited to a robot, a mobile phone, a vehicle, etc. The movable device is provided with a camera module, which can be a camera, an independent visual sensor, or a camera module.

[0061] The electronic device may include, but is not limited to, a processor 10 and a memory 11. Those skilled in the art will understand that Figure 1 It is only an example of the electronic device 1 and does not constitute a limitation of the electronic device 1. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0062] The processor 10 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0063] The memory 11 may be an internal storage unit of the electronic device 1, such as a hard disk or memory of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1. Furthermore, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 is used to store the computer program and other programs and data required by the electronic device. The memory 11 may also be used to temporarily store data that has been output or is to be output.

[0064] Exemplarily, the computer program 12 may be divided into one or more modules / units, which are stored in the memory 11 and executed by the processor 10 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 12 in the electronic device 1.

[0065] When the processor 10 executes the computer program 12, steps S101 to S104 are implemented as follows:

[0066] S101: Acquire a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points.

[0067] In the embodiments of the present application, the first image is an image captured by the camera module at a first moment, and the second image is an image captured by the camera module at a second moment, where the first moment is earlier than the second moment. In one embodiment, the first image and the second image are two consecutive frames of images captured by the camera module; that is, if the second image is the image captured by the camera module at the current moment, the first image is the previous frame of image captured by the camera module at the previous moment.

[0068] The first image of the embodiment of the present application includes a preset number of feature points, and the feature points on the first image are referred to as first feature points. In one embodiment, the first image is the first frame image, and the preset number of first feature points are: points obtained by extracting feature points from the first image using a preset feature point extraction algorithm. In another embodiment, if the first image is not the first frame image, the preset number of first feature points may include: feature points obtained by optical flow tracking based on the previous frame image of the first image; and may also include additional feature points extracted from the first image using a feature point extraction algorithm.

[0069] S102: Determine a predicted position and posture of the electronic device at the second moment based on the motion data from the first moment to the second moment.

[0070] After acquiring the first image and the second image, the predicted position of the electronic device at the second moment can be determined based on the motion data between the moments when the two frames of images were captured, that is, the motion data from the first moment to the second moment.

[0071] Specifically, a calculation can be performed based on the first pose corresponding to the first moment and the motion data to obtain a predicted pose of the electronic device corresponding to the second moment. The first pose is predetermined before the second moment. In one embodiment, if the first image is the first frame image, the first pose can be directly the initial pose in the reference coordinate system set in advance. In another embodiment, if the first image is not the first frame image, the first pose can be the pose in the reference coordinate system obtained by pose calculation based on the first feature point on the first image (specifically, it can be the pose obtained by adding the offset of the first image relative to the initial pose to the initial pose).

[0072] S103: Determine, based on the predicted pose, a predicted point position of the first feature point corresponding to the second image.

[0073] In one embodiment, after determining the predicted pose corresponding to the second moment, the offset of the second image relative to the first image in a preset reference coordinate system can be determined based on the predicted pose. Based on the offset, the position coordinates of each first feature point in the first image corresponding to the position in the second image are calculated, thereby obtaining the predicted point position corresponding to each first feature point.

[0074] S104: Determine a second feature point corresponding to the first feature point in the second image using a preset optical flow algorithm based on the first feature point and the predicted point position.

[0075] After determining the predicted point positions corresponding to each first feature point in the second image, a preset optical flow algorithm is used to determine, for each first feature point in the first image, the tracking points that match the first feature point tracked in the second image. Subsequently, based on these tracking points, the second feature points corresponding to the first feature points in the second image are determined. In one embodiment, all tracked tracking points can be directly used as the second feature points in the second image. In another embodiment, points that do not meet preset conditions can be further identified among the tracking points as outliers and removed, with the remaining tracking points after removal being used as the second feature points.

[0076] In an embodiment of the present application, the preset optical flow algorithm is an algorithm for determining matching feature points between two images based on the grayscale invariance of pixels between the two images. Specifically, for a first feature point in the first image, based on the grayscale invariance, a feature point in the second image that has substantially the same pixel information as the first feature point can be determined as a second feature point. The pixel information can be a grayscale value or color data. For example, the color data can be an RGB value in a red, green, and blue (RGB) color mode.

[0077] In an embodiment of the present application, after determining the predicted point position of the first feature point in the second image, based on the predicted point position, an optical flow tracking area corresponding to the first feature point can be quickly and accurately located in the second image, and within the optical flow tracking area, based on grayscale invariance, a second feature point corresponding to the first feature point can be found.

[0078] Specifically, for each first feature point, based on the predicted point position corresponding to the first feature point in the second image, a neighborhood centered on the predicted point position is determined in the second image as an optical flow tracking area. This optical flow tracking area is then used as the optical flow search range for the first feature point in the second image. Points within the optical flow tracking area whose pixel information similarity with the first feature point in the first image exceeds a preset value are searched for as tracked points. Based on the tracked tracking points, a second feature point corresponding to the first feature point can be determined.

[0079] In an optional embodiment, the first feature point in the first image and the second feature point in the second image are the same entity or very close entities in the actual scene.

[0080] For example, in the embodiment of the present application, the preset optical flow algorithm may be the LK (Lucas-Kanade) algorithm, which is a two-frame difference optical flow estimation algorithm. Alternatively, the preset optical flow algorithm may be the Gunnar Farneback algorithm (a dense optical flow algorithm).

[0081] After determining the second feature point of the second image, the posture corresponding to the second moment can be accurately calculated based on the association between the second feature point and the first feature point; alternatively, the information of the second image can be converted into a preset three-dimensional map to continue to complete the synchronous construction of the map; alternatively, image registration and stitching of the first image and the second image can be achieved.

[0082] In an embodiment of the present application, since the predicted point position of the first feature point in the second image can be preliminarily determined through the predicted posture corresponding to the second moment, an accurate initial position can be provided for the subsequent preset optical flow algorithm, thereby reducing the search range of the optical flow tracking iteration in the second image, and quickly and accurately determining the second feature point corresponding to the first feature point, thereby reducing the optical flow tracking time and improving the success rate and accuracy of the optical flow tracking, thereby improving the efficiency of the optical flow tracking.

[0083] Optionally, the motion data includes sensor data and / or motion state information, and determining the predicted posture corresponding to the second moment based on the motion data from the first moment to the second moment includes:

[0084] Obtaining a first posture corresponding to the electronic device at the first moment;

[0085] Determining a target posture increment corresponding to the second moment based on sensor data and / or motion state information from the first moment to the second moment;

[0086] Determine a predicted pose corresponding to the second moment according to the target pose increment and the first pose.

[0087] In an embodiment of the present application, before determining the predicted posture corresponding to the second moment, the first posture of the electronic device corresponding to the first moment is first obtained. In one embodiment, the pre-stored first posture can be read from the cache unit of the electronic device or from the data input by the user. As a possible implementation method, when the first image is the first frame image, the first posture can be directly the initial posture in the reference coordinate system stored in the cache unit. In another embodiment, the first image is not the first frame image, then the first posture can be the posture stored in the cache unit obtained by posture calculation based on the first feature point on the first image.

[0088] Specifically, when the motion data includes sensor data, multiple sensor data collected by the sensor within a target time period starting at the first moment and ending at the second moment (including the first moment and the second moment) can be counted, and an integral operation can be performed based on these multiple sensor data to determine the target posture increment corresponding to the second moment. As a possible implementation method, if the frame rate of the camera module is inconsistent with the acquisition frequency of the sensor, assuming that the sensor data acquired within the target time is N (N is a positive integer greater than 1), then when there is no sensor data collected at the second moment, two sensor data collected before and after the second moment are obtained and interpolated to obtain a new sensor data, and the N sensor data acquired within the target time are combined with the new sensor data to obtain N+1 sensor data. Performing an integral operation based on this N+1 sensor data can more accurately determine the target posture increment corresponding to the second moment, thereby improving the accuracy of posture prediction.

[0089] Specifically, when the motion data includes motion state information, a motion model can be established based on the motion state information to determine the target pose increment. For example, when the motion state information corresponding to the first moment to the second moment indicates a uniform motion state, the uniform motion model can be used to directly use the pre-determined pose increment corresponding to the first moment as the target pose increment for the second moment. When the motion state information corresponding to the first moment to the second moment indicates a uniformly accelerated motion state, the uniformly accelerated motion model can be used to calculate the target pose increment for the second moment based on the velocity and acceleration information corresponding to the first moment.

[0090] Typically, the target pose increment can be determined by any of the above methods. In some embodiments, a first target pose increment can be calculated using sensor data, and a second target pose increment can be calculated using motion state information, and the average of these two target pose increments can be calculated as the final target pose increment.

[0091] After determining the target pose increment, the first pose corresponding to the first moment is added to the target pose increment corresponding to the second moment to obtain the predicted pose corresponding to the second moment.

[0092] In one embodiment, the electronic device is a robot equipped with a wheel odometer and a gyroscope, the sensor data includes translational velocity data collected by the wheel odometer and angular velocity data collected by the gyroscope, and the target pose increment includes a translational increment and a rotation angle increment. Correspondingly, determining the target pose increment corresponding to the second moment based on the sensor data and / or motion state information from the first moment to the second moment includes:

[0093] determining a translation increment of the robot according to each of the translation speed data from the first moment to the second moment;

[0094] determining a rotation angle increment of the robot according to each of the angular velocity data from the first moment to the second moment;

[0095] Correspondingly, determining the predicted pose corresponding to the second moment according to the target pose increment and the predetermined first pose corresponding to the first moment includes:

[0096] Determine a predicted position and posture of the robot at the second moment according to the translation increment, the rotation angle increment, and the first position and posture corresponding to the first moment.

[0097] In the embodiment of the present application, the translation increment ΔP of the robot can be determined by integrating the translation velocity data collected from the first moment to the second moment. The rotation angle increment ΔR of the robot can be determined by integrating the angular velocity data collected from the first moment to the second moment.

[0098] Let the first pose corresponding to the first moment be T wi =[R wi ,P wi ], then the predicted pose corresponding to the second moment is T wj =[R wi *ΔR,P wi +R wi *ΔP]. Wherein, “*” represents the multiplication sign.

[0099] In an embodiment of the present application, the target pose increment corresponding to the second moment can be accurately calculated through sensor data and / or motion state information; based on the target pose increment and the first pose at the first moment, the predicted pose corresponding to the second moment can be accurately predicted, so that the corresponding predicted point position can be accurately determined based on the predicted pose, thereby improving the efficiency and accuracy of optical flow tracking based on the predicted point position.

[0100] Optionally, determining, based on the predicted pose, a predicted point position of the first feature point corresponding to the second image includes:

[0101] According to the first three-dimensional feature points corresponding to the first three-dimensional feature points in the preset three-dimensional map, and the predicted posture, the projection position of each first three-dimensional feature point is projected to the second image as the corresponding predicted point position; wherein, the preset three-dimensional map is a three-dimensional map constructed based on each frame image collected at the first moment and before the first moment.

[0102] In the embodiment of the present application, for each image captured by the camera module, after accurately determining the feature points of the image, the precise pose corresponding to the moment the image was captured is accurately calculated. Based on this precise pose, a three-dimensional map is constructed using triangulation technology, and the feature points on the image are mapped to three-dimensional feature points in the three-dimensional map. Immediately after the second image is acquired at the second moment, a three-dimensional map has been obtained using triangulation technology based on the frames of images captured at and before the first moment. This three-dimensional map is referred to as a preset three-dimensional map. The preset three-dimensional map includes each first three-dimensional feature point corresponding to each first feature point of the first image.

[0103] After determining the predicted posture corresponding to the second image, each first three-dimensional feature point in the preset three-dimensional map can be projected into the second image according to the predicted posture, so as to obtain the predicted point position of each first feature point of the first image corresponding to the second image.

[0104] For example, let the coordinates of a first three-dimensional feature point on the preset three-dimensional map be: P = [X, Y, Z], and the predicted pose corresponding to the second image be T wj , then the projection formula for projecting the first three-dimensional feature point to the preset three-dimensional feature point is:

[0105] Puv=Ω(T wj -1 *P)

[0106] Wherein, Ω(...) represents a universal camera projection function, which represents the projection equation for projecting the 3D camera coordinates onto the pixel plane. The camera projection function is determined according to the camera parameters; “T wj -1” indicates the predicted pose T wj The reciprocal of ; “*” represents a multiplication sign; Puv is the predicted point position projected to the second image.

[0107] As a possible implementation method, when the projection fails or the first feature point does not establish a corresponding first three-dimensional feature point in the preset three-dimensional map, the position in the second image that is consistent with the coordinates of the pixel position is directly determined based on the pixel position of the first feature point in the first image as the predicted point position corresponding to the first feature point in the second image.

[0108] In an embodiment of the present application, based on the predicted posture, the first three-dimensional feature point corresponding to the first feature point in the preset three-dimensional map is projected into the second image, so that the position of the predicted point corresponding to the first feature point of the first image in the second image can be accurately determined, and the initial search range of the optical flow tracking can be accurately determined, thereby improving the efficiency and accuracy of the optical flow tracking.

[0109] Optionally, determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm includes:

[0110] According to the first feature point and the predicted point position, a second feature point in the second image is determined by running a preset optical flow algorithm in a preset thread pool; wherein the preset thread pool includes at least two threads.

[0111] In an embodiment of the present application, a preset optical flow algorithm is specifically run through a preset thread pool, and the preset thread pool includes two or more threads.

[0112] Specifically, in this step, the first image, the position information of each first feature point in the first image, and the second image, the predicted point position corresponding to each first feature point in the second image, and other information can be input into the preset thread pool, and a specified number of first feature points and their corresponding predicted point positions are obtained simultaneously each time. Through the preset optical flow algorithm, the optical flow tracking operation of the specified number of first feature points is performed simultaneously to obtain a specified number of second feature points that match the first feature points. The specified number is equal to the number of threads included in the preset thread pool. For example, if the thread pool includes 6 threads, then 6 feature points and their corresponding 6 predicted point positions are obtained each time to perform optical flow tracking operations, so that the optical flow tracking of 6 feature points can be completed simultaneously.

[0113] Specifically, the steps of obtaining a specified number of first feature points for optical flow tracking calculation can be executed cyclically from all first feature points contained in the first image until the optical flow tracking calculation has been completed for all first feature points in the first image. Then, the optical flow tracking is completed, and tracking points matching the first feature points of the first image are obtained in the second image. Second feature points corresponding to the first feature points in the second image are determined based on these tracking points.

[0114] In an embodiment of the present application, since optical flow tracking of multiple feature points can be performed simultaneously through at least two threads of a preset thread pool, the time required for optical flow tracking can be reduced, especially when the number of feature points is large, which can greatly improve the efficiency of optical flow tracking.

[0115] Optionally, determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm includes:

[0116] Determine, based on the first feature point and the predicted point position, a first tracking point in the second image corresponding to the first feature point using an optical flow algorithm of a first pyramid model;

[0117] If a feature point to be tracked exists in the first image, determining a second tracking point in the second image using an optical flow algorithm of a second pyramid model based on the feature point to be tracked and a position of a predicted point corresponding to the feature point to be tracked; wherein the feature point to be tracked is a first feature point for which no corresponding first tracking point exists; and the number of levels of the second pyramid model is greater than the number of levels of the first pyramid model;

[0118] A second feature point in the second image is determined based on the first tracking point and the second tracking point.

[0119] In the embodiment of the present application, optical flow tracking of feature points between images is performed using an optical flow tracking algorithm of a pyramid model. In the pyramid model, each pyramid layer reduces the image to a preset ratio of the previous layer. The preset ratio can be determined according to actual conditions, for example, 0.8.

[0120] In an embodiment of the present application, optical flow tracking operations can be performed using a first pyramid model and / or a second pyramid model. The number of levels of the second pyramid model is greater than the number of levels of the first pyramid model. For example, the first pyramid model can be a two-layer pyramid model, and the second pyramid model can be a four-layer pyramid model. The two-layer pyramid model includes two pyramid layers. The first pyramid layer reduces the original image to a preset ratio of the original image to obtain a first reduced image; the second pyramid layer further reduces the first reduced image to a preset ratio of the first reduced image to obtain a second reduced image. The four-layer pyramid model includes four pyramid layers, and so on. Each pyramid layer in the four pyramid layers further reduces the reduced image obtained in the previous layer according to a preset ratio. The images corresponding to each layer are the first reduced image, the second reduced image, the third reduced image, and the fourth reduced image. In the pyramid model, the more pyramids it contains, the smaller the reduction ratio of the reduced image of the last pyramid layer relative to the original image, and the more information of the original image contained in the neighborhood determined according to the prediction point position. That is, the second pyramid model with more hierarchical structures has a wider search range for optical flow tracking than the first pyramid model, and the optical flow tracking takes longer, but the accuracy and probability of successful tracking are higher.

[0121] In an embodiment of the present application, after determining the predicted point positions of each first feature point of the first image in the second image, an optical flow tracking operation is first performed using the optical flow algorithm of the first pyramid model to determine the first tracking point corresponding to the first feature point of the first image in the second image.

[0122] If all first feature points in the first image are successfully tracked to corresponding first tracking points in the second image, the second feature points in the second image can be directly determined based on the first tracking points. That is, in this case, the second tracking points can be directly considered an empty set, and the process skips to the step of determining the second feature points in the second image based on the first and second tracking points.

[0123] If the optical flow algorithm based on the first pyramid model fails to track any first feature points in the first image, that is, if there are first feature points in the first image that cannot be tracked to corresponding first tracking points in the second image, these first feature points that cannot be tracked to corresponding first tracking points are determined as feature points to be tracked. At this point, an optical flow tracking operation is further performed using the optical flow algorithm based on the second pyramid model, based on each feature point to be tracked and its corresponding predicted point position, to determine the second tracking point in the second image corresponding to the feature point to be tracked in the first image.

[0124] The first tracking point and the second tracking point tracked in the second image are then merged to obtain a tracking point set. In one embodiment, each tracking point in the tracking point set can be directly used as a second feature point in the second image. In another embodiment, the tracking point set can be further processed to remove outliers to obtain an updated tracking point set. Each tracking point in the updated tracking point set is then determined as a second feature point in the second image corresponding to the first feature point.

[0125] In an embodiment of the present application, an optical flow tracking operation is first performed based on an optical flow algorithm of a less complex first pyramid model to obtain a first tracking point. When there are first feature points in the first image that fail to be tracked using the optical flow algorithm of the first pyramid model, i.e., feature points to be tracked, an optical flow tracking operation is further performed on these feature points to be tracked using an optical flow algorithm of a more complex second pyramid model to obtain second tracking points. Because the optical flow algorithm of the second pyramid model has a wider and more accurate search range, feature points to be tracked that fail to be tracked using the optical flow algorithm of the less complex first pyramid model can be successfully tracked to corresponding second tracking points in the second image using the optical flow algorithm of the second pyramid model. This increases the number of second feature points subsequently determined based on the first and second tracking points and makes them more accurate, thereby improving the success rate of optical flow tracking. In other words, by combining the optical flow algorithms of the first and second pyramid models, the success rate of optical flow tracking can be guaranteed while minimizing the amount of computation, thereby improving the efficiency of optical flow tracking.

[0126] Optionally, determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm includes:

[0127] Determining a third tracking point tracked in the second image using a preset optical flow algorithm based on the first feature point and the predicted point position;

[0128] Determining, according to a reverse optical flow tracking algorithm, the third tracking point that can be reversely tracked to the corresponding first feature point as a target tracking point;

[0129] Determine a second feature point in the second image based on the target tracking point.

[0130] In the embodiment of the present application, an optical flow tracking operation is first performed according to a preset optical flow algorithm based on the first feature point and its corresponding predicted point position to determine a tracking point in the second image corresponding to the first feature point. For purposes of distinction, this tracking point is referred to as a third tracking point. In one embodiment, this third tracking point may include the first tracking point and the second tracking point described above, that is, the third tracking point may include each tracking point in the aforementioned tracking point set.

[0131] After the third tracking point is determined, reverse optical flow tracking is performed according to a single-layer reverse optical flow tracking algorithm. The reverse optical flow tracking algorithm is an algorithm that uses the third tracking point in the second image and, according to a preset optical flow algorithm, reversely tracks the points associated with the third tracking point in the first image. Specifically, based on the third tracking point in the second image, the position of the third tracking point corresponding to the first feature point in the first image is used as the initial position of optical flow tracking, and the points matching the third tracking point in the first image are tracked as reverse tracking feature points; the distance between the reverse tracking feature point and the first feature point originally corresponding to the third tracking point is calculated. When the distance is less than or equal to a preset first preset distance, it is determined that the third tracking point can be successfully reversely tracked to the corresponding first feature point, and the third tracking point is determined as the target tracking point. Conversely, if the distance is greater than the first preset distance, it is determined that the third tracking point cannot be reversely tracked to the first feature point, and it is removed as an outlier and not included in the target tracking points.

[0132] Afterwards, each obtained target tracking point is directly used as the second feature point of the second image, or the second feature point of the second image is obtained after removing the external points according to the target tracking point.

[0133] In an embodiment of the present application, after forward tracking the corresponding third tracking point in the second image based on the first feature point of the first image and the corresponding predicted point position in the second image, the reverse optical flow tracking algorithm is used to determine the third feature point that can be reversely tracked to the corresponding first feature point as the target tracking point. Therefore, the accuracy of optical flow tracking can be further improved through reverse verification, so that the second feature point determined based on the target tracking point is more accurate.

[0134] Optionally, determining a second feature point in the second image according to the target tracking point includes:

[0135] determining an essential matrix between the first image and the second image based on the target tracking point and the first feature point corresponding thereto;

[0136] determining, according to the essential matrix, an epipolar line corresponding to the first feature point in the second image;

[0137] Determine second feature points in the second image based on the distances between each of the target tracking points and the epipolar line; wherein the second feature points include target tracking points whose distances from the epipolar line are less than or equal to a second preset distance.

[0138] In an embodiment of the present application, for a target tracking point determined in the second image, the first normalized coordinates of the first feature point corresponding to the target tracking point can be calculated, as well as the second normalized coordinates of the target tracking point. Subsequently, based on the first normalized coordinates and the target normalized coordinates, an essential matrix in the first image and the second image is calculated. The essential matrix is a matrix used to describe the coordinate position correspondence between the first image and the second image in space. In one embodiment, the essential matrix can be obtained using a random sample consensus algorithm (RANSAC).

[0139] Next, based on the essential matrix, the epipolar lines corresponding to each first feature point in the first image in the second image are determined. Specifically, let the target tracking point corresponding to the first feature point P1 in the first image be P2 in the second image, the spatial point corresponding to the first feature point P1 and the second feature point P2 in the normalized coordinate system be P3, the plane formed by these three points P1, P2, and P3 be S, and let the line L1 be the line where plane S intersects with the plane of the second image. Then, this line L1 is the epipolar line corresponding to the first feature point P1 in the second image.

[0140] After determining the epipolar line corresponding to each first feature point in the second image, the distance between the target tracking point corresponding to each first feature point and its epipolar line is calculated. When the distance between the target tracking point and its corresponding epipolar line is less than or equal to a second predetermined distance, the target tracking point is considered a second feature point in the second image. Conversely, when the distance between the target tracking point and its corresponding epipolar line is greater than the second predetermined distance, the target tracking point is considered an outlier and is removed.

[0141] In the embodiment of the present application, after determining the target tracking point, the external points can be further eliminated by obtaining the essential matrix and the epipolar line, and the target tracking point that meets the conditions can be accurately used as the second feature point, thereby further improving the accuracy of optical flow tracking.

[0142] Optionally, after determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm, the method further includes:

[0143] If the number of the second feature points is less than a preset threshold, feature points of the second image are extracted using a preset feature point extraction algorithm to obtain new feature points; wherein the sum of the number of the second feature points and the new feature points is equal to the preset threshold.

[0144] In the embodiment of the present application, the preset threshold may be equal to the preset number described in step S101, or may be the preset number multiplied by a certain ratio (eg, three quarters), or may be other values set in advance according to actual conditions.

[0145] After determining the second feature points in the second image using a preset optical flow algorithm, if the number of these second feature points is less than a preset threshold, it is determined that there are too many feature points that have failed tracking, resulting in an insufficient number of second feature points to support optical flow tracking for the next frame. In this case, additional feature point extraction can be performed on the second image using a preset feature point extraction algorithm to obtain new feature points, such that the sum of the number of these new feature points and the tracked second feature points equals the preset threshold to meet the optical flow tracking requirements for the next frame.

[0146] Exemplarily, the preset feature point extraction algorithm may be a Hrarris corner detection algorithm, a Fast feature point extraction algorithm, or a Scale Invariant Feature Transform (SIFT) algorithm.

[0147] In the embodiment of the present application, since additional feature points can be extracted through a preset feature point extraction algorithm when the number of second feature points is small, the number of feature points of the second image is guaranteed, and the subsequent optical flow tracking can be accurately continued.

[0148] As an example but not a limitation, the electronic device is a robot, and the schematic diagram of the robot is as follows Figure 2 As shown, it includes a sensor synchronization module 21 and an optical flow tracking module 22. The sensor synchronization module 21 includes two sensors, a gyroscope and a wheel odometer, as well as a camera for collecting images and a sensor synchronization unit. The optical flow tracking module 22 includes a two-layer optical flow tracking unit, a four-layer optical flow tracking unit, a reverse tracking unit, an outlier removal unit, an additional extraction unit, and a second feature point output unit. The operation process of these modules is as follows:

[0149] (1) After the robot's camera captures the first image and the second image, it sends the timestamp information of the first image and the timestamp information of the second image to the sensor synchronization unit, and the gyroscope transmits the collected angular velocity data and the translational velocity data collected by the wheel odometer to the sensor synchronization unit.

[0150] (2) The sensor synchronization unit determines the first moment when the first image is taken and the second moment when the second image is taken by obtaining the timestamp information when the first image and the second image are taken, thereby extracting the angular velocity data from the first moment to the second moment and the translation velocity data from the first moment to the second moment for integration operation, determining the target posture increment, and adding the first posture increment to obtain the predicted posture corresponding to the second moment.

[0151] (3) The two-layer optical flow tracking unit of the optical flow tracking module obtains a first image containing a preset number of first feature points and a second image transmitted by the camera in the sensor synchronization module, and obtains a predicted pose transmitted by the sensor synchronization unit. Based on the first feature points and the predicted pose, the predicted point position corresponding to the first feature point in the second image is determined; then, based on the first feature points of the first image and the corresponding predicted point position, optical flow tracking is performed using an optical flow algorithm of a two-layer pyramid model to determine the first tracking point tracked in the second image.

[0152] (4) The first image, the second image, the first feature point and the corresponding predicted point position, and the first tracking point are transmitted to the four-layer optical flow tracking unit. For the first feature point of the first image that failed to be tracked in the two-layer optical flow tracking unit, i.e., the feature point to be tracked, optical flow tracking is performed in the four-layer optical flow tracking unit using the optical flow algorithm of the four-layer pyramid model to determine the second tracking point tracked in the second image.

[0153] (5) The first image, the second image, the first tracking point, the second tracking point, and the first feature point are transmitted to the reverse tracking unit. The first tracking point and the second tracking point are first merged to obtain a third tracking point. The first feature point in the first image is reversely tracked based on the third tracking point in the second image, and the third feature point that is successfully tracked to the corresponding first feature point is used as the target tracking point.

[0154] (6) The first image, the second image and the target tracking point are transmitted to the external point removal unit. By solving the essential matrix and the epipolar line, the external points of the target tracking point are removed to obtain the second feature point in the second image.

[0155] (7) When the number of the second feature points after the external points are removed is less than the preset threshold, the additional extraction unit is instructed to extract a new feature point, and the new feature point is combined with the second feature point and transmitted to the second feature point output unit to obtain a new second feature point for output.

[0156] Example 2:

[0157] Figure 3 FIG1 shows a flow chart of an optical flow tracking method provided by an embodiment of the present application, wherein the execution subject of the optical flow tracking method is an electronic device. Figure 3 The optical flow tracing method shown is detailed as follows:

[0158] In S301 , a first image corresponding to a first moment and a second image corresponding to a second moment are acquired, wherein the first image includes a preset number of first feature points.

[0159] In S302, a predicted position and posture of the electronic device at the second moment is determined based on the motion data from the first moment to the second moment.

[0160] In S303 , based on the predicted pose, a predicted point position of the first feature point corresponding to the second image is determined.

[0161] In S304, a second feature point corresponding to the first feature point in the second image is determined by a preset optical flow algorithm based on the first feature point and the predicted point position.

[0162] Optionally, the motion data includes sensor data and / or motion state information, and step S302 includes:

[0163] Obtaining a first posture corresponding to the electronic device at the first moment;

[0164] Determining a target position increment corresponding to the electronic device at the second moment based on sensor data and / or motion state information from the first moment to the second moment;

[0165] Determine a predicted posture of the electronic device at the second moment according to the target posture increment and the first posture.

[0166] Optionally, step S303 includes:

[0167] According to the first three-dimensional feature points corresponding to the first three-dimensional feature points in the preset three-dimensional map, and the predicted posture, the projection position of each first three-dimensional feature point is projected to the second image as the corresponding predicted point position; wherein, the preset three-dimensional map is a three-dimensional map constructed based on each frame image collected at the first moment and before the first moment.

[0168] Optionally, the step S304 includes:

[0169] According to the first feature point and the predicted point position, a second feature point in the second image is determined by running a preset optical flow algorithm in a preset thread pool; wherein the preset thread pool includes at least two threads.

[0170] Optionally, step S304 includes:

[0171] Determine, based on the first feature point and the predicted point position, a first tracking point in the second image corresponding to the first feature point using an optical flow algorithm of a first pyramid model;

[0172] If a feature point to be tracked exists in the first image, determining a second tracking point in the second image using an optical flow algorithm of a second pyramid model based on the feature point to be tracked and a position of a predicted point corresponding to the feature point to be tracked; wherein the feature point to be tracked is a first feature point for which no corresponding first tracking point exists; and the number of levels of the second pyramid model is greater than the number of levels of the first pyramid model;

[0173] A second feature point in the second image is determined based on the first tracking point and the second tracking point.

[0174] Optionally, step S304 includes:

[0175] Determining a third tracking point tracked in the second image using a preset optical flow algorithm based on the first feature point and the predicted point position;

[0176] Determining, according to a reverse optical flow tracking algorithm, the third tracking point that can be reversely tracked to the corresponding first feature point as a target tracking point;

[0177] Determine a second feature point in the second image based on the target tracking point.

[0178] Optionally, after step S304, the method further includes:

[0179] If the number of the second feature points is less than a preset threshold, feature points of the second image are extracted using a preset feature point extraction algorithm to obtain new feature points; wherein the sum of the number of the second feature points and the new feature points is equal to the preset threshold.

[0180] It should be noted that since the electronic device implementation steps of the method embodiment of this application and the previous embodiment are based on the same concept, its specific functions and technical effects can be found in the relevant description in Example 1 and will not be repeated here.

[0181] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0182] Example 3:

[0183] Figure 4 The following is a schematic diagram of the structure of an optical flow tracking device provided by an embodiment of the present application, which is applied to an electronic device. For ease of explanation, only the parts related to the embodiment of the present application are shown:

[0184] The optical flow tracking device includes: an image acquisition module 41, a predicted pose determination module 42, a predicted point position determination module 43, and a second feature point determination module 44.

[0185] The image acquisition module 41 is configured to acquire a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points.

[0186] The predicted posture determination module 42 is used to determine the predicted posture corresponding to the electronic device at the second moment based on the motion data from the first moment to the second moment.

[0187] The predicted point position determination module 43 is configured to determine the predicted point position of the first feature point corresponding to the second image according to the predicted posture.

[0188] The second feature point determination module 44 is configured to determine, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point using a preset optical flow algorithm.

[0189] Optionally, the motion data includes sensor data and / or motion state information, and the predicted posture determination module 42 is specifically used to determine the target posture increment corresponding to the electronic device at the second moment based on the sensor data and / or motion state information from the first moment to the second moment; and determine the predicted posture corresponding to the electronic device at the second moment based on the target posture increment and the first posture corresponding to the first moment determined in advance.

[0190] Optionally, the predicted point position determination module 43 is specifically used to project each of the first three-dimensional feature points to the second image as the corresponding predicted point position based on the first three-dimensional feature points corresponding to the first three-dimensional feature points in the preset three-dimensional map and the predicted posture; wherein the preset three-dimensional map is a three-dimensional map constructed based on each frame image collected at the first moment and before the first moment.

[0191] Optionally, the second feature point determination module 44 is specifically used to determine the second feature point in the second image based on the first feature point and the predicted point position by running a preset optical flow algorithm in a preset thread pool; wherein the preset thread pool includes at least two threads.

[0192] Optionally, the second feature point determination module 34 is specifically configured to determine, based on the first feature point and the predicted point position, a first tracking point corresponding to the first feature point in the second image using an optical flow algorithm of a first pyramid model; if there is a feature point to be tracked in the first image, determine, based on the feature point to be tracked and the predicted point position corresponding to the feature point to be tracked, a second tracking point in the second image using an optical flow algorithm of a second pyramid model; wherein the feature point to be tracked is a first feature point for which there is no corresponding first tracking point; the number of levels of the second pyramid model is greater than the number of levels of the first pyramid model; and determine the second feature point in the second image based on the first tracking point and the second tracking point.

[0193] Optionally, the second feature point determination module 34 is specifically used to determine the third tracking point tracked in the second image based on the first feature point and the predicted point position through a preset optical flow algorithm; determine the third tracking point that can be reversely tracked to the corresponding first feature point as the target tracking point based on the reverse optical flow tracking algorithm; and determine the second feature point in the second image based on the target tracking point.

[0194] Optionally, the optical flow tracking device further includes:

[0195] A feature point extraction module is used to extract feature points from the second image using a preset feature point extraction algorithm to obtain new feature points if the number of the second feature points is less than a preset threshold; wherein the sum of the number of the second feature points and the new feature points is equal to the preset threshold.

[0196] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as Example 1. Their specific functions and technical effects can be found in the relevant description in Example 1 and will not be repeated here.

[0197] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0198] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0199] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0200] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0201] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0202] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0203] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the following steps are implemented: Acquire a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points; Determining a predicted position and posture of the electronic device at the second moment based on the motion data from the first moment to the second moment; Determining, based on the predicted pose, a predicted point position of the first feature point corresponding to the second image; According to the first feature point and the predicted point position, a second feature point corresponding to the first feature point in the second image is determined by a preset optical flow algorithm.

2. The electronic device according to claim 1, wherein The motion data includes sensor data and / or motion state information. When the processor executes the computer program, determining the predicted position and posture of the electronic device at the second moment based on the motion data from the first moment to the second moment includes: Obtaining a first posture corresponding to the electronic device at the first moment; Determining a target position increment corresponding to the electronic device at the second moment based on sensor data and / or motion state information from the first moment to the second moment; Determine a predicted posture of the electronic device at the second moment according to the target posture increment and the first posture.

3. The electronic device according to claim 1, wherein When the processor executes the computer program, determining, based on the predicted pose, a predicted point position of the first feature point corresponding to the second image includes: According to the first three-dimensional feature points corresponding to the first three-dimensional feature points in the preset three-dimensional map, and the predicted posture, the projection position of each first three-dimensional feature point is projected to the second image as the corresponding predicted point position; wherein, the preset three-dimensional map is a three-dimensional map constructed based on each frame image collected at the first moment and before the first moment.

4. The electronic device according to claim 1, wherein When the processor executes the computer program, determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm includes: According to the first feature point and the predicted point position, a second feature point in the second image is determined by running a preset optical flow algorithm in a preset thread pool; wherein the preset thread pool includes at least two threads.

5. The electronic device according to claim 1, wherein When the processor executes the computer program, determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm includes: Determine, based on the first feature point and the predicted point position, a first tracking point in the second image corresponding to the first feature point using an optical flow algorithm of a first pyramid model; If a feature point to be tracked exists in the first image, determining a second tracking point in the second image using an optical flow algorithm of a second pyramid model based on the feature point to be tracked and a position of a predicted point corresponding to the feature point to be tracked; wherein the feature point to be tracked is a first feature point for which no corresponding first tracking point exists; and the number of levels of the second pyramid model is greater than the number of levels of the first pyramid model; A second feature point in the second image is determined based on the first tracking point and the second tracking point.

6. The electronic device according to claim 1, wherein When the processor executes the computer program, determining, based on the first feature point and the predicted point position, a second feature point in the second image corresponding to the first feature point by using a preset optical flow algorithm includes: Determining a third tracking point tracked in the second image using a preset optical flow algorithm based on the first feature point and the predicted point position; Determining, according to a reverse optical flow tracking algorithm, the third tracking point that can be reversely tracked to the corresponding first feature point as a target tracking point; Determine a second feature point in the second image based on the target tracking point.

7. The electronic device according to any one of claims 1 to 6, wherein: When the processor executes the computer program, the following steps are also implemented: If the number of the second feature points is less than a preset threshold, feature points of the second image are extracted using a preset feature point extraction algorithm to obtain new feature points; wherein the sum of the number of the second feature points and the new feature points is equal to the preset threshold.

8. An optical flow tracking method, characterized in that: include: Acquire a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points; Determining a predicted position and posture of the electronic device at the second moment based on the motion data from the first moment to the second moment; Determining, based on the predicted pose, a predicted point position of the first feature point corresponding to the second image; According to the first feature point and the predicted point position, a second feature point corresponding to the first feature point in the second image is determined by a preset optical flow algorithm.

9. An optical flow tracking device, characterized in that: include: An image acquisition module, configured to acquire a first image corresponding to a first moment and a second image corresponding to a second moment, wherein the first image includes a preset number of first feature points; a predicted posture determination module, configured to determine a predicted posture corresponding to the electronic device at the second moment based on the motion data from the first moment to the second moment; a predicted point position determination module, configured to determine the predicted point position of the first feature point corresponding to the second image based on the predicted pose; The second feature point determination module is used to determine a second feature point corresponding to the first feature point in the second image through a preset optical flow algorithm based on the first feature point and the predicted point position.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the electronic device implements the steps of the method according to claim 8.

Images