Image tracking method and system, pose fusion method and system, device and medium

Abstract

The application provides an image tracking method and system, a pose fusion method and system, a device and a medium. The image tracking method comprises: performing feature point matching on a current image frame and a template to obtain a first matching result; caching the current image frame and the first matching result into an optical flow queue; tracking positions of feature points of a previous image frame in the current image frame by using an optical flow method to obtain at least one optical flow matching result of the current image frame and at least one previous image frame; obtaining at least one second matching result of the current image frame and the template according to the optical flow matching result and a fusion matching result of the previous image frame and the template; and performing weighted fusion on the first matching result and all second matching results to obtain a fusion matching result of the current image frame and the template. The application can avoid error accumulation caused by changes in extraction and matching of feature points of each frame, enhance image tracking effect, and realize robust and stable tracking effect.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of image recognition tracking, in particular to an image tracking method and system, a pose fusion method and system, equipment and a medium. BACKGROUND

[0002] The prior art scheme generally models the environment based on a positioning and map construction method (Simultaneous Localization and Mapping, SLAM), and identifies and tracks the to-be-recognized image, and based on a general image matching method, feature points of each frame image are matched with the to-be-recognized image, and then an Iterative Closest Point (ICP) algorithm or a Perspective-n-Point (PNP) method is used to solve the pose.

[0003] However, such a method will cause the extraction and matching of feature points of each frame to change, resulting in error accumulation, affecting the image tracking effect, and causing the finally calculated pose to obviously jitter, thereby affecting the final AR effect. SUMMARY

[0004] The embodiment of the present application provides an image tracking method and system, a pose fusion method and system, equipment and a medium to solve the problem of error accumulation caused by the extraction and matching of feature points of each frame changing, thereby affecting the image tracking effect.

[0005] In order to solve the above technical problems, the present application is implemented as follows:

[0006] In a first aspect, the embodiment of the present application provides an image tracking method, comprising:

[0007] Matching feature points of a current image frame and a template to obtain a first matching result, wherein the template is an image containing only a target object in the current image frame;

[0008] Caching the current image frame and the first matching result into an optical flow queue, wherein the optical flow queue further includes at least one previous image frame before the current image frame and a fusion matching result of the previous image frame and the template;

[0009] Tracking positions of feature points of the previous image frame in the current image frame using an optical flow method to obtain at least one optical flow matching result of the current image frame and the at least one previous image frame;

[0010] Obtaining at least one second matching result of the current image frame and the template according to the optical flow matching result and the fusion matching result of the previous image frame and the template;

[0011] The first matching result and all second matching results are weightedly fused to obtain a fused matching result of the current image frame and the template.

[0012] In the embodiments of the present application, optionally, the feature point matching of the current image frame and the template to obtain the first matching result comprises:

[0013] The feature points of the current image frame and the first descriptors corresponding to the feature points of the current image frame are obtained, and the feature points of the template and the second descriptors corresponding to the feature points of the template are obtained.

[0014] The first descriptors are compared with the second descriptors to obtain a descriptor matching relationship between the first descriptors and the second descriptors.

[0015] According to the descriptor matching relationship, a feature point matching relationship between the feature points of the current image frame and the feature points of the template is obtained, and the feature point matching relationship is the first matching result of the current image frame and the template.

[0016] In the embodiments of the present application, optionally, the obtaining of the first descriptors corresponding to the feature points of the current image frame comprises:

[0017] The current image frame is obtained.

[0018] The current image frame is subjected to a grayscale processing to obtain a grayscale image of the current image frame.

[0019] The grayscale image is subjected to a Gaussian filtering to obtain a filtered grayscale image.

[0020] All feature points of the filtered grayscale image are extracted.

[0021] The descriptor extraction is performed according to each feature point extracted from the current image frame to obtain a descriptor corresponding to each feature point.

[0022] In the embodiments of the present application, optionally, the obtaining of the feature points of the current image frame comprises:

[0023] All pixel points of the current image frame and pixel points in a N*N neighborhood of the pixel points are obtained, and the following operations are performed on a single pixel point to obtain a candidate feature point of the current image frame.

[0024] According to the pixel point and the neighborhood pixel points in the neighborhood of the pixel point, a first normalized gradient value of the pixel point in an X direction and a second normalized gradient value of the pixel point in a Y direction are determined.

[0025] The first normalized gradient value and the second normalized gradient value are fused to obtain a normalized gradient value of the pixel point.

[0026] determining whether the normalized gradient value of the pixel point is greater than normalized gradient values of all neighborhood pixel points next to the pixel point in an X direction and normalized gradient values of all neighborhood pixel points next to the pixel point in a Y direction;

[0027] if the normalized gradient value of the pixel point is greater than normalized gradient values of all neighborhood pixel points next to the pixel point in an X direction and normalized gradient values of all neighborhood pixel points next to the pixel point in a Y direction, taking the pixel point as a candidate feature point of the current image frame;

[0028] taking a feature point with a normalized gradient value within a specified range in the candidate feature points as a feature point of the current image frame.

[0029] In the embodiments of the present application, after the current image frame and the first matching result are cached into the optical flow queue, the method further comprises:

[0030] determining whether the number of image frames in the optical flow queue is greater than a specified value;

[0031] if the number of image frames in the optical flow queue is greater than the specified value, deleting an image frame first entering the queue in the optical flow queue and a fusion matching result of the first entering queue image frame and a template.

[0032] In the embodiments of the present application, after the fusion matching result of the current image frame and the template is obtained, the method further comprises:

[0033] updating the first matching result corresponding to the current image frame in the optical flow queue to the fusion matching result of the current image frame and the template.

[0034] In the embodiments of the present application, the method further comprises:

[0035] if feature point matching is performed on the current image frame and the template, no first matching result is obtained;

[0036] tracking a previous image frame of the current image frame using an optical flow method to obtain a first optical flow matching result;

[0037] saving the first optical flow matching result as the first matching result into the optical flow queue.

[0038] In a second aspect, the embodiments of the present application provide a pose fusion method, comprising:

[0039] determine a first pose of the current image frame according to a fusion matching result of the current image frame and the template and a corresponding relationship between 2D points of the current image frame and 3D points of the template, wherein the fusion matching result is obtained according to the image tracking method of any one of the first aspect;

[0040] obtain a first rotation amount of a previous image frame and a first rotation amount of the current image frame by using a sensor;

[0041] obtain a second rotation amount of the previous image frame and a second rotation amount of the current image frame by using the first pose;

[0042] obtain a first rotation change amount according to the first rotation amount of the previous image frame and the first rotation amount of the current image frame;

[0043] obtain a second rotation change amount according to the second rotation amount of the previous image frame and the second rotation amount of the current image frame;

[0044] perform weighted fusion on the first rotation change amount and the second rotation change amount to obtain a rotation fusion result;

[0045] obtain a rotated fusion pose of the current image frame according to the rotation fusion result.

[0046] In the embodiments of the present application, optionally, the performing weighted fusion on the first rotation change amount and the second rotation change amount to obtain a rotation fusion result comprises:

[0047] convert the first rotation change amount and the second rotation change amount into quaternion form to obtain a first rotation change amount quaternion and a second rotation change amount quaternion;

[0048] perform weighted fusion on the first rotation change amount quaternion and the second rotation change amount quaternion to obtain a fused rotation change amount quaternion result;

[0049] convert the fused rotation change amount quaternion result into a rotation matrix;

[0050] perform fusion on the rotation matrix and the second rotation amount of the current image frame to obtain the rotation fusion result.

[0051] In a third aspect, the embodiments of the present application provide an image tracking system, comprising:

[0052] a first matching module configured to perform feature point matching on a current image frame and a template to obtain a first matching result, wherein the template is an image containing only a target object in the current image frame;

[0053] a queue module, configured to cache the current image frame and the first matching result into an optical flow queue, wherein the optical flow queue further comprises at least one previous image frame before the current image frame and a fusion matching result of the previous image frame and the template;

[0054] a second matching module, configured to track positions of feature points of the previous image frame in the current image frame using an optical flow method, to obtain at least one optical flow matching result of the current image frame and the at least one previous image frame;

[0055] a first fusion module, configured to obtain at least one second matching result of the current image frame and the template according to the optical flow matching result and the fusion matching result of the previous image frame and the template;

[0056] a second fusion module, configured to perform weighted fusion on the first matching result and all the second matching results, to obtain a fusion matching result of the current image frame and the template.

[0057] In a fourth aspect, an embodiment of the present application provides a pose fusion system, comprising:

[0058] a first pose module, configured to determine a first pose of the current image frame according to a fusion matching result of the current image frame and the template and a corresponding relationship between 2D points of the current image frame and 3D points of the template, wherein the fusion matching result is obtained according to the image tracking method in any one of the first aspect;

[0059] a first rotation module, configured to obtain a first rotation amount of a previous image frame and a first rotation amount of the current image frame by using a sensor;

[0060] a second rotation module, configured to obtain a second rotation amount of the previous image frame and a second rotation amount of the current image frame by using the first pose;

[0061] a first rotation change module, configured to obtain a first rotation change amount according to the first rotation amount of the previous image frame and the first rotation amount of the current image frame;

[0062] a second rotation change module, configured to obtain a second rotation change amount according to the second rotation amount of the previous image frame and the second rotation amount of the current image frame;

[0063] a rotation fusion module, configured to perform weighted fusion on the first rotation change amount and the second rotation change amount, to obtain a rotation fusion result;

[0064] a second pose module, configured to obtain a pose of the current image frame after rotation fusion according to the rotation fusion result.

[0065] In a fifth aspect, an electronic device is provided, which includes a processor, a memory, and a program stored in the memory and capable of running on the processor. When the program is executed by the processor, the steps of the image tracking method according to the first aspect are implemented. Alternatively, when the program is executed by the processor, the steps of the pose fusion method according to the second aspect are implemented.

[0066] In a sixth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the image tracking method according to the first aspect are implemented. Alternatively, when the computer program is executed by a processor, the steps of the pose fusion method according to the second aspect are implemented.

[0067] In the present application, a first matching result is obtained by performing feature point matching on a current image frame and a template, wherein the template is an image containing only a target object in the current image frame; a previous image frame and the first matching result are cached into an optical flow queue, wherein the optical flow queue further includes at least one previous image frame before the current image frame and a fusion matching result of the previous image frame and the template; a position of a feature point of the previous image frame in the current image frame is tracked using an optical flow method, and at least one optical flow matching result of the current image frame and the at least one previous image frame is obtained; at least one second matching result of the current image frame and the template is obtained according to the optical flow matching result and the fusion matching result of the previous image frame and the template; and finally, the first matching result and all the second matching results are weighted and fused to obtain a fusion matching result of the current image frame and the template. The first matching result obtained by performing feature point matching with the template and the optical flow matching result obtained by the optical flow method are weighted and fused, so as to reduce the error of the fusion matching result of the previous image frame and the template, and further avoid error accumulation caused by changes in extraction and matching of feature points of each frame, thereby enhancing the image tracking effect and achieving robust and stable tracking effect. BRIEF DESCRIPTION OF DRAWINGS

[0068] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of preferred embodiments, and are not meant to limit the present application. Moreover, the same reference numerals in different figures represent the same or similar components. In the drawings:

[0069] Figure 1 is a flowchart of an image tracking method provided by an embodiment of the present application;

[0070] Figure 2 is a flowchart of a pose fusion method based on image tracking provided by an embodiment of the present application;

[0071] Figure 3is a schematic diagram of a 3*3 neighborhood pixel point provided by an embodiment of the present application;

[0072] Figure 4 is a schematic diagram of obtaining a fusion matching result provided by an embodiment of the present application;

[0073] Figure 5 is a flowchart of a pose fusion method provided by an embodiment of the present application;

[0074] Figure 6 is a structural schematic diagram of an image tracking system provided by an embodiment of the present application;

[0075] Figure 7 is a structural schematic diagram of a pose fusion system provided by an embodiment of the present application;

[0076] Figure 8 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0078] Please refer to Figure 1 and Figure 2 An image tracking method provided by an embodiment of the present application comprises the following steps.

[0079] Step 11: performing feature point matching on a current image frame and a template to obtain a first matching result, wherein the template is an image containing only a target object in the current image frame.

[0080] The template can be saved in a database, and different templates containing different objects can be saved in the database, and the different templates are distinguished by different IDs.

[0081] Step 12: caching the current image frame and the first matching result into an optical flow queue, wherein the optical flow queue further comprises at least one previous image frame before the current image frame and a fusion matching result of the previous image frame and the template.

[0082] Step 13: tracking positions of feature points of the previous image frame in the current image frame using an optical flow method to obtain at least one optical flow matching result of the current image frame and the at least one previous image frame.

[0083] Further, all the previous image frames cached in the queue can be used in sequence to perform optical flow matching with the current image frame to obtain corresponding optical flow matching results.

[0084] Step 14: obtaining at least one second matching result of the current image frame and the template according to the optical flow matching result and the fusion matching result of the previous image frame and the template.

[0085] Step 15: performing weighted fusion on the first matching result and all the second matching results to obtain a fusion matching result of the current image frame and the template.

[0086] In the present application, a first matching result is obtained by performing feature point matching on the current image frame and a template, wherein the template is an image containing only a target object in the current image frame; then a previous image frame and a fusion matching result of the previous image frame and the template are cached in an optical flow queue, wherein the optical flow queue also includes at least one previous image frame before the current image frame and the fusion matching result of the previous image frame and the template; at least one optical flow matching result of the current image frame and the at least one previous image frame is obtained by using an optical flow method to track the positions of feature points of the previous image frame in the current image frame; at least one second matching result of the current image frame and the template is obtained according to the optical flow matching result and the fusion matching result of the previous image frame and the template; finally, a fusion matching result of the current image frame and the template is obtained by performing weighted fusion on the first matching result and all the second matching results. By performing weighted fusion on the first matching result obtained by feature point matching with the template and the optical flow matching result obtained by the optical flow method, the error of the fusion matching result of the previous image frame and the template is reduced, thereby avoiding error accumulation caused by changes in extraction and matching of feature points of each frame, enhancing image tracking effect, and achieving robust and stable tracking effect.

[0087] In the present application, the feature point matching on the current image frame and the template to obtain the first matching result includes:

[0088] obtaining feature points of the current image frame and first descriptors corresponding to the feature points of the current image frame, and feature points of the template and second descriptors corresponding to the feature points of the template;

[0089] comparing the first descriptors with the second descriptors to obtain a descriptor matching relationship between the first descriptors and the second descriptors;

[0090] obtaining a feature point matching relationship between the feature points of the current image frame and the feature points of the template according to the descriptor matching relationship, wherein the feature point matching relationship is the first matching result of the current image frame and the template.

[0091] In the embodiment of the present application, the matching relationship is obtained by matching the first descriptor corresponding to the extracted current image frame and the second descriptor corresponding to the template, so as to obtain the corresponding relationship between the current image frame feature 2D point (2D point coordinates, descriptor) and the template feature 3D point (3D point coordinates, descriptor and ID).

[0092] The matching relationship obtained by matching between the descriptors has good accuracy and reliability, and the similarity of a specific region in an image can be quickly found.

[0093] The embodiment of the present application provides a method for calculating the matching relationship. Taking a 96-bit binary descriptor composed of 0 and 1 as an example, the first descriptor and the second descriptor are compared 1v1, and the Manhattan distance is used to calculate the similarity algorithm, and the calculation formula is as follows:

[0094]

[0095] Wherein, featImg j is the jth first descriptor corresponding to the feature point extracted from the current image frame, featTmp i,j is the jth second descriptor corresponding to the ith template feature point, and fabs represents the absolute value.

[0096] The matching relationship of the descriptor obtained by using the Manhattan distance is more accurate, has strong fault tolerance, can resist the influence of noise, and has faster operation speed, and can effectively run in the case of limited performance and resources such as Web.

[0097] Please refer to Figure 2 In the embodiment of the present application, optionally, the method for obtaining the feature points of the current image frame and the first descriptor corresponding to the feature points of the current image frame comprises:

[0098] Obtaining the current image frame;

[0099] Performing gray scale processing on the current image frame to obtain a gray scale image of the current image frame;

[0100] Performing Gaussian filtering on the gray scale image to obtain a filtered gray scale image;

[0101] Extracting all feature points of the filtered gray scale image;

[0102] Extracting a descriptor for each feature point extracted from the current image frame to obtain a descriptor corresponding to each feature point.

[0103] In the embodiment of the present application, the filtered gray image can reduce the influence of noise on the subsequent feature point and descriptor extraction, and the gray image after gray processing can also process the difficulty, by performing gray processing on the acquired color image frame, reducing the steps of complex processing of the image, and improving the processing speed.

[0104] Further, the embodiment of the present application can extract feature points in real time by acquiring real-time image frames, thereby realizing real-time image tracking. For example, in the case of 2D image tracking based on Web-based Augmented Reality (WebAR), a video stream image frame can be parsed from the camera to acquire real-time image frames.

[0105] In the embodiment of the present application, the FREAK (Fast Retina Key point) method can be used for feature extraction to obtain the feature points of the filtered gray image and the FREAK feature descriptor. The FREAK method can resist the influence of noise, making the extraction result more accurate, faster in calculation, and less in memory consumption, and can perform better in the case of limited performance and resources.

[0106] Further, the embodiment of the present application also provides a specific method for feature point extraction:

[0107] In the embodiment of the present application, optionally, the acquiring of the feature points of the current image frame comprises:

[0108] All pixel points of the current image frame and pixel points in the N*N neighborhood of the pixel points are acquired, and the following operations are performed on a single pixel point to obtain the candidate feature points of the current image frame:

[0109] According to the pixel point and the neighborhood pixel points in the neighborhood of the pixel point, a first normalized gradient value of the pixel point in the X direction and a second normalized gradient value of the pixel point in the Y direction are determined;

[0110] The first normalized gradient value and the second normalized gradient value are fused to obtain the normalized gradient value of the pixel point;

[0111] It is judged whether the normalized gradient value of the pixel point is greater than the normalized gradient values of all neighborhood pixel points adjacent to the pixel point in the X direction and the normalized gradient values of all neighborhood pixel points adjacent to the pixel point in the Y direction;

[0112] If the normalized gradient value of the pixel point is greater than the normalized gradient values of all neighborhood pixel points next to the pixel point in the X direction and the normalized gradient values of all neighborhood pixel points next to the pixel point in the Y direction, the pixel point is taken as a candidate feature point of the current image frame;

[0113] The feature point with the normalized gradient value in the specified range in the candidate feature points is taken as a feature point of the current image frame.

[0114] In the embodiments of the present application, the neighborhood pixel points are used for feature point extraction, which can obtain accurate image feature points while avoiding the influence of single pixel point noise, and has a faster processing speed.

[0115] For example, the adjacent N*N neighborhood is 3*3 neighborhood, please refer to Figure 3 Each pixel point and the pixel points in the adjacent 3*3 size region are marked, wherein X represents the current pixel, and the other 8 pixel points P 11 …P 33 The marking is as shown in Figure 3 The present application also provides a calculation formula for determining the normalized gradient corresponding to the current pixel point.

[0116] d x =(P 13 -P 11 +P 23 -P 21 +P 33 -P 31 ) / 3.0 / 256

[0117] d y =(P 31 -P 11 +P 32 -P 12 +P 33 -P 13 ) / 3.0 / 256

[0118] d=sqrt((d x *d x +d y *d y ) / 2.0)

[0119] Wherein, d x is the first normalized gradient value of the current pixel point in the X direction, and the value range is [0, 1]; d y is the second normalized gradient value of the current pixel point in the Y direction, and the value range is [0, 1]; d is the normalized gradient value corresponding to the current pixel point; P 11 …P 33 is the pixel value of the corresponding marked point.

[0120] Referring to Figure 3 , if the normalized gradient value of the current pixel point X is greater than P 12 , P 23 , P 32 , P 21 The normalized gradient value of the position pixel point, the current pixel point X is taken as the feature point of the current image frame.

[0121] Further, the embodiment of the application further provides a method for further screening the obtained feature points of the current image frame. The feature points of the current image frame obtained through the above-mentioned neighborhood pixel points are taken as candidate feature points. The gradient values corresponding to the candidate feature points are histogram counted, and the feature points with smaller gradient values are removed by 10% to obtain more optimal feature points. Specifically:

[0122] (a) multiplying the gradient value of the candidate feature point by 100 and taking an integer, so that the gradient value D i is in the range of [0, 100], wherein i represents the i-th candidate feature point;

[0123] (b) using an array with a size of 100 to count the gradients of the candidate feature points, and the pseudo code is as follows, wherein N represents the total number of candidate feature points.

[0124] for i←0to N

[0125] do Bin[Di]←Bin[Di]+1

[0126] (c) counting the histogram array, removing the feature points with smaller gradient values by 10%, and the pseudo code is as follows. After obtaining i, i is used as a threshold to delete the candidate feature points with gradient values less than i, and the remaining feature points are the feature points of the current image frame.

[0127] S←0

[0128] i←99

[0129] for i←99to 0

[0130] if S<0.9*N

[0131] S←S+Bin[Di]

[0132] else break

[0133] The feature points of the current image frame obtained by further screening the candidate feature points are used for feature point matching between the current image frame and the template to obtain a first matching result, and are also used for tracking the positions of the feature points of the previous image frame in the current image frame by using the optical flow method to obtain at least one optical flow matching result between the current image frame and at least one previous image frame.

[0134] Please refer to Figure 4 In the embodiment of the present application, after the current image frame and the first matching result are cached into the optical flow queue, the method further comprises:

[0135] determining whether the number of image frames in the optical flow queue is greater than a specified value;

[0136] if the number of image frames in the optical flow queue is greater than the specified value, deleting the image frame that first enters the queue and the fusion matching result of the image frame that first enters the queue and the template in the optical flow queue.

[0137] In the embodiment of the present application, the optical flow between frames is matched in the form of an optical flow queue. In order to avoid too many image frames in the queue affecting the image processing speed, the number of image frames in the optical flow queue is limited.

[0138] Taking the specified value of the number of image frames in the optical flow queue as 5 for example, when the previous image frame and the first matching result are inserted into the optical flow queue to determine whether the number of cached data in the current optical flow queue is greater than 5, if it is greater than 5, the oldest data is removed, that is, an element is removed from the tail of the queue.

[0139] In the embodiment of the present application, the optical flow matching is performed between the 5 cached previous image frames in the queue and the current image frame, that is, the feature points of the first four frames correspond to the positions in the current image frame to obtain the corresponding matching relationship Match i wherein i represents the optical flow matching result of the i th previous image frame in the optical flow queue.

[0140] Further, the first matching result and all second matching results are weighted and fused according to the matching relationship corresponding to the 3D point ID to obtain the fusion matching result of the current image frame and the template, specifically:

[0141] (1) all 3D point IDs corresponding to the current image frame matching result are obtained, that is, all target objects;

[0142] (2) the optical flow method matching results with the same 3D point ID (the same target object) are weighted and averaged, and the specific formula is as follows:

[0143]

[0144] wherein M represents the number of 2D matching points with the same 3D point ID as the current feature point, wherein x, y are coordinates of the current feature point, x i , y i are coordinates of the matching point corresponding to the current feature point.

[0145] The embodiment of the present application solves the problem of multiple matching results of a feature point of a target object with one ID in a current image frame.

[0146] Further, the weighted fusion can assign weights according to the distance between the previous image frame and the current image frame. For example, if the current frame is the 8th image frame, the weight of the matching relationship obtained according to the 7th image frame can be higher than the weight of the matching relationship obtained according to the 6th image frame. The specific weight distribution and weight are not limited in the embodiment of the present application.

[0147] After obtaining the fused matching result, in the embodiment of the present application, the method further includes, optionally:

[0148] updating the first matching result corresponding to the current image frame in the optical flow queue to the fused matching result of the current image frame and the template.

[0149] Finally, after obtaining the fused matching result of the current image frame, the data of the optical flow queue can be updated for optical flow tracking of the next image frame, and the error can be reduced by updating the data in the optical flow queue, so that the image tracking result is more accurate.

[0150] In the embodiment of the present application, the method further includes, optionally:

[0151] if the feature point matching of the current image frame and the template is not performed, the first matching result is not obtained;

[0152] tracking the previous image frame of the current image frame using the optical flow method to obtain a first optical flow matching result;

[0153] saving the first optical flow matching result as the first matching result to the optical flow queue.

[0154] In the embodiment of the present application, the first optical flow matching result is saved as the first matching result to the optical flow queue, which avoids the abnormality of the optical flow matching when the first matching result is not obtained.

[0155] For reference Figure 5 , the embodiment of the present application also provides a pose fusion method, including:

[0156] Step 51: determining a first pose of the current image frame according to a fusion matching result of the current image frame and the template and a corresponding relationship between 2D points of the current image frame and 3D points of the template, wherein the fusion matching result is according to any one of the above.

[0157] Step 52: acquiring a first rotation amount of a previous image frame and a first rotation amount of the current image frame by using a sensor.

[0158] Wherein the first rotation amount of the previous image frame can be denoted as R The first rotation amount of the current image frame can be denoted as R

[0159] Step 53: acquiring a second rotation amount of the previous image frame and a second rotation amount of the current image frame by using the first pose.

[0160] Wherein the second rotation amount of the previous image frame can be denoted as R n-1 The second rotation amount of the current image frame can be denoted as R n .

[0161] Step 54: acquiring a first rotation change amount according to the first rotation amount of the previous image frame and the first rotation amount of the current image frame.

[0162] Wherein the first rotation change amount ΔR imu can be obtained by the following formula:

[0163]

[0164] Step 55: acquiring a second rotation change amount according to the second rotation amount of the previous image and the second rotation amount of the current image frame.

[0165] Wherein the second rotation change amount ΔR can be obtained by the following formula:

[0166] ΔR=(R n-1 ) T R n .

[0167] Step 56: weighted fusion of the first rotation change amount and the second rotation change amount to obtain a rotation fusion result.

[0168] Step 57: acquiring a rotation fused pose of the current image frame according to the rotation fusion result.

[0169] In the embodiment of the present application, according to the fusion matching result obtained by the image tracking method and the corresponding relationship between the current image frame 2D point and the template 3D point, the ICP algorithm can be used to calculate the six degrees of freedom (Six Degrees of Freedom, 6DOF) pose corresponding to the current frame.

[0170] Further, the inertial measurement unit (IMU) data may be dropped, and for the rotation amount, the IMU data alone is not accurate enough, and the pose solved by the ICP also has certain error. The embodiment of the present application can complement the IMU data and the fusion matching result obtained by the image tracking method by weighting, reduce the error accumulation in image tracking, obtain more stable rotation result, and further obtain better pose.

[0171] In the embodiment of the present application, optionally, the weighting fusion of the first rotation change amount and the second rotation change amount to obtain a rotation fusion result comprises:

[0172] Converting the first rotation change amount and the second rotation change amount into quaternion form to obtain a first rotation change amount quaternion and a second rotation change amount quaternion;

[0173] Weighting fusion of the first rotation change amount quaternion and the second rotation change amount quaternion to obtain a fused rotation change amount quaternion result;

[0174] Converting the fused rotation change amount quaternion result into a rotation matrix;

[0175] Fusing the rotation matrix and the second rotation amount of the current image frame to obtain the rotation fusion result.

[0176] In the embodiment of the present application, the two change amounts can be converted into quaternion form for weighting fusion, and the specific calculation formula is as follows:

[0177] ΔQ Result =α(ΔQ imu )+(1-α)(ΔQ);

[0178] Wherein, ΔQ imu and ΔQ are the first rotation change amount ΔR imu corresponding quaternion and the second rotation change amount ΔR corresponding quaternion, ΔQ Result is the fused quaternion result, and a is the fusion weight, generally taking a value of [0.5, 0.7].

[0179] Further, the fused rotation change amount quaternion result is converted into a rotation matrix form R ResultAnd combine the previous frame result R n-1 The final rotation result R n The specific calculation formula is as follows:

[0180] R n = R n-1 *R Result .

[0181] In this application, the estimation accuracy after weighting is improved and the calculation efficiency is improved by using quaternions for calculation.

[0182] Please refer to Figure 6 The application also provides an image tracking system 60, comprising:

[0183] A first matching module 61 is configured to perform feature point matching on a current image frame and a template to obtain a first matching result, wherein the template is an image containing only a target object in the current image frame.

[0184] A queue module 62 is configured to cache the current image frame and the first matching result into an optical flow queue, wherein the optical flow queue further comprises at least one previous image frame before the current image frame and a fusion matching result of the previous image frame and the template.

[0185] A second matching module 63 is configured to track positions of feature points of the previous image frame in the current image frame using an optical flow method to obtain at least one optical flow matching result of the current image frame and the at least one previous image frame.

[0186] A first fusion module 64 is configured to obtain at least one second matching result of the current image frame and the template according to the optical flow matching result and the fusion matching result of the previous image frame and the template.

[0187] A second fusion module 65 is configured to perform weighted fusion on the first matching result and all second matching results to obtain a fusion matching result of the current image frame and the template.

[0188] In the embodiments of the application, optionally, the first matching module 61 comprises:

[0189] A first matching sub-module is configured to obtain feature points of the current image frame and first descriptors corresponding to the feature points of the current image frame, and feature points of the template and second descriptors corresponding to the feature points of the template.

[0190] The first descriptors are compared with the second descriptors to obtain a descriptor matching relationship between the first descriptors and the second descriptors.

[0191] According to the description sub-matching relationship, a feature point matching relationship between the feature points of the current image frame and the feature points of the template is obtained, and the feature point matching relationship is a first matching result of the current image frame and the template.

[0192] In the embodiments of the present application, the first matching sub-module comprises:

[0193] The first matching sub-module is configured to obtain the current image frame.

[0194] The current image frame is subjected to a grayscale processing to obtain a grayscale image of the current image frame.

[0195] The grayscale image is subjected to a Gaussian filtering to obtain a filtered grayscale image.

[0196] All feature points of the filtered grayscale image are extracted.

[0197] A descriptor is extracted according to each feature point of the current image frame to obtain a corresponding descriptor of each feature point.

[0198] In the embodiments of the present application, the first matching sub-module comprises:

[0199] The first matching sub-module is configured to obtain all pixel points of the current image frame and pixel points in a N*N neighborhood of the pixel points, and perform the following operations on a single pixel point to obtain a candidate feature point of the current image frame:

[0200] According to the pixel point and the neighborhood pixel points in the neighborhood of the pixel point, a first normalized gradient value of the pixel point in an X direction and a second normalized gradient value of the pixel point in a Y direction are determined.

[0201] The first normalized gradient value and the second normalized gradient value are fused to obtain a normalized gradient value of the pixel point.

[0202] It is determined whether the normalized gradient value of the pixel point is greater than normalized gradient values of all neighborhood pixel points immediately adjacent to the pixel point in the X direction and normalized gradient values of all neighborhood pixel points immediately adjacent to the pixel point in the Y direction.

[0203] If the normalized gradient value of the pixel point is greater than the normalized gradient values of all neighborhood pixel points immediately adjacent to the pixel point in the X direction and the normalized gradient values of all neighborhood pixel points immediately adjacent to the pixel point in the Y direction, the pixel point is taken as a candidate feature point of the current image frame.

[0204] The feature points of the candidate feature points whose normalized gradient values are within a specified range are taken as feature points of the current image frame.

[0205] Optionally, the image tracking system 60 further comprises:

[0206] a queue determining module, configured to determine whether the number of image frames in the optical flow queue is greater than a specified value;

[0207] if the number of image frames in the optical flow queue is greater than the specified value, delete the image frame first entering the queue in the optical flow queue and the fusion matching result of the image frame first entering the queue and the template.

[0208] Optionally, the image tracking system 60 further comprises:

[0209] an updating module, configured to update the first matching result corresponding to the current image frame in the optical flow queue to the fusion matching result of the current image frame and the template.

[0210] Optionally, the image tracking system 60 further comprises:

[0211] a second queue module, configured to, if the feature point matching of the current image frame and the template is not performed, acquire the first matching result;

[0212] track the previous image frame of the current image frame using the optical flow method to acquire the first optical flow matching result;

[0213] save the first optical flow matching result as the first matching result into the optical flow queue.

[0214] The image tracking system provided in the embodiments of the present application can implement Figures 1 to 4 The method embodiments implement various processes and achieve the same technical effects. To avoid repetition, details are not described herein.

[0215] Please refer to Figure 7 The present application also provides a pose fusion system 70, comprising:

[0216] a first pose module 71, configured to determine a first pose of the current image frame according to the fusion matching result of the current image frame and the template and the corresponding relationship between the 2D points of the current image frame and the 3D points of the template, wherein the fusion matching result is obtained according to the image tracking method of any one of the above embodiments;

[0217] a first rotation module 72, configured to acquire the first rotation amount of the previous image frame and the first rotation amount of the current image frame using a sensor;

[0218] a second rotation module 73, configured to acquire the second rotation amount of the previous image frame and the second rotation amount of the current image frame using the first pose;

[0219] The first rotation transformation module 74 is used to obtain the first rotation transformation amount based on the first rotation amount of the previous image frame and the first rotation amount of the current image frame;

[0220] The second rotation transformation module 75 is used to obtain the second rotation transformation amount based on the second rotation amount of the previous frame image and the second rotation amount of the current image frame;

[0221] The rotation fusion module 76 is used to perform weighted fusion of the first rotation change and the second rotation change to obtain a rotation fusion result.

[0222] The second pose module 77 is used to obtain the pose of the current image frame after rotation fusion based on the rotation fusion result.

[0223] In this embodiment of the application, optionally, the rotation fusion module 76 includes:

[0224] The rotation fusion submodule is used to convert the first rotation change amount and the second rotation change amount into quaternion form to obtain the first rotation change amount quaternion and the second rotation change amount quaternion;

[0225] The first rotation change quaternion and the second rotation change quaternion are weighted and fused to obtain the fused rotation change quaternion result.

[0226] Convert the quaternion result of the fused rotation change into a rotation matrix;

[0227] The rotation fusion result is obtained by fusing the rotation matrix with the second rotation amount of the current image frame.

[0228] The image tracking system provided in this application embodiment can achieve... Figure 5 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0229] This application provides an electronic device 80, see [link to relevant documentation] Figure 8 As shown, Figure 8 This is a schematic block diagram of an electronic device 80 according to an embodiment of this application, including a processor 81, a memory 82, and a program or instructions stored in the memory 82 and executable on the processor 81. When the program or instructions are executed by the processor, they implement the steps in any image tracking method of this application; or, when the program or instructions are executed by the processor, they implement the steps in any pose fusion method of this application.

[0230] The embodiment of the present application provides a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize each process of the embodiment of the image tracking method or the pose fusion method of any one of the above, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0231] Computer readable media includes permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer readable medium does not include transitory computer readable medium, such as modulated data signals and carriers.

[0232] It should be noted that in this paper, the term "including", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0233] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0234] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software plus necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a service classification device (which can be a mobile phone, computer, server, air conditioner or network device) execute the method described in each embodiment of the present application.

[0235] The above merely preferred embodiments of the present application, it should be noted that for those of ordinary skill in the art without departing from the principles of the present application, can also make a number of improvements and refinements, these improvements and refinements should also be considered as the scope of protection of the present application.

Claims

1. An image tracking method characterized by, The method comprises the following steps: performing feature point matching on a current image frame and a template to obtain a first matching result, wherein the template is an image containing only a target object in the current image frame; caching the current image frame and the first matching result into an optical flow queue, wherein the optical flow queue further comprises at least one previous image frame before the current image frame and a fusion matching result of the previous image frame and the template; tracking positions of feature points of the previous image frame in the current image frame using an optical flow method to obtain at least one optical flow matching result of the current image frame and the at least one previous image frame; obtaining at least one second matching result of the current image frame and the template according to the optical flow matching result and the fusion matching result of the previous image frame and the template; performing weighted fusion on the first matching result and all second matching results to obtain a fusion matching result of the current image frame and the template; after the step of caching the current image frame and the first matching result into the optical flow queue, the method further comprises the following steps: determining whether the number of image frames in the optical flow queue is greater than a specified value; if the number of image frames in the optical flow queue is greater than the specified value, deleting an image frame first entering the queue and a fusion matching result of the image frame first entering the queue and the template in the optical flow queue.

2. The method of claim 1, wherein, The step of performing feature point matching on a current image frame and a template to obtain a first matching result comprises the following steps: obtaining feature points of the current image frame and first descriptors corresponding to the feature points of the current image frame, and feature points of the template and second descriptors corresponding to the feature points of the template; comparing the first descriptors with the second descriptors to obtain a descriptor matching relationship between the first descriptors and the second descriptors; obtaining a feature point matching relationship between the feature points of the current image frame and the feature points of the template according to the descriptor matching relationship, wherein the feature point matching relationship is the first matching result of the current image frame and the template.

3. The method of claim 2, wherein, The step of obtaining the feature points of the current image frame and the first descriptors corresponding to the feature points of the current image frame comprises the following steps: obtaining the current image frame; performing grayscale processing on the current image frame to obtain a grayscale image of the current image frame; performing Gaussian filtering on the grayscale image to obtain a filtered grayscale image; extracting all feature points of the filtered grayscale image; extracting a descriptor corresponding to each feature point according to the current image frame.

4. The method of claim 2, wherein, The step of obtaining the feature points of the current image frame comprises the following steps: obtaining all pixel points of the current image frame and pixel points in a neighborhood of N*N adjacent to the pixel points, and performing the following operations on a single pixel point to obtain a candidate feature point of the current image frame: determining a first normalized gradient value of the pixel point in an X direction and a second normalized gradient value of the pixel point in a Y direction according to the pixel point and neighborhood pixel points in the neighborhood of the pixel point; fusing the first normalized gradient value and the second normalized gradient value to obtain a normalized gradient value of the pixel point; determining whether the normalized gradient value of the pixel point is greater than normalized gradient values of all neighborhood pixel points next to the pixel point in an X direction and normalized gradient values of all neighborhood pixel points next to the pixel point in a Y direction; if the normalized gradient value of the pixel point is greater than normalized gradient values of all neighborhood pixel points next to the pixel point in an X direction and normalized gradient values of all neighborhood pixel points next to the pixel point in a Y direction, taking the pixel point as a candidate feature point of the current image frame; taking a feature point with a normalized gradient value within a specified range in the candidate feature point as a feature point of the current image frame.

5. The method of claim 1, wherein, the obtaining of the fusion matching result of the current image frame and the template further comprises: updating the first matching result corresponding to the current image frame in the optical flow queue as the fusion matching result of the current image frame and the template.

6. The method of claim 1, wherein, further comprising: if feature point matching is performed on the current image frame and the template, no first matching result is obtained; tracking a previous image frame of the current image frame using an optical flow method to obtain a first optical flow matching result; saving the first optical flow matching result as the first matching result into the optical flow queue.

7. A pose fusion method, characterized in that, comprising: determining a first pose of the current image frame according to a fusion matching result of the current image frame and the template and a corresponding relationship between a 2D point of the current image frame and a 3D point of the template, wherein the fusion matching result is obtained according to the image tracking method in any one of claims 1-6; obtaining a first rotation amount of a previous image frame and a first rotation amount of the current image frame using a sensor; obtaining a second rotation amount of the previous image frame and a second rotation amount of the current image frame using the first pose; obtaining a first rotation change amount according to the first rotation amount of the previous image frame and the first rotation amount of the current image frame; obtaining a second rotation change amount according to the second rotation amount of the previous image frame and the second rotation amount of the current image frame; performing weighted fusion on the first rotation change amount and the second rotation change amount to obtain a rotation fusion result; obtaining a pose of the current image frame after rotation fusion according to the rotation fusion result.

8. The method of claim 7, wherein, the performing of the weighted fusion on the first rotation change amount and the second rotation change amount to obtain the rotation fusion result comprises: converting the first rotation change amount and the second rotation change amount into quaternion forms to obtain a first rotation change amount quaternion and a second rotation change amount quaternion; performing weighted fusion on the first rotation change amount quaternion and the second rotation change amount quaternion to obtain a fused rotation change amount quaternion result; converting the fused rotation change amount quaternion result into a rotation matrix; performing fusion on the rotation matrix and the second rotation amount of the current image frame to obtain the rotation fusion result.

9. An image tracking system, characterized by comprising: a first matching module, configured to perform feature point matching on a current image frame and a template to obtain a first matching result, wherein the template is an image containing only a target object in the current image frame; a queue module, configured to cache the current image frame and the first matching result into an optical flow queue, wherein the optical flow queue further comprises at least one previous image frame before the current image frame and a fusion matching result of the previous image frame and the template; a second matching module, configured to track positions of feature points of the previous image frame in the current image frame using an optical flow method, to obtain at least one optical flow matching result of the current image frame and the at least one previous image frame; a first fusion module, configured to obtain at least one second matching result of the current image frame and the template according to the optical flow matching result and the fusion matching result of the previous image frame and the template; a second fusion module, configured to perform weighted fusion on the first matching result and all the second matching results, to obtain a fusion matching result of the current image frame and the template; wherein after the current image frame and the first matching result are cached into the optical flow queue, the method further comprises: judging whether a number of image frames in the optical flow queue is greater than a specified value; and if the number of image frames in the optical flow queue is greater than the specified value, deleting an image frame first entering the queue and a fusion matching result of the image frame first entering the queue and the template in the optical flow queue.

10. A pose fusion system, comprising: comprising: a first pose module, configured to determine a first pose of the current image frame according to a fusion matching result of the current image frame and the template and a corresponding relationship between 2D points of the current image frame and 3D points of the template, wherein the fusion matching result is obtained according to the image tracking method in any one of claims 1 to 6; a first rotation module, configured to obtain a first rotation amount of a previous image frame and a first rotation amount of the current image frame using a sensor; a second rotation module, configured to obtain a second rotation amount of the previous image frame and a second rotation amount of the current image frame using the first pose; a first rotation change module, configured to obtain a first rotation change amount according to the first rotation amount of the previous image frame and the first rotation amount of the current image frame; a second rotation change module, configured to obtain a second rotation change amount according to the second rotation amount of the previous image frame and the second rotation amount of the current image frame; a rotation fusion module, configured to perform weighted fusion on the first rotation change amount and the second rotation change amount, to obtain a rotation fusion result; a second pose module, configured to obtain a pose of the current image frame after rotation fusion according to the rotation fusion result.

11. An electronic device, comprising: comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements steps of the image tracking method in any one of claims 1 to 6; or the program, when executed by the processor, implements steps of the pose fusion method in any one of claims 7 or 8.

12. A computer-readable storage medium, characterized in that, a computer program is stored on a computer readable storage medium, and the computer program, when executed by a processor, implements steps of the image tracking method in any one of claims 1 to 6; and the computer program, when executed by the processor, implements steps of the pose fusion method in any one of claims 7 or 8.

Images