A planar target tracking method, device, terminal equipment and storage medium

Abstract

The present application is applicable to the field of target tracking technology, and provides a planar target tracking method, device, terminal equipment and storage medium. In the embodiment of the present application, the transformation matrix of each frame of the current video is obtained, the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame is calculated, and the transformation matrix to be tested is obtained; the feature point set to be tested and the circumscribed rectangle to be tested are determined according to the transformation matrix to be tested, and the initial feature point set and the initial circumscribed rectangle of the first frame are obtained; the optical flow is calculated according to the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and the optical flow point set of the current frame is matched, and a calibration optical flow point set that meets the preset conditions is selected from the optical flow point set of the current frame after matching, and four point pairs with the same ID are selected from the calibration optical flow point set and the initial feature point set, and the calibration matrix is ​​determined according to the four point pairs, and the planar target of the current frame is determined according to the calibration matrix. This process improves the accuracy of the overall target tracking algorithm.

Description

Technical Field

[0001] The present application belongs to the field of target tracking technology, and in particular, relates to a planar target tracking method, device, terminal equipment and storage medium. Background Art

[0002] With the continuous development of society, computers are becoming more and more common in people's lives. Among them, visual target tracking is one of the core issues of computer vision and an indispensable part of most visual systems. It has a wide range of application scenarios, such as augmented reality technology and robotics research technology. The target tracking technology of three-dimensional structure itself is extremely challenging. It can be reasonably replaced by two-dimensional plane structure. Therefore, plane target tracking plays an extremely important role in vision-related applications. How to correctly locate the target in each frame is crucial.

[0003] In the prior art, the planar target tracking can be performed using a matching algorithm based on the appearance or external conditions of the target. However, this method is often interfered by various factors, resulting in inaccurate planar target tracking. Summary of the invention

[0004] The embodiments of the present application provide a planar target tracking method, apparatus, terminal device and storage medium, which can solve the problem of inaccurate planar target tracking.

[0005] In a first aspect, an embodiment of the present application provides a planar target tracking method, comprising:

[0006] Obtaining transformation matrices of each frame of the current video, selecting a transformation matrix of the current frame and a transformation matrix of a previous frame from the transformation matrices, calculating a mean value of the transformation matrix of the current frame and the transformation matrix of the previous frame, and obtaining a transformation matrix to be tested;

[0007] Determine a feature point set to be measured and a bounding rectangle to be measured according to the transformation matrix to be measured, and obtain an initial feature point set and an initial bounding rectangle of the first frame;

[0008] When the distance between the feature point set to be measured and the feature point with the same ID in the initial feature point set is greater than a preset distance threshold or the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be measured is greater than a preset area difference, a preset optical flow point set of the previous frame is obtained, and optical flow calculation is performed based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and matching processing is performed on the optical flow point set of the current frame to obtain the matched optical flow point set of the current frame;

[0009] A calibration optical flow point set that meets preset conditions is selected from the optical flow point set of the current frame after matching, four point pairs with the same ID are selected from the calibration optical flow point set and the initial feature point set, a calibration matrix is ​​determined according to the four point pairs, and a planar target of the current frame is determined according to the calibration matrix.

[0010] Optionally, after obtaining the initial feature point set of the first frame, including:

[0011] Determine an initial circumscribed rectangle of the first frame according to the initial feature point set;

[0012] grayscale the initial bounding rectangle, and divide the processed initial bounding rectangle into regions equally according to a first preset number;

[0013] The key points of each region are acquired, and the key points of each region are processed to obtain feature descriptors of the key points of each region.

[0014] Optionally, obtaining the key points of each area includes:

[0015] A sliding window is formed with each pixel in the region as the center, and a pixel value of each sub-pixel in the sliding window is obtained;

[0016] Calculating the gradient modulus and direction value of each sub-pixel in the sliding window according to the pixel value of each sub-pixel;

[0017] Calculating the cumulative value of each sub-pixel according to the gradient modulus value and the direction value;

[0018] Comparing the accumulated values ​​of the sub-pixels, and taking the sub-pixel with the largest accumulated value as the sub-pixel to be processed;

[0019] Taking the accumulated value of the sub-pixel to be processed as the accumulated value of the pixel;

[0020] The accumulated values ​​of the pixels in the region are compared, and the pixel with the largest accumulated value is taken as the key point of the region.

[0021] Optionally, the processing of the key points of each region to obtain feature descriptors of the key points of each region includes:

[0022] Constructing a model region of a first preset range with the key point as the center, and dividing the model region into sub-regions equally according to a second preset number;

[0023] Acquire the pixel value of each sub-pixel in the sub-region, and calculate the gradient modulus and direction value of each sub-pixel in the sub-region according to the pixel value of each sub-pixel in the sub-region;

[0024] Determine a representative vector of the sub-region according to the gradient magnitude value and the direction value;

[0025] The feature descriptor of the key point is determined according to the representation vector of each sub-region.

[0026] Optionally, selecting a calibration optical flow point set that meets a preset condition from the matched optical flow point set of the current frame includes:

[0027] Obtain the position of the key point, select an optical flow point set within a second preset range with the position of the key point as the center according to the matched optical flow point set of the current frame, and use the optical flow point set as a candidate optical flow point set;

[0028] The feature descriptors of each candidate optical flow point in the candidate optical flow point set are obtained, the feature descriptors of each candidate optical flow point are respectively compared with the feature descriptors of the key points for similarity, and the candidate optical flow point with the highest similarity is selected from the candidate optical flow point set as the calibration optical flow point.

[0029] Optionally, after determining the feature point set to be measured and the bounding rectangle to be measured according to the transformation matrix to be measured, and obtaining the initial feature point set and the initial bounding rectangle of the first frame, the method further includes:

[0030] When the distance between the feature points of the to-be-tested feature point set and the feature points of the same ID in the initial point set is less than or equal to the distance threshold and the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the to-be-tested circumscribed rectangle is less than or equal to the area difference, iterative calculation is performed according to the to-be-tested transformation matrix;

[0031] When the number of iterative calculations is equal to a preset number threshold, a calibration matrix is ​​determined, and the planar target of the current frame is determined according to the calibration matrix.

[0032] Optionally, the iterative calculation according to the transformation matrix to be tested includes:

[0033] Performing an inverse transformation on the transformation matrix to be tested to obtain a feature point set of a previous frame, and updating the feature point set to be tested according to the feature point set of the previous frame, the initial feature point set and the transformation matrix to be tested;

[0034] Four point pairs with the same ID are selected from the updated feature point set to be measured and the initial feature point set, a transformation matrix to be processed is determined according to the four point pairs, and the transformation matrix to be processed is used as the transformation matrix to be measured for the next iterative calculation.

[0035] In a second aspect, an embodiment of the present application provides a planar target tracking device, comprising:

[0036] A mean calculation module is used to obtain the transformation matrix of each frame of the current video, select the transformation matrix of the current frame and the transformation matrix of the previous frame from the transformation matrix, calculate the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame, and obtain the transformation matrix to be tested;

[0037] A first determination module, used to determine a feature point set to be measured and a bounding rectangle to be measured according to the transformation matrix to be measured, and obtain an initial feature point set and an initial bounding rectangle of a first frame;

[0038] An optical flow calculation module, used for obtaining a preset optical flow point set of a previous frame, performing optical flow calculation based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and performing matching processing on the optical flow point set of the current frame to obtain the matched optical flow point set of the current frame when the distance between the feature point set to be measured and the feature point with the same ID in the initial feature point set is greater than a preset distance threshold or the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be measured is greater than a preset area difference;

[0039] A planar target determination module is used to select a calibration optical flow point set that meets preset conditions from the optical flow point set of the current frame after matching, select four point pairs with the same ID from the calibration optical flow point set and the initial feature point set, determine a calibration matrix based on the four point pairs, and determine the planar target of the current frame based on the calibration matrix.

[0040] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned planar target tracking methods when executing the computer program.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned planar target tracking methods are implemented.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the planar target tracking methods in the first aspect above.

[0043] In the embodiment of the present application, the transformation matrix of each frame of the current video is obtained, the transformation matrix of the current frame and the transformation matrix of the previous frame are selected from the transformation matrix, and the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame is calculated to obtain the transformation matrix to be tested; the feature point set to be tested and the bounding rectangle to be tested are determined according to the transformation matrix to be tested, and the initial feature point set and the initial bounding rectangle of the first frame are obtained; when the distance between the feature points with the same ID in the feature point set to be tested and the initial feature point set is greater than a preset distance threshold or the area of ​​the initial bounding rectangle and the bounding rectangle to be tested When the area difference between the two is greater than a preset area difference, a preset optical flow point set of a previous frame is obtained, and optical flow calculation is performed based on the optical flow point set of the previous frame to obtain an optical flow point set of a current frame, and matching processing is performed on the optical flow point set of the current frame to obtain a matched optical flow point set of the current frame; a calibration optical flow point set that meets preset conditions is selected from the matched optical flow point set of the current frame, four point pairs with the same ID are selected from the calibration optical flow point set and the initial feature point set, a calibration matrix is ​​determined based on the four point pairs, and a planar target of the current frame is determined based on the calibration matrix. Through the embodiments of the present application, the accuracy of the transformation matrix of the current frame is improved by calculating the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame. Considering that the planar target of the first frame is the most accurate compared with other frames, the initial feature point set and the initial circumscribed rectangle of the first frame are compared, and the optical flow calculation is used to obtain the optical flow point set of the current frame, and the optical flow point set of the current frame is further matched to improve the accuracy of the optical flow point set of the current frame. The calibration optical flow point set is reselected according to preset conditions, and the calibration matrix of the current frame after calibration is determined according to the calibration optical flow point set and the initial feature point set, and the planar target of the current frame is finally confirmed according to the above-mentioned calibrated matrix, thereby improving the accuracy of the overall target tracking algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a schematic diagram of a first flow chart of a planar target tracking method provided in an embodiment of the present application;

[0046] Figure 2 This is a second flow chart of the planar target tracking method provided in an embodiment of the present application;

[0047] Figure 3 This is a third flow chart of the planar target tracking method provided in an embodiment of the present application;

[0048] Figure 4 This is a fourth flow chart of the planar target tracking method provided in an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of the area division of the planar target tracking method provided in an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of the structure of a planar target tracking device provided in an embodiment of the present application;

[0051] Figure 7 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0053] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0054] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0055] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0056] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0057] Figure 1FIG. 1 is a flow chart of a planar target tracking method in an embodiment of the present application. The execution subject of the method may be a terminal device, such as Figure 1 As shown, the planar target tracking method may include the following steps:

[0058] Step S101, obtaining transformation matrices of each frame of the current video, selecting the transformation matrix of the current frame and the transformation matrix of the previous frame from the transformation matrices, calculating the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame, and obtaining the transformation matrix to be tested.

[0059] In this embodiment, the transformation matrix of each frame of the current video is obtained from the transformation matrix set storing each frame of the video, and the transformation matrix of the current frame and the transformation matrix of the previous frame are selected from the obtained transformation matrices of each frame of the current video, and the transformation matrix to be tested is obtained by calculating the mean of the transformation matrices of the current frame and the previous frame. It can be understood that in the process of continuous video playback, the plane target area is also equivalent to the perspective transformation between each frame, so the transformation of the plane target area is expressed by the transformation matrix. The above-mentioned transformation matrix to be tested is the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame, so it is equivalent to processing the transformation matrix of the current frame, so that the transformation matrix of the current frame is closer to the transformation matrix of the previous frame, which is convenient for subsequent matrix processing and improves the accuracy of the plane target. In addition, because the transformation matrix of the current frame and the transformation matrix of the previous frame need to be taken, the matrix length of the transformation matrix of each frame of the current video obtained should be greater than or equal to 2, that is, there should be at least two transformation matrices.

[0060] Step S102: determining a feature point set to be measured and a bounding rectangle to be measured according to the transformation matrix to be measured, and obtaining an initial feature point set and an initial bounding rectangle of the first frame.

[0061] In this embodiment, the feature point set to be tested and the bounding rectangle to be tested are determined according to the above-mentioned transformation matrix to be tested. Since the transformation matrix to be tested refers to the transformation matrix of the current frame after processing, the feature point set to be tested refers to each feature point of the current frame, and the plane target area of ​​the current frame can be determined by each feature point of the current frame. The bounding rectangle to be tested refers to the minimum bounding rectangle composed of each feature point of the current frame. At the same time, the initial feature point set and initial bounding rectangle of the first frame are obtained, that is, the initial feature points of the first frame of the current video and the minimum initial bounding rectangle composed of the initial feature points. Among them, the initial feature point set of the first frame can be manually selected by the user, or the initial feature point set can be determined by detecting each feature part of the plane target area of ​​the first frame. It can be understood that the plane target area determined by the initial feature point set of the first frame can be said to be the most accurate set of feature points of the plane target in the entire video sequence. Therefore, the initial feature point set of the first frame is used for subsequent processing operations to improve the accuracy of the plane target in subsequent operations.

[0062] It is understandable that, since the transformation matrix of the first frame does not perform perspective transformation, it can be calculated using the initial bounding rectangle of the first frame, and the coordinates of the upper left corner of the initial bounding rectangle of the first frame are used to obtain:

[0063]

[0064] Among them, the above T is the transformation matrix of the first frame; the above rect.x is the horizontal coordinate of the upper left corner point of the initial circumscribed rectangle of the first frame; and the above rect.y is the vertical coordinate of the upper left corner point of the initial circumscribed rectangle of the first frame.

[0065] Optionally, the above-mentioned feature point set to be tested can be obtained through the transformation matrix. For example, the coordinates of the feature point A in the previous frame are (u, v, w). Since the perspective transformation is performed in a 3D space, and we are processing a 2D image in this embodiment, the coordinate w of the feature point A is 1, which can be calculated through the transformation matrix to be tested:

[0066]

[0067] Among them, the above (x', y', w') is the coordinates of the feature point A of the previous frame after transformation; Indicates image scaling, rotation, and other transformations; Represents the perspective transformation of the image; the above T3=|a 31 a 32 | represents the translation of the image; the above a 33 The usual value is 1.

[0068] Step S103: When the distance between the feature point set to be tested and the feature point with the same ID in the initial feature point set is greater than a preset distance threshold or the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be tested is greater than a preset area difference, a preset optical flow point set of the previous frame is obtained, and optical flow calculation is performed based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and matching processing is performed on the optical flow point set of the current frame to obtain the matched optical flow point set of the current frame.

[0069] In this embodiment, the distance between the feature points of the same ID in the feature point set to be tested and the initial feature point set is calculated. When the distance between at least one set of feature points of the same ID among all the calculated distances between the feature points of the same ID is greater than the preset distance threshold, the optical flow point set of the previous frame is obtained, and the optical flow calculation is performed based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and the optical flow point set of the current frame is matched to obtain the optical flow point set of the current frame after matching. It can be understood that if the distance between at least one set of feature points of the same ID in the feature point set to be tested and the initial feature point set is greater than the preset distance threshold, it means that the movement distance of the planar target area of ​​the current frame relative to the planar target area of ​​the first frame exceeds the preset distance, and optical flow movement needs to be used for correction.

[0070] In this embodiment, the difference between the area of ​​the measured circumscribed rectangle and the area of ​​the initial circumscribed rectangle is calculated. When the difference between the calculated areas is greater than the preset area difference, the preset optical flow point set of the previous frame is obtained, and the optical flow calculation is performed based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and the optical flow point set of the current frame is matched to obtain the matched optical flow point set of the current frame.

[0071] It can be understood that the optical flow points in the above optical flow point set represent the places where the brightness changes the most in the image. In this embodiment, it refers to the places where the brightness changes the most in the circumscribed rectangle obtained by the feature points. When the video time interval is small, which is equivalent to the time between the two consecutive frames of the video, the change of the optical flow point is equivalent to the displacement of the target point. Therefore, the optical flow point can also be called an extreme point, that is, a point with the maximum or minimum intensity in certain attributes. By using the structural information of the target image itself, such as the optical flow point, to track the planar target, the tracking failure caused by various factors is avoided.

[0072] Optionally, the above optical flow points include but are not limited to intersections between image contours; intersections between two or more edges of an image; points in an image where the rate of change of gradient value and gradient direction are both higher than a certain threshold, etc. The above optical flow points can be obtained by sliding a window of a fixed size on the image in any direction, and comparing the situation before and after the sliding at any time, that is, the degree of grayscale change of pixels in the window. If there is a grayscale value change greater than a preset change threshold, then an optical flow point exists in the window, and then the position of the optical flow point is obtained.

[0073] Optionally, the optical flow point set of the current frame is matched to obtain the matched optical flow point set of the current frame, which specifically includes: taking the optical flow points of the current frame as reference points, performing reverse operations using optical flow calculation to obtain the optical flow points of the previous frame, comparing the positions of the optical flow points of the previous frame with those of the current frame, and if the position difference is greater than a certain threshold, eliminating the optical flow points of the current frame, and performing the above-mentioned matching processing on all optical flow points in the optical flow point set of the current frame, and eliminating the optical flow points with excessive deviations after matching, to finally obtain the matched optical flow point set of the current frame.

[0074] As a specific example but not limitation, the optical flow point set of the previous frame is obtained by optical flow calculation. Assuming that the grayscale value of the pixel in the target area between two consecutive frames of images remains unchanged and the adjacent pixels have the same motion, based on the above assumption, the pixel A(x, y, t) of the previous frame moves a distance of (dx, dy) to the next frame, and it takes dt time, then it can be obtained:

[0075] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0076] After Taylor expansion on the right side of the formula, we can get:

[0077]

[0078] Let u and v be the velocity vectors of the optical flow along the X-axis and Y-axis respectively:

[0079]

[0080] Order I x ,I y ,I t Respectively represent the partial derivatives of the grayscale of the pixel in the image along the x, y, and t directions:

[0081]

[0082] The final calculation formula is:

[0083] I x u+I y v+I t =0

[0084] Among them, u and v are the desired optical flow vectors. According to the calculated u and v, the position of the optical flow point in the previous frame in the next frame can be found.

[0085] Step S104, selecting a calibration optical flow point set that meets preset conditions from the optical flow point set of the current frame after matching, selecting four point pairs with the same ID from the calibration optical flow point set and the initial feature point set, determining a calibration matrix based on the four point pairs, and determining the planar target of the current frame based on the calibration matrix.

[0086] In this embodiment, a calibration optical flow point set that meets preset conditions is further selected from the optical flow point set of the matched current frame, so that the accuracy of the selected calibration optical flow point set can be more accurately guaranteed. Since each point has an ID number, four point pairs with the same ID can be selected from the calibration optical flow point set and the initial feature point set. The calibration matrix is ​​determined using the selected four point pairs. The accuracy of the obtained calibration matrix can be improved by using the initial feature point set, that is, the feature points of the planar target of the first frame and the selected calibration optical flow point set. The planar target of the current frame is determined from the calibration matrix to reduce the deviation between the planar target of the current frame and the actual one, and the processed calibration matrix is ​​replaced with the transformation matrix of the current frame in the above-mentioned transformation matrix set, so as to provide a more accurate basis for the processing of subsequent video frame transformation matrices.

[0087] like Figure 2 As shown, after obtaining the initial feature point set of the first frame, it includes:

[0088] Step S201: determining an initial bounding rectangle of the first frame according to the initial feature point set.

[0089] Step S202: grayscale the initial bounding rectangle, and divide the processed initial bounding rectangle into various regions equally according to a first preset number.

[0090] Step S203: Acquire key points of each region, process the key points of each region, and obtain feature descriptors of the key points of each region.

[0091] In this embodiment, the initial bounding rectangle of the current first frame is determined based on the feature point set initially selected in the first frame, and the obtained initial bounding rectangle is grayed to make the above initial bounding rectangle a single-channel image for later processing. The processed initial bounding rectangle is equally divided into various regions according to the first preset number, and the key points of each region are obtained, and the key points are processed to obtain the feature descriptors of the key points of each region, which provide a basis for subsequent similarity comparison. Among them, the above first preset number can be set according to user needs. The above key points are points that can represent each divided region.

[0092] Optionally, the above-mentioned initial circumscribed rectangle can be determined by connecting the initial feature points in the initial feature point set in a certain order to form a polygon of any shape, and then determining the boundary of the initial circumscribed rectangle based on the maximum horizontal coordinate, minimum horizontal coordinate, maximum vertical coordinate, and minimum vertical coordinate of the polygon.

[0093] like Figure 3 As shown, the key points of obtaining each area include:

[0094] Step S301: Make a sliding window with each pixel in the area as the center, and obtain the pixel value of each sub-pixel in the sliding window.

[0095] In this embodiment, a sliding window is formed with each pixel of each divided area as the center, and the pixel value of each sub-pixel in the sliding window corresponding to each pixel is obtained. The size of the sliding window is determined according to user requirements.

[0096] As a specific example but not limitation, if the size of the rectangular area is 16*16, and the rectangular area is equally divided into 4*4 areas, then the size of each divided area is 4*4, which means that the area contains 16 pixels, and a sliding window of a preset size is made with each pixel as the center. For example, the size of the sliding window is 5*5, and the pixel value of each sub-pixel in the sliding window is obtained by moving the sliding window.

[0097] Step S302: Calculate the gradient modulus and direction value of each sub-pixel in the sliding window according to the pixel value of each sub-pixel.

[0098] In this embodiment, the gradient modulus and direction value of each sub-pixel in the sliding window are calculated according to the pixel value of each sub-pixel obtained. The calculation formula of the above gradient modulus is:

[0099] mag=sqrt((right-cur) 2 +(bottom-cur) 2 )

[0100] The calculation formula for the above direction value is:

[0101] ori=arctan((bottom-cur),(right-cur))

[0102] Wherein, the above mag is the gradient modulus value of the current sub-pixel; the above ori is the direction value of the current sub-pixel; the above right is the pixel value of the sub-pixel adjacent to the right of the current sub-pixel; the above cur is the pixel value of the current sub-pixel; and the above bottom is the pixel value of the sub-pixel adjacent to the bottom of the current sub-pixel. It can be understood that the movement of the sliding window starts from the pixel point in the upper left corner and moves to the right and downward, so the pixel value of the sub-pixel adjacent to the right of the current sub-pixel and the pixel value of the sub-pixel adjacent to the bottom of the current sub-pixel are used to calculate the gradient modulus value and direction value of the current sub-pixel.

[0103] Step S303: Calculate the cumulative value of each sub-pixel according to the gradient modulus value and the direction value.

[0104] Step S304: Compare the accumulated values ​​of the sub-pixels, and use the sub-pixel with the largest accumulated value as the sub-pixel to be processed.

[0105] Step S305: taking the accumulated value of the sub-pixel to be processed as the accumulated value of the pixel.

[0106] In this embodiment, the cumulative value of each sub-pixel is calculated according to the gradient modulus and direction value calculated above, and then the cumulative values ​​of each sub-pixel are compared to select the sub-pixel with the largest cumulative value from the cumulative values ​​of the above sub-pixels, and the sub-pixel with the largest cumulative value is used as the sub-pixel to be processed, and the cumulative value of the sub-pixel to be processed is used as the cumulative value of the center pixel in the above window.

[0107] Optionally, the above-mentioned calculation of the cumulative value of each sub-pixel based on the gradient modulus and the direction value includes: equally dividing the modulus accumulation area according to a preset threshold value, and accumulating the gradient modulus values ​​of the sub-pixels into the divided modulus accumulation area according to the above-mentioned direction value; and calculating the cumulative value of the gradient modulus in each modulus accumulation area, selecting a modulus accumulation area with the largest cumulative value from the divided modulus accumulation areas, and using the cumulative value of the selected modulus accumulation area with the largest cumulative value as the cumulative value of the sub-pixel.

[0108] As a specific example but not limitation, the modulus accumulation area is divided into 36 areas according to 0 to 360 degrees, for example, the first area is 0 to 9 degrees, the second area is 10 to 19 degrees, the third area is 20 to 29 degrees, and so on. The gradient modulus values ​​of the sub-pixels are accumulated in the modulus accumulation area of ​​the corresponding angle according to the above-mentioned direction value. When the gradient modulus values ​​of all sub-pixels are accumulated in the corresponding modulus accumulation area, they are compared to select the modulus accumulation area with the largest cumulative value.

[0109] Step S306: Compare the cumulative values ​​of the pixels in the region, and take the pixel with the largest cumulative value as the key point of the region.

[0110] In this embodiment, the cumulative value of each pixel in the area is obtained by the above method, the cumulative values ​​of each pixel are compared, the pixel with the largest cumulative value is selected, and the pixel with the largest cumulative value is used as the key point of the area, that is, the point that can best represent the characteristics of the area.

[0111] like Figure 4 As shown, the key points of each region are processed to obtain feature descriptors of the key points of each region, including:

[0112] Step S401: construct a model region of a first preset range with the key point as the center, and divide the model region into sub-regions equally according to a second preset number.

[0113] Step S402: Obtain pixel values ​​of each sub-pixel in the sub-region, and calculate the gradient modulus and direction value of each sub-pixel in the sub-region according to the pixel values ​​of each sub-pixel in the sub-region.

[0114] Step S403: Determine a representative vector of the sub-region according to the gradient modulus value and the direction value.

[0115] Step S404: Determine the feature descriptor of the key point according to the representation vector of each sub-region.

[0116] In this embodiment, a model area of ​​a first preset range is constructed with the position of the key point as the center, and the model area is equally divided into each sub-area according to the second preset number, and the pixel value of each sub-pixel in each sub-area is obtained, and the gradient modulus and direction value of each sub-pixel are calculated according to the pixel value of each sub-pixel, and the modulus accumulation area is equally divided according to the preset threshold value, and the gradient modulus of the sub-pixel is accumulated into the divided modulus accumulation area according to the above direction value; and the cumulative value of the gradient modulus in each modulus accumulation area is calculated, and the length of the representation vector of the above sub-area is determined by the divided modulus accumulation area, and the length is determined by the number of divided modulus accumulation areas. After the representation vector of each sub-area is determined, the representation vector of each sub-area is combined to determine the feature descriptor of the key point. Among them, the first preset range and the second preset number can be set according to user needs.

[0117] As a specific example but not limitation, a 16*16 area is constructed with the current coordinate (x, y) as the center, and is equally divided according to the number 16 to obtain 16 sub-areas. The pixel in the upper left corner of each divided sub-area is used as the starting point, and the gradient modulus and direction value of each sub-pixel in the sub-area are calculated to the right and downward, and the modulus accumulation area from 0 to 360 degrees is divided into 8 areas, and the cumulative value is calculated. Each modulus accumulation area represents a representation vector of the sub-area, and the sub-area forms a representation vector with a length of 8. The representation vectors of the 16 sub-areas are calculated, and the representation vectors of the 16 sub-areas with a length of 8 are combined to obtain the 128-dimensional feature descriptor of the key point, such as Figure 5 shown.

[0118] Optionally, selecting a calibration optical flow point set that meets a preset condition from the matched optical flow point set of the current frame includes:

[0119] The position of the key point is obtained, and an optical flow point set within a second preset range is selected with the position of the key point as the center according to the matched optical flow point set of the current frame, and the optical flow point set is used as a candidate optical flow point set.

[0120] The feature descriptors of each candidate optical flow point in the candidate optical flow point set are obtained, the feature descriptors of each candidate optical flow point are respectively compared with the feature descriptors of the key points for similarity, and the candidate optical flow point with the highest similarity is selected from the candidate optical flow point set as the calibration optical flow point.

[0121] In this embodiment, the position of the key point obtained in the first frame is obtained, and the optical flow point set within the second preset range is selected from the optical flow point set of the current frame after the above matching with the position of the key point as the center, and the optical flow point set is used as the candidate optical flow point set. According to the calculation method of the feature descriptor, the feature descriptors of each candidate optical flow point in the candidate optical flow point set are calculated and obtained, and then the Euclidean distance is used to compare the similarity with the feature descriptors of the key points calculated above, and the candidate optical flow points with the highest similarity are selected as calibration optical flow points. It can be understood that the key points of each area are respectively subjected to the above operations to obtain a calibration optical flow point set. Among them, the second preset range can be set according to user needs.

[0122] Optionally, after determining the feature point set to be measured and the bounding rectangle to be measured according to the transformation matrix to be measured, and obtaining the initial feature point set and the initial bounding rectangle of the first frame, the method further includes:

[0123] When the distance between the feature points with the same ID in the feature point set to be tested and the initial point set is less than or equal to the distance threshold and the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be tested is less than or equal to the area difference, iterative calculation is performed according to the transformation matrix to be tested.

[0124] When the number of iterative calculations is equal to a preset number threshold, a calibration matrix is ​​determined, and the planar target of the current frame is determined according to the calibration matrix.

[0125] In this embodiment, when the distance between the feature points of the same ID in the feature point set to be measured and the initial point set is less than or equal to the preset distance threshold and the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be measured is less than or equal to the preset area difference, iterative calculation is performed according to the above-mentioned transformation matrix to be measured; when the number of iterative calculations is equal to the preset number threshold, the currently calculated matrix is ​​determined as the calibration matrix, and then the plane target of the current frame is determined according to the above-mentioned calibration matrix. And the processed calibration matrix is ​​replaced with the transformation matrix of the current frame in the above-mentioned transformation matrix set, providing a more accurate basis for the subsequent processing of the video frame transformation matrix. Among them, the above-mentioned preset number threshold can be set according to user needs, generally 1 / 2 of the number of feature points in the initial point set.

[0126] Optionally, the iterative calculation according to the transformation matrix to be tested includes:

[0127] The transformation matrix to be tested is inversely transformed to obtain a feature point set of a previous frame, and the feature point set to be tested is updated according to the feature point set of the previous frame, the initial feature point set and the transformation matrix to be tested.

[0128] Four point pairs with the same ID are selected from the updated feature point set to be measured and the initial feature point set, a transformation matrix to be processed is determined according to the four point pairs, and the transformation matrix to be processed is used as the transformation matrix to be measured for the next iterative calculation.

[0129] In this embodiment, the transformation matrix to be tested is inversely transformed, that is, the feature point set of the previous frame is obtained by using the matrix inversion method, and the feature point set to be tested is updated according to the feature point set of the previous frame, the initial feature point set and the transformation matrix to be tested, that is, the coordinates of each feature point in the feature point set to be tested are calculated, and the coordinates of each feature point are updated to the calculated coordinates to reduce the deviation between the feature point set obtained by the transformation matrix and the actual feature point set. Four point pairs with the same ID are selected from the updated feature point set to be tested and the initial feature point set, and the transformation matrix to be processed is determined according to the selected four point pairs, and the transformation matrix to be processed is used as the transformation matrix to be tested for the next iterative calculation, and iterative calculation is performed again until the number of iterative calculations meets the preset number, and then the iterative calculation is stopped.

[0130] Optionally, the formula for calculating the coordinates of each feature point in the feature point set to be measured is:

[0131] x′=x+(1-PredWarp)*(InitPoints.x-InitPoints′.x)

[0132] y′=y+(1-PredWarp)*(InitPoints.y-InitPoints′.y)

[0133] Among them, the above x is the horizontal coordinate of the feature point A in the feature point set to be tested; the above y is the vertical coordinate of the feature point A in the feature point set to be tested; the above x' is the updated horizontal coordinate of the feature point A in the feature point set to be tested; the above y' is the updated vertical coordinate of the feature point A in the feature point set to be tested; the above PredWarp is the transformation matrix to be tested; the above InitPoints.x is the horizontal coordinate of the feature point A in the initial feature point set; the above InitPoints'.x is the horizontal coordinate of the feature point A in the feature point set of the previous frame; the above InitPoints.y is the vertical coordinate of the feature point A in the initial feature point set; the above InitPoints'.y is the vertical coordinate of the feature point A in the feature point set of the previous frame.

[0134] In the embodiment of the present application, the transformation matrix of each frame of the current video is obtained, the transformation matrix of the current frame and the transformation matrix of the previous frame are selected from the transformation matrix, and the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame is calculated to obtain the transformation matrix to be tested; the feature point set to be tested and the bounding rectangle to be tested are determined according to the transformation matrix to be tested, and the initial feature point set and the initial bounding rectangle of the first frame are obtained; when the distance between the feature points with the same ID in the feature point set to be tested and the initial feature point set is greater than a preset distance threshold or the area of ​​the initial bounding rectangle and the bounding rectangle to be tested When the area difference between the two is greater than a preset area difference, a preset optical flow point set of a previous frame is obtained, and optical flow calculation is performed based on the optical flow point set of the previous frame to obtain an optical flow point set of a current frame, and matching processing is performed on the optical flow point set of the current frame to obtain a matched optical flow point set of the current frame; a calibration optical flow point set that meets preset conditions is selected from the matched optical flow point set of the current frame, four point pairs with the same ID are selected from the calibration optical flow point set and the initial feature point set, a calibration matrix is ​​determined based on the four point pairs, and a planar target of the current frame is determined based on the calibration matrix. Through the embodiments of the present application, the accuracy of the transformation matrix of the current frame is improved by calculating the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame. Considering that the planar target of the first frame is the most accurate compared with other frames, the initial feature point set and the initial circumscribed rectangle of the first frame are compared, and the optical flow calculation is used to obtain the optical flow point set of the current frame, and the optical flow point set of the current frame is further matched to improve the accuracy of the optical flow point set of the current frame. The calibration optical flow point set is reselected according to preset conditions, and the calibration matrix of the current frame after calibration is determined according to the calibration optical flow point set and the initial feature point set, and the planar target of the current frame is finally confirmed according to the above-mentioned calibrated matrix, thereby improving the accuracy of the overall target tracking algorithm.

[0135] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0136] Figure 6 FIG. 1 is a schematic diagram of a planar target tracking device according to an embodiment of the present application. Figure 6 As shown, the planar target tracking device may include:

[0137] The mean calculation module 601 is used to obtain the transformation matrix of each frame of the current video, select the transformation matrix of the current frame and the transformation matrix of the previous frame from the transformation matrix, calculate the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame, and obtain the transformation matrix to be tested.

[0138] The first determination module 602 is used to determine the feature point set to be measured and the bounding rectangle to be measured according to the transformation matrix to be measured, and obtain the initial feature point set and the initial bounding rectangle of the first frame.

[0139] The optical flow calculation module 603 is used to obtain a preset optical flow point set of the previous frame when the distance between the feature point set to be tested and the feature point with the same ID in the initial feature point set is greater than a preset distance threshold or the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be tested is greater than a preset area difference, and perform optical flow calculation based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and perform matching processing on the optical flow point set of the current frame to obtain the matched optical flow point set of the current frame.

[0140] The planar target determination module 604 is used to select a calibration optical flow point set that meets preset conditions from the optical flow point set of the current frame after matching, select four point pairs with the same ID from the calibration optical flow point set and the initial feature point set, determine a calibration matrix based on the four point pairs, and determine the planar target of the current frame based on the calibration matrix.

[0141] Optionally, the planar target tracking device may further include:

[0142] The second determining module is used to determine an initial circumscribed rectangle of the first frame according to the initial feature point set.

[0143] The grayscale processing module is used to perform grayscale processing on the initial bounding rectangle, and divide the processed initial bounding rectangle into various regions equally according to a first preset number.

[0144] The key point acquisition module is used to acquire the key points of each region, process the key points of each region, and obtain feature descriptors of the key points of each region.

[0145] Optionally, the planar target tracking device may further include:

[0146] The pixel value acquisition module is used to make a sliding window with each pixel in the area as the center, and obtain the pixel value of each sub-pixel in the sliding window.

[0147] The sub-pixel calculation module is used to calculate the gradient modulus and direction value of each sub-pixel in the sliding window according to the pixel value of each sub-pixel.

[0148] The selection module is used to calculate the cumulative value of each sub-pixel according to the gradient modulus value and the direction value.

[0149] The cumulative value comparison module is used to compare the cumulative values ​​of the sub-pixels and take the sub-pixel with the largest cumulative value as the sub-pixel to be processed.

[0150] As a module, it is used to use the accumulated value of the sub-pixel to be processed as the accumulated value of the pixel.

[0151] The comparison module is used to compare the cumulative values ​​of each pixel in the area, and take the pixel with the largest cumulative value as the key point of the area.

[0152] Optionally, the planar target tracking device may further include:

[0153] A region construction module is used to construct a model region of a first preset range with the key point as the center, and divide the model region into various sub-regions equally according to a second preset number.

[0154] The acquisition module is used to acquire the pixel value of each sub-pixel in the sub-region, and calculate the gradient modulus and direction value of each sub-pixel in the sub-region according to the pixel value of each sub-pixel in the sub-region.

[0155] The representative vector determination module is used to determine the representative vector of the sub-region according to the gradient modulus value and the direction value.

[0156] The feature descriptor determination module is used to determine the feature descriptor of the key point according to the representation vector of each sub-region.

[0157] Optionally, the planar target tracking device may further include:

[0158] The position acquisition module is used to obtain the position of the key point, select an optical flow point set within a second preset range with the position of the key point as the center according to the matched optical flow point set of the current frame, and use the optical flow point set as a candidate optical flow point set.

[0159] The similarity comparison module is used to obtain the feature descriptors of each candidate optical flow point in the candidate optical flow point set, compare the feature descriptors of each candidate optical flow point with the feature descriptors of the key points respectively, and select the candidate optical flow point with the highest similarity from the candidate optical flow point set as the calibration optical flow point.

[0160] Optionally, the planar target tracking device may further include:

[0161] An iterative calculation module is used to perform iterative calculation according to the transformation matrix to be tested when the distance between the feature points with the same ID in the feature point set to be tested and the initial point set is less than or equal to the distance threshold and the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be tested is less than or equal to the area difference.

[0162] The first calibration matrix determination module is used to determine the calibration matrix when the number of iterative calculations is equal to a preset number threshold, and determine the planar target of the current frame according to the calibration matrix.

[0163] Optionally, the planar target tracking device may further include:

[0164] The inverse transformation module is used to perform an inverse transformation on the transformation matrix to be tested to obtain a feature point set of a previous frame, and to update the feature point set to be tested according to the feature point set of the previous frame, the initial feature point set and the transformation matrix to be tested.

[0165] The second calibration matrix determination module is used to select four point pairs with the same ID from the updated feature point set to be measured and the initial feature point set, determine the transformation matrix to be processed according to the four point pairs, and use the transformation matrix to be processed as the transformation matrix to be measured for the next iterative calculation.

[0166] In the embodiment of the present application, the transformation matrix of each frame of the current video is obtained, the transformation matrix of the current frame and the transformation matrix of the previous frame are selected from the transformation matrix, and the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame is calculated to obtain the transformation matrix to be tested; the feature point set to be tested and the bounding rectangle to be tested are determined according to the transformation matrix to be tested, and the initial feature point set and the initial bounding rectangle of the first frame are obtained; when the distance between the feature points with the same ID in the feature point set to be tested and the initial feature point set is greater than a preset distance threshold or the area of ​​the initial bounding rectangle and the bounding rectangle to be tested When the area difference between the two is greater than a preset area difference, a preset optical flow point set of a previous frame is obtained, and optical flow calculation is performed based on the optical flow point set of the previous frame to obtain an optical flow point set of a current frame, and matching processing is performed on the optical flow point set of the current frame to obtain a matched optical flow point set of the current frame; a calibration optical flow point set that meets preset conditions is selected from the matched optical flow point set of the current frame, four point pairs with the same ID are selected from the calibration optical flow point set and the initial feature point set, a calibration matrix is ​​determined based on the four point pairs, and a planar target of the current frame is determined based on the calibration matrix. Through the embodiments of the present application, the accuracy of the transformation matrix of the current frame is improved by calculating the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame. Considering that the planar target of the first frame is the most accurate compared with other frames, the initial feature point set and the initial circumscribed rectangle of the first frame are compared, and the optical flow calculation is used to obtain the optical flow point set of the current frame, and the optical flow point set of the current frame is further matched to improve the accuracy of the optical flow point set of the current frame. The calibration optical flow point set is reselected according to preset conditions, and the calibration matrix of the current frame after calibration is determined according to the calibration optical flow point set and the initial feature point set, and the planar target of the current frame is finally confirmed according to the above-mentioned calibrated matrix, thereby improving the accuracy of the overall target tracking algorithm.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned system embodiments and method embodiments, and will not be repeated here.

[0168] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0169] like Figure 7 As shown, the terminal device 7 of this embodiment includes: at least one processor 700 ( Figure 7 Only one is shown in the figure), a memory 701 connected to the processor 700, and a computer program 702 stored in the memory 701 and executable on the at least one processor 700, such as a planar target tracking program. When the processor 700 executes the computer program 702, the steps in the above-mentioned various planar target tracking method embodiments are implemented, such as Figure 1 Alternatively, when the processor 700 executes the computer program 702, the functions of the modules in the above-mentioned device embodiments are realized, for example, Figure 6 Functions of modules 601 to 604 are shown.

[0170] Exemplarily, the computer program 702 may be divided into one or more modules, which are stored in the memory 701 and executed by the processor 700 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 702 in the terminal device 7. For example, the computer program 702 may be divided into a mean calculation module 601, a first determination module 602, an optical flow calculation module 603, and a plane target determination module 604, and the specific functions of each module are as follows:

[0171] The mean calculation module 601 is used to obtain the transformation matrix of each frame of the current video, select the transformation matrix of the current frame and the transformation matrix of the previous frame from the transformation matrix, calculate the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame, and obtain the transformation matrix to be tested;

[0172] A first determination module 602 is used to determine a feature point set to be measured and a bounding rectangle to be measured according to the transformation matrix to be measured, and obtain an initial feature point set and an initial bounding rectangle of a first frame;

[0173] The optical flow calculation module 603 is used to obtain a preset optical flow point set of the previous frame when the distance between the feature point set to be measured and the feature point with the same ID in the initial feature point set is greater than a preset distance threshold or the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be measured is greater than a preset area difference, and perform optical flow calculation based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and perform matching processing on the optical flow point set of the current frame to obtain the matched optical flow point set of the current frame;

[0174] The planar target determination module 604 is used to select a calibration optical flow point set that meets preset conditions from the optical flow point set of the current frame after matching, select four point pairs with the same ID from the calibration optical flow point set and the initial feature point set, determine a calibration matrix based on the four point pairs, and determine the planar target of the current frame based on the calibration matrix.

[0175] The terminal device 7 may include, but is not limited to, a processor 700 and a memory 701. Those skilled in the art will appreciate that Figure 7 It is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, buses, etc.

[0176] The processor 700 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0177] In some embodiments, the memory 701 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. In other embodiments, the memory 701 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 7. Further, the memory 701 may also include both an internal storage unit and an external storage device of the terminal device 7. The memory 701 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 701 may also be used to temporarily store data that has been output or is to be output.

[0178] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0179] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0180] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0181] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0185] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A planar target tracking method, characterized in that: include: Obtaining transformation matrices of each frame of the current video, selecting a transformation matrix of the current frame and a transformation matrix of a previous frame from the transformation matrices, calculating a mean value of the transformation matrix of the current frame and the transformation matrix of the previous frame, and obtaining a transformation matrix to be tested; Determine a feature point set to be measured and a bounding rectangle to be measured according to the transformation matrix to be measured, and obtain an initial feature point set and an initial bounding rectangle of the first frame; When the distance between the feature point set to be measured and the feature point with the same ID in the initial feature point set is greater than a preset distance threshold or the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be measured is greater than a preset area difference, a preset optical flow point set of the previous frame is obtained, and optical flow calculation is performed based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and matching processing is performed on the optical flow point set of the current frame to obtain the matched optical flow point set of the current frame; A calibration optical flow point set that meets preset conditions is selected from the optical flow point set of the current frame after matching, four point pairs with the same ID are selected from the calibration optical flow point set and the initial feature point set, a calibration matrix is ​​determined according to the four point pairs, and a planar target of the current frame is determined according to the calibration matrix.

2. The planar target tracking method according to claim 1, characterized in that: After obtaining the initial feature point set of the first frame, including: Determine an initial circumscribed rectangle of the first frame according to the initial feature point set; grayscale the initial bounding rectangle, and divide the processed initial bounding rectangle into regions equally according to a first preset number; The key points of each region are acquired, and the key points of each region are processed to obtain feature descriptors of the key points of each region.

3. The planar target tracking method according to claim 2, characterized in that: The step of obtaining the key points of each area includes: A sliding window is formed with each pixel in the region as the center, and a pixel value of each sub-pixel in the sliding window is obtained; Calculating the gradient modulus and direction value of each sub-pixel in the sliding window according to the pixel value of each sub-pixel; Calculating the cumulative value of each sub-pixel according to the gradient modulus value and the direction value; Comparing the accumulated values ​​of the sub-pixels, and taking the sub-pixel with the largest accumulated value as the sub-pixel to be processed; Taking the accumulated value of the sub-pixel to be processed as the accumulated value of the pixel; The accumulated values ​​of the pixels in the region are compared, and the pixel with the largest accumulated value is taken as the key point of the region.

4. The planar target tracking method according to claim 2, characterized in that: The processing of the key points of each region to obtain the feature descriptors of the key points of each region includes: Constructing a model region of a first preset range with the key point as the center, and dividing the model region into sub-regions equally according to a second preset number; Acquire the pixel value of each sub-pixel in the sub-region, and calculate the gradient modulus and direction value of each sub-pixel in the sub-region according to the pixel value of each sub-pixel in the sub-region; Determine a representative vector of the sub-region according to the gradient magnitude value and the direction value; The feature descriptor of the key point is determined according to the representation vector of each sub-region.

5. The planar target tracking method according to claim 4, characterized in that: The step of selecting a calibration optical flow point set that meets a preset condition from the optical flow point set of the current frame after matching includes: Obtain the position of the key point, select an optical flow point set within a second preset range with the position of the key point as the center according to the matched optical flow point set of the current frame, and use the optical flow point set as a candidate optical flow point set; The feature descriptors of each candidate optical flow point in the candidate optical flow point set are obtained, the feature descriptors of each candidate optical flow point are respectively compared with the feature descriptors of the key points for similarity, and the candidate optical flow point with the highest similarity is selected from the candidate optical flow point set as the calibration optical flow point.

6. The planar target tracking method according to claim 1, characterized in that: After determining the feature point set to be measured and the bounding rectangle to be measured according to the transformation matrix to be measured, and obtaining the initial feature point set and the initial bounding rectangle of the first frame, the method includes: When the distance between the feature points with the same ID in the feature point set to be tested and the initial feature point set is less than or equal to the distance threshold and the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be tested is less than or equal to the area difference, iterative calculation is performed according to the transformation matrix to be tested; When the number of iterative calculations is equal to a preset number threshold, a calibration matrix is ​​determined, and the planar target of the current frame is determined according to the calibration matrix.

7. The planar target tracking method according to claim 6, characterized in that: The iterative calculation according to the transformation matrix to be tested comprises: Performing an inverse transformation on the transformation matrix to be tested to obtain a feature point set of a previous frame, and updating the feature point set to be tested according to the feature point set of the previous frame, the initial feature point set and the transformation matrix to be tested; Four point pairs with the same ID are selected from the updated feature point set to be measured and the initial feature point set, a transformation matrix to be processed is determined according to the four point pairs, and the transformation matrix to be processed is used as the transformation matrix to be measured for the next iterative calculation.

8. A planar target tracking device, characterized in that: include: A mean calculation module is used to obtain the transformation matrix of each frame of the current video, select the transformation matrix of the current frame and the transformation matrix of the previous frame from the transformation matrix, calculate the mean of the transformation matrix of the current frame and the transformation matrix of the previous frame, and obtain the transformation matrix to be tested; A first determination module, used to determine a feature point set to be measured and a bounding rectangle to be measured according to the transformation matrix to be measured, and obtain an initial feature point set and an initial bounding rectangle of a first frame; An optical flow calculation module, used for obtaining a preset optical flow point set of a previous frame, performing optical flow calculation based on the optical flow point set of the previous frame to obtain the optical flow point set of the current frame, and performing matching processing on the optical flow point set of the current frame to obtain the matched optical flow point set of the current frame when the distance between the feature point set to be measured and the feature point with the same ID in the initial feature point set is greater than a preset distance threshold or the difference between the area of ​​the initial circumscribed rectangle and the area of ​​the circumscribed rectangle to be measured is greater than a preset area difference; A planar target determination module is used to select a calibration optical flow point set that meets preset conditions from the optical flow point set of the current frame after matching, select four point pairs with the same ID from the calibration optical flow point set and the initial feature point set, determine a calibration matrix based on the four point pairs, and determine the planar target of the current frame based on the calibration matrix.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a planar target tracking method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a planar target tracking method as described in any one of claims 1 to 7 are implemented.

Images