A pose determination method, system, electronic device and readable storage medium

Abstract

This application provides a pose determination method, system, electronic device, and readable storage medium. The pose determination method includes: acquiring an initial pose and initial target position of a target object in an initial frame; acquiring a first feature point and the corresponding flow feature of an image frame to be analyzed, wherein the flow feature is the position difference between the feature point position of the previous frame and the first feature point position; determining the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow feature corresponding to the first feature point; and determining the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object. The feature points obtained in this application do not gradually decrease, and the error of the current target position in image tracking does not gradually increase, thus enabling the calculated pose to maintain high accuracy and stability.

Description

Technical Field

[0001] This application relates to the field of image tracking technology, and in particular to a pose determination method, system, electronic device, and readable storage medium. Background Technology

[0002] Existing two-dimensional image tracking schemes can be implemented using two parts: a detector and a tracker. The detector uses some simple features to perform a 1v1 match between the template image and the current input video frame, and then uses the Perspective-n-Point (PnP) method to calculate the six degrees of freedom (6DOF) pose of the current frame.

[0003] However, since feature point extraction and matching are time-consuming, detectors typically operate on the first frame or a set interval frame, and then optical flow is used to track the intermediate video frames. Tracking via inter-frame optical flow leads to a decreasing number of feature points, resulting in a larger accumulated error and consequently, a larger calculated 6DOF pose error. Furthermore, the detector may not be able to guarantee the detection of the target's 2D image in the current frame, thus potentially preventing timely correction of the 6DOF pose result. Summary of the Invention

[0004] This application provides a pose determination method, system, electronic device, and readable storage medium to solve the problem that existing optical flow tracking methods result in fewer and fewer image feature points, leading to increasingly larger accumulated errors and consequently larger pose errors.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a pose determination method, including:

[0007] Obtain the initial pose and initial target position of the target object in the initial frame, wherein the initial frame is the first image frame containing the target object;

[0008] The first feature point and the flow feature corresponding to the first feature point of the image frame to be analyzed are obtained. The flow feature is the position difference between the feature point position of the previous frame of the image frame to be analyzed and the position of the first feature point. The feature point position of the previous frame of the image frame to be analyzed is obtained by optical flow tracking.

[0009] Based on the initial target position and the flow features corresponding to the first feature point, the current target position of the target object in the image frame to be analyzed is determined;

[0010] The pose of the target object in the image frame to be analyzed is determined based on the initial pose and the current target position of the target object in the image frame to be analyzed.

[0011] Optionally, before obtaining the initial pose and initial target position of the target object in the initial frame, the method further includes:

[0012] Perform target object detection on the current frame to determine whether there is a target object in the current frame;

[0013] If the target object is present in the current frame, the current frame is used as the image frame to be analyzed.

[0014] Optionally, obtaining the initial pose of the target object in the initial frame includes:

[0015] Obtain the template feature points of the template to be identified and the first descriptor corresponding to the template feature points, wherein the template to be identified is an image containing only the target object;

[0016] Feature points are extracted from the initial frame to obtain the initial feature points of the initial frame and the second descriptor corresponding to the initial feature points;

[0017] The second descriptor and the first descriptor are matched to obtain the matching relationship between the initial feature points and the template feature points;

[0018] The initial pose of the target object in the initial frame is determined based on the matching relationship between the initial feature points and the template feature points.

[0019] Optionally, obtaining the first feature point of the image frame to be analyzed includes:

[0020] Feature point extraction is performed on the image frame to be analyzed to obtain all feature points of the image frame to be analyzed;

[0021] The optical flow method is used to track the image frame to be analyzed, and the position of the previous feature point corresponding to the previous frame of the image frame to be analyzed is obtained.

[0022] Based on the difference between the position of the previous feature point and the position of the current feature point, determine the flow features corresponding to all feature points;

[0023] The image frame to be analyzed is divided into blocks of at least one size to obtain at least one image frame to be analyzed with multiple image blocks;

[0024] Obtain the number of feature points within all image patches;

[0025] Filter out image blocks in which the number of feature points is less than a threshold, and obtain the first filtered image block;

[0026] Based on the flow feature magnitude corresponding to the feature points in the first image block, the feature points in the first image block are filtered to obtain the internal feature points in the first image block.

[0027] Merge all internal feature points within all image blocks and use the internal feature points as the first feature points. Merging all internal feature points within all image blocks means retaining non-repeating internal feature points in all image blocks and merging and retaining internal feature points that are repeated at the same position in all image blocks.

[0028] Optionally, the initial target position is the position of the initial target object bounding box containing the target object in the initial frame;

[0029] Determining the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow features corresponding to the first feature point includes:

[0030] Based on the position of the initial target object box, obtain the in-frame feature points of the first feature points that are located within the initial target object box;

[0031] Select the camera motion corresponding transformation model;

[0032] Obtain the flow features corresponding to the feature points within the frame;

[0033] The current target position of the target object in the image frame to be analyzed is obtained by using the camera motion corresponding transformation model and the flow features corresponding to the feature points in the frame.

[0034] Optionally, obtaining the intra-box feature points located within the initial target object box from the first feature points based on the position of the initial target object box includes:

[0035] Based on the initial target object frame, determine the straight lines corresponding to the four sides of the initial target object frame;

[0036] Based on the positional relationship between the first feature point and the corresponding straight lines of the four sides of the initial target object box, the feature points within the first box are determined;

[0037] Based on the motion-correspondence transformation model and the first feature point, determine the background motion flow features generated by the camera motion of the image frame to be analyzed;

[0038] Based on the difference between the flow feature corresponding to the first feature point and the background motion flow feature, the foreground motion flow feature corresponding to the foreground motion of the image frame to be analyzed is determined;

[0039] Filter out feature points in the first bounding box whose foreground motion flow feature magnitude is less than a first magnitude threshold, and use the filtered feature points in the first bounding box as the feature points in the bounding box.

[0040] Optionally, the selection of the camera motion corresponding transformation model includes:

[0041] Based on the width and height of the image frame to be analyzed, and the width and height of the flow feature corresponding to the first feature point, the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion are obtained sequentially using the iterative weighted least squares method IRLS, according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, until the acquisition is successful.

[0042] Based on the transformation parameters of the successfully acquired camera motion, determine the transformation matrix corresponding to the transformation parameters;

[0043] The transformation matrix is ​​used as the transformation model corresponding to camera motion.

[0044] Optionally, the step of obtaining the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion sequentially using iterative weighted least squares (IRLS) according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, until successful acquisition, includes:

[0045] Obtain the width and height of the image frame to be analyzed;

[0046] Obtain the flow feature corresponding to the first feature point and the width and height of the flow feature corresponding to the first feature point;

[0047] The width and height of the image frame to be analyzed are normalized based on the diagonal length, width, and height of the image frame to be analyzed.

[0048] Based on the diagonal length of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, normalize the width and height of the flow feature corresponding to the first feature point.

[0049] The normalized width and height of the image frame to be analyzed are used as the width and height of the image frame to be analyzed.

[0050] The width and height of the normalized flow feature are used as the width and height of the flow feature corresponding to the first feature point.

[0051] Optionally, the step of obtaining the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion sequentially using iterative weighted least squares (IRLS) according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, until successful acquisition, includes:

[0052] Based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, the IRLS weight homography of the flow feature corresponding to the first feature point is initialized using a Gaussian distribution;

[0053] The average moving modulus is determined based on the modulus of the flow feature corresponding to the first feature point.

[0054] Based on the average moving modulus and the weights of the flow features corresponding to the first feature point, the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion are obtained sequentially through the IRLS algorithm in the order of priority of homography transformation, affine transformation, linear similarity transformation, and translation transformation until the acquisition is successful.

[0055] Optionally, determining the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed includes:

[0056] Determine the position scaling amount based on the initial target position and the current target position;

[0057] Based on the position scaling amount, camera parameters, the initial target position, and the current target position, determine the three-dimensional coordinates of the target object in the initial frame and the three-dimensional coordinates of the target object in the image frame to be analyzed;

[0058] Obtain the first rotation amount of the target object in the initial frame and the second rotation amount of the target object in the image frame to be analyzed;

[0059] Based on the first rotation amount and the second rotation amount, determine the rotational change of the target object;

[0060] The translational change of the target object is determined based on the rotational change, the three-dimensional coordinates of the target object in the initial frame, and the three-dimensional coordinates of the target object in the image frame to be analyzed.

[0061] The pose of the target object in the image frame to be analyzed is determined based on the rotation change, the translation change, and the initial pose of the target object in the initial frame.

[0062] Secondly, embodiments of this application provide a pose determination system, including:

[0063] An initial module is used to obtain the initial pose and initial target position of the target object in an initial frame, wherein the initial frame is the first image frame containing the target object;

[0064] The feature module is used to obtain the first feature point of the image frame to be analyzed and the flow feature corresponding to the first feature point. The flow feature is the position difference between the feature point position of the previous frame of the image frame to be analyzed and the first feature point position. The feature point position of the previous frame of the image frame to be analyzed is obtained by optical flow tracking.

[0065] The location module is used to determine the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow features corresponding to the first feature point;

[0066] The pose module is used to determine the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed.

[0067] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the pose determination method as described in the first aspect above.

[0068] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the pose determination method described in the first aspect above.

[0069] In this application, the initial pose and initial target position of the target object in the initial frame are first obtained; the first feature point and the flow feature corresponding to the first feature point of the image frame to be analyzed are obtained; based on the initial target position and the flow feature corresponding to the first feature point, the current target position of the target object in the image frame to be analyzed is determined. Since the flow feature is the position difference between the feature point position of the previous frame of the image frame to be analyzed and the first feature point position obtained by optical flow method, each image frame to be analyzed will be backtracked to the feature point of the previous frame by optical flow method to obtain the flow feature. Therefore, the feature points obtained in each image frame to be analyzed will not gradually decrease, and the error of the current target position of image tracking will not gradually increase. Therefore, the error of determining the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed will not gradually accumulate, so that the calculated pose can maintain high accuracy and stability. Even if the target image is not detected, image tracking and pose calculation can still be performed on the image frame to be analyzed according to the method described in this application, which enhances robustness. Attached Figure Description

[0070] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0071] Figure 1 This is a flowchart of a pose determination method provided in an embodiment of this application;

[0072] Figure 2 This is an overall flowchart of a pose determination method provided in an embodiment of this application;

[0073] Figure 3 This is a flowchart of an image detector provided in an embodiment of this application;

[0074] Figure 4 This is a flowchart of an image tracker algorithm provided in an embodiment of this application;

[0075] Figure 5-1 This is a schematic diagram of the first type of image segmentation provided in the embodiments of this application;

[0076] Figure 5-2 This is a schematic diagram of the second type of image segmentation provided in the embodiments of this application;

[0077] Figure 6-1 This is a schematic diagram of the pose of an initial image frame provided in an embodiment of this application;

[0078] Figure 6-2 This is a schematic diagram of the pose of an image frame to be analyzed, provided in an embodiment of this application;

[0079] Figure 7 This is a schematic diagram of the structure of a pose determination system provided in an embodiment of this application;

[0080] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0081] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0082] Please refer to Figure 1 This application provides a pose determination method, including:

[0083] Step 11: Obtain the initial pose and initial target position of the target object in the initial frame, wherein the initial frame is the first image frame containing the target object;

[0084] Step 12: Obtain the first feature point and the flow feature corresponding to the first feature point of the image frame to be analyzed, wherein the flow feature is the position difference between the feature point position of the previous frame of the image frame to be analyzed and the first feature point position, and the feature point position of the previous frame of the image frame to be analyzed is obtained by optical flow tracking.

[0085] Step 13: Determine the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow features corresponding to the first feature point;

[0086] Step 14: Determine the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed.

[0087] In this application, the initial pose and initial target position of the target object in the initial frame are first obtained; the first feature point and the flow feature corresponding to the first feature point of the image frame to be analyzed are obtained; based on the initial target position and the flow feature corresponding to the first feature point, the current target position of the target object in the image frame to be analyzed is determined. Since the flow feature is the position difference between the feature point position of the previous frame of the image frame to be analyzed and the first feature point position obtained by optical flow method, each image frame to be analyzed will be backtracked to the feature point of the previous frame by optical flow method to obtain the flow feature. Therefore, the feature points obtained in each image frame to be analyzed will not gradually decrease, and the error of the current target position of image tracking will not gradually increase. Therefore, the error of determining the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed will not gradually accumulate, so that the calculated pose can maintain high accuracy and stability. Even if the target image is not detected, image tracking and pose calculation can still be performed on the image frame to be analyzed according to the method described in this application, which enhances robustness.

[0088] The overall technical solution of this application is as follows: Figure 2 As shown, the pose determination method described in this application can be implemented using a detector and a tracker, specifically:

[0089] First, the detector is initialized, a preset target template is loaded, and target object detection is performed on the incoming video frame (i.e., image frame) to obtain the initial pose of the target object in the image frame to be analyzed, i.e., the initial pose of the initial frame. When the detector detects the target object, the tracker is initialized using the detected initial pose of the target object, and the tracker is used to track the target object in the image frame to be analyzed. Finally, when the tracked target object does not go out of bounds, the current target position and the acquired rotation amount of the target object in the image frame to be analyzed are used to estimate the pose of the target object in the image frame to be analyzed. Otherwise, the detector is activated again to detect, and the above steps are repeated.

[0090] Specifically, in this embodiment of the application, optionally, before obtaining the initial pose and initial target position of the target object in the initial frame, the method further includes:

[0091] Perform target object detection on the current frame to determine whether there is a target object in the current frame;

[0092] If the target object is present in the current frame, the current frame is used as the image frame to be analyzed.

[0093] In this embodiment, it is first determined whether there is a target object in the current frame, and the current frame containing the target object is taken as the image frame to be analyzed. Furthermore, the initial pose of the target object in the image frame to be analyzed is obtained, that is, the initial pose of the initial frame.

[0094] In this embodiment of the application, by first determining whether there is a target object in the current frame and then selecting whether to perform image tracking, the target object can be tracked more accurately, and errors caused by the absence of a target object can be avoided, thereby improving the accuracy and reliability of real-time tracking.

[0095] Furthermore, in this embodiment of the application, optionally, obtaining the initial pose of the target object in the initial frame includes:

[0096] Obtain the template feature points of the template to be identified and the first descriptor corresponding to the template feature points, wherein the template to be identified is an image containing only the target object;

[0097] Feature points are extracted from the initial frame to obtain the initial feature points of the initial frame and the second descriptor corresponding to the initial feature points;

[0098] The second descriptor and the first descriptor are matched to obtain the matching relationship between the initial feature points and the template feature points;

[0099] The initial pose of the target object in the initial frame is determined based on the matching relationship between the initial feature points and the template feature points.

[0100] Please refer to Figure 3 The initial pose can be obtained by an image detector. The specific operation process includes:

[0101] For templates ( Figure 3 Feature extraction is performed on the 2D image to be identified and the real-time acquired video frames. Then, 1v1 matching is performed on the feature point descriptors extracted from both to obtain the matching relationship between the feature points. Next, the Iterative Closest Point (ICP) algorithm is used to solve the pose of the current video frame, i.e., the view matrix of the model, and this view matrix is ​​used as the initial pose T of the target object. init .

[0102] In this embodiment of the application, feature matching using descriptors can effectively reduce the number of feature points while still ensuring accurate feature matching. Moreover, it can more accurately locate the object position, which is beneficial for further image processing.

[0103] For further details, please refer to... Figure 4 This application can perform image tracking using a 2D image tracker. The 2D image tracker algorithm mainly includes three steps: calculating the region flow features, fitting the weighted motion vector using Iteratively Reweighted Least Squares (IRLS), and estimating the target position.

[0104] The calculation of region flow features involves dividing the image into blocks of different sizes, and then filtering the flow features within the image blocks using number thresholds and modulus thresholds to obtain region flow features. Specifically:

[0105] In this embodiment of the application, optionally, obtaining the first feature point of the image frame to be analyzed includes:

[0106] Feature point extraction is performed on the image frame to be analyzed to obtain all feature points of the image frame to be analyzed;

[0107] The optical flow method is used to track the image frame to be analyzed, and the position of the previous feature point corresponding to the previous frame of the image frame to be analyzed is obtained.

[0108] Based on the difference between the position of the previous feature point and the position of the current feature point, determine the flow features corresponding to all feature points;

[0109] The image frame to be analyzed is divided into blocks of at least one size to obtain at least one image frame to be analyzed with multiple image blocks;

[0110] Obtain the number of feature points within all image patches;

[0111] Filter out image blocks in which the number of feature points is less than a threshold, and obtain the first filtered image block;

[0112] Based on the flow feature magnitude corresponding to the feature points in the first image block, the feature points in the first image block are filtered to obtain the internal feature points in the first image block.

[0113] Merge all internal feature points within all image blocks and use the internal feature points as the first feature points. Merging all internal feature points within all image blocks means retaining non-repeating internal feature points in all image blocks and merging and retaining internal feature points that are repeated at the same position in all image blocks.

[0114] Traditional optical flow methods extract feature points from the first frame and then use optical flow for tracking in subsequent frames, which leads to a large cumulative error. This application's embodiment differs from traditional optical flow methods by extracting feature points from each video frame being analyzed, thus obtaining all feature points of the frame being analyzed. i = 1, 2, ..., M, then optical flow tracking is used to obtain the position of the feature point corresponding to the previous frame of the image frame to be analyzed. i = 1, 2, ..., M, where each feature point corresponds to a coordinate (x, y), where x represents the horizontal coordinate of the feature point in the image frame and y represents the vertical coordinate of the feature point in the image frame. The flow feature corresponding to each feature point is calculated. i The calculation formula is as follows:

[0115]

[0116] in, The location of the feature point corresponding to the previous frame of the image to be analyzed. This represents the locations of all feature points in the frame to be analyzed.

[0117] The large number of feature points and corresponding flow features obtained through the above application embodiments may increase the computational and storage requirements in subsequent calculations. Furthermore, redundant feature points may interfere with the detection of other feature points, reducing the accuracy and stability of image tracking. Therefore, in order to further extract more obvious feature points, this application embodiment further filters feature points by dividing the image into blocks.

[0118] In this application embodiment, the image frame to be analyzed is divided into blocks of at least one size to obtain at least one image frame to be analyzed having multiple image blocks. For example, please refer to... Figure 5-1 and Figure 5-2 The image frames to be analyzed were divided into blocks with two block sizes: (160*120) and (80*60). Figure 5-1 The block size is (160*120). Figure 5-2 The image is divided into blocks of size (80*60), with each block corresponding to one... Where i = 1, 2, 3... N j j = 1, 2, j = 1 represents an image block with a block size of (160*120), j = 2 represents an image block with a block size of (160*120), i represents the i-th cell, and the features are assigned to the corresponding image blocks.

[0119] Secondly, the first image block in which the number of feature points is less than the threshold is filtered out. The formula for calculating the threshold is as follows:

[0120]

[0121] in, For the i-th image patch of the j-th size; N j There are N image blocks of the j-th size. j Image blocks.

[0122] Next, the first image block obtained after filtering the number of feature points using Random Sample Consensus (RANSAC) is used to filter feature points, thereby obtaining the corresponding interior points (i.e., interior feature points) of the image frame to be analyzed. The specific interior point calculation process is as follows:

[0123] Step a: For each block, randomly select a feature point and use the magnitude of the flow feature corresponding to the current feature point to obtain a magnitude threshold. m The calculation formula is shown below:

[0124] threshold m =0.1*sqrt(pow(flow.x,2)+pow(flow.y,2));

[0125] Where flow.x is the width of the current flow feature flow; flow.y is the height of the current flow feature flow; pow represents exponential calculation; sqrt represents square root operation.

[0126] Step b: Calculate the flow features for each stream in the current block. The magnitude of the deviation from the current flow characteristic The calculation formula is shown below:

[0127]

[0128]

[0129] in, The current block contains the flow features; flow represents the current flow features. For the current flow feature and each flow feature Deviation; The modulus of deviation.

[0130] Step c: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Feature points are added to a temporary inner point set. When the number of feature points in the temporary inner point dataset is greater than 80% of the number of feature points in the current block, or the number of iterations is greater than 15, the iteration stops, and the feature points in the temporary feature point dataset are set as inner points. If a randomly selected feature point does not make the number of feature points in the temporary inner point dataset greater than 80% of the number of feature points in the current block, the above steps ac are repeated until the number of feature points in the temporary inner point dataset is greater than 80% of the number of feature points in the current block, or the number of iterations is greater than 15.

[0131] Step d: Merge all interior points within all image blocks. This involves merging all non-repeating interior points obtained from the two block divisions, retaining all non-repeating interior points in all blocks, and merging and retaining interior points that are repeated at the same position in all blocks. The flow features corresponding to all interior points of the image frame to be analyzed constitute the region flow features corresponding to the image frame to be analyzed.

[0132] The embodiments of this application use the modulus of the flow feature corresponding to the feature point as the basis for filtering the feature point, which can effectively ignore background interference and obtain accurate feature point positions, thereby reducing the error of the feature point and further improving the image processing efficiency.

[0133] To further obtain motion vectors, this application calculates the flow features corresponding to camera motion and foreground motion respectively. This application provides a method for IRLS fitting weighted motion vectors to determine the transformation model corresponding to camera motion. In the process of IRLS fitting weighted motion vectors, the maximum and minimum values ​​of the flow feature magnitudes are removed to obtain a rough camera motion. At this time, the obtained camera motion is greater than or equal to the actual camera motion and less than or equal to the camera motion superimposed on the object motion. The obtained camera motion is then used to filter out feature points and flow features that are irrelevant to motion. The remaining inliers can be used to accurately estimate the motion of objects within the target box of the candidate image frame.

[0134] First, based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion are obtained sequentially using the iterative weighted least squares method (IRLS) according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, until the parameters are successfully obtained.

[0135] Specifically, in this embodiment of the application, optionally, the step of obtaining the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion sequentially using the iterative weighted least squares (IRLS) method according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, until successful acquisition, includes:

[0136] Obtain the width and height of the image frame to be analyzed;

[0137] Obtain the flow feature corresponding to the first feature point and the width and height of the flow feature corresponding to the first feature point;

[0138] The width and height of the image frame to be analyzed are normalized based on the diagonal length, width, and height of the image frame to be analyzed.

[0139] Based on the diagonal length of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, normalize the width and height of the flow feature corresponding to the first feature point.

[0140] The normalized width and height of the image frame to be analyzed are used as the width and height of the image frame to be analyzed.

[0141] The width and height of the normalized flow feature are used as the width and height of the flow feature corresponding to the first feature point.

[0142] In this embodiment of the application, before obtaining the motion vector, the width and height of the normalized image frame are used to estimate the motion vector, which can reduce the amount of computation, effectively reduce errors, maintain the accuracy of feature extraction, and improve the accuracy and reliability of real-time tracking. Specifically, this is achieved through the following step A.

[0143] Step A: Normalize the width and height of the image frame to be analyzed to Nw and Nh, respectively, and normalize the extracted flow features (first-flow features) of each region to Nf. i The calculation formula is shown below:

[0144]

[0145] Where width is the width of the image frame to be analyzed; height is the height of the image frame to be analyzed; norm corresponds to the length of the image diagonal; flow i .x represents the flow feature. i width; flow i .y represents the flow characteristic. i of high.

[0146] Further, in this embodiment, optionally, the step of obtaining the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion sequentially using iterative weighted least squares (IRLS) according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, until successful acquisition, includes:

[0147] Based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, the IRLS weight homography of the flow feature corresponding to the first feature point is initialized using a Gaussian distribution;

[0148] The average moving modulus is determined based on the modulus of the flow feature corresponding to the first feature point.

[0149] Based on the average moving modulus and the weights of the flow features corresponding to the first feature point, the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion are obtained sequentially through the IRLS algorithm in the order of priority of homography transformation, affine transformation, linear similarity transformation, and translation transformation until the acquisition is successful.

[0150] In this embodiment, iteratively obtaining camera motion using the IRLS algorithm enables fast and accurate estimation of camera motion, exhibiting robustness and accuracy. Specifically, this is achieved through the following steps.

[0151] Step B: Initialize the IRLS weighted homography hw for each region flow feature (first feature point) using a Gaussian distribution. i The calculation formula is shown below:

[0152]

[0153] Step C: Estimate the average moving norm (avg_move_norm). First, calculate the norm of the flow characteristics for each region. i Then, the module lengths are sorted, and the largest and smallest 10% of module lengths are removed respectively; finally, the average moving module length avg_move_norm is obtained by averaging the remaining module lengths, and the calculation formula is shown in the following formula:

[0154]

[0155] Here, avg_move_norm' is the average modulus calculated based on the width and height of the normalized flow characteristics.

[0156] Furthermore, in this embodiment of the application, optionally, the selection of the camera motion corresponding transformation model includes:

[0157] Based on the width and height of the image frame to be analyzed, and the width and height of the flow feature corresponding to the first feature point, the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion are obtained sequentially using the iterative weighted least squares method IRLS, according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, until the acquisition is successful.

[0158] Based on the transformation parameters of the successfully acquired camera motion, determine the transformation matrix corresponding to the transformation parameters;

[0159] The transformation matrix is ​​used as the transformation model corresponding to camera motion.

[0160] In this application, embodiments for obtaining different variable model parameters are given.

[0161] Translation Transformation: Estimating the translation transformation T: dx, dy, M of camera motion T This is the translation transformation matrix.

[0162] First, assign the IRL weights w to each feature. i Set it to 1.0f, then calculate the current weighted average moving average T: dx,dy, and based on T and diff i Update the irls weights w for each feature i As shown in the following equation, by repeating the iteration step ten times, the parameters T: dx, dy of the translation transformation motion and the corresponding translation transformation matrix M can be obtained. T .

[0163]

[0164]

[0165] Among them, diff i.x diff i width; diff i.y diff i The height; the scale calculation formula is as follows:

[0166]

[0167] Linear similarity transformation: A linear similarity transformation S: dx, dy, a, b, M for estimating camera motion. S Let be the linear similarity transformation matrix.

[0168] First, assign the IRL weights w to each feature. i Set it to 1.0f, and then construct the least squares matrix A. i As shown in the following equation, orthogonal matrix decomposition (QR) is used to obtain the corresponding similar models S: dx, dy, a, b, and the irls weights w of each feature are updated. i Where scale is the same as in the translation transformation calculation formula above, repeating the above iterative steps ten times will yield the linear similarity transformation S: dx, dy, a, b and the corresponding linear similarity transformation matrix M. S .

[0169]

[0170]

[0171] Where sim is the linear similarity transformation matrix M S .

[0172] Estimating Affine Transformations: Estimating the affine transformations P: d_x, d_y, a, b, c, d, M for estimating camera motion. P Linear similarity transformation matrix.

[0173] First, assign the IRL weights w to each feature. i Set it to 1.0f, and then construct the least squares matrix J. i As shown in the following equation, orthogonal matrix decomposition (QR) is used to obtain the corresponding similar models p:dx,dy,a,b,c,d, and the irls weights w of each feature are updated. i Where scale is the same as in the translation transformation calculation formula above, repeating the above iterative steps ten times will yield the linear similarity transformation P: d_x, d_y, a, b, c, d and the corresponding estimated affine transformation matrix M. P The calculation formula is shown below.

[0174]

[0175]

[0176] Where p is the matrix M of the linear similarity transformation. P .

[0177] The homography H: a, b, t1, c, d, t2, w1, w2 and the corresponding homography matrix M are estimated using a method similar to that described above. H .

[0178]

[0179] Furthermore, based on the transformation obtained above, the true flow characteristics normalized to a size of 256x192 are solved.

[0180] First, according to the priority order of homography, affine transformation, similarity transformation, and translation transformation, the corresponding matrices for the four transformations are M. H M P M S M T As shown in the following equation, the camera motion corresponding transformation model M is selected.

[0181]

[0182] In this embodiment, the camera motion corresponding transformation model is obtained based on the regional flow characteristics, which can more accurately estimate the camera motion and has higher accuracy and reliability.

[0183] It is worth noting that all the results obtained from the above steps are calculations of the motion from the current frame to the previous frame. Therefore, the actual motion from the previous frame to the current frame can be obtained by multiplying the above results by -1.

[0184] Furthermore, this application provides a method for calculating the initial target position (target bounding box position), specifically:

[0185] In this embodiment of the application, optionally, the initial target position is the position of the initial target object box containing the target object in the initial frame;

[0186] Determining the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow features corresponding to the first feature point includes:

[0187] Based on the position of the initial target object box, obtain the in-frame feature points of the first feature points that are located within the initial target object box;

[0188] Select the camera motion corresponding transformation model;

[0189] Obtain the flow features corresponding to the feature points within the frame;

[0190] The current target position of the target object in the image frame to be analyzed is obtained by using the camera motion corresponding transformation model and the flow features corresponding to the feature points in the frame.

[0191] In this embodiment of the application, a method is provided to estimate the current target position of a target object in an image frame to be analyzed by using an initial target position and flow features. The target box position estimation process is as follows:

[0192] Step aa: Based on the initial target object box position passed in, calculate the straight lines corresponding to the four sides of the initial target object box;

[0193] Step bb: Based on the positional relationship between the inner points and the four lines extracted in the above application embodiment, the BoxInliers (inside the box) and BoxOutliers (outside the box) are obtained. Specifically:

[0194] In this embodiment of the application, optionally, obtaining the intra-frame feature points located within the initial target object frame from the first feature points based on the position of the initial target object frame includes:

[0195] Based on the initial target object frame, determine the straight lines corresponding to the four sides of the initial target object frame;

[0196] Based on the positional relationship between the first feature point and the corresponding straight lines of the four sides of the initial target object box, the feature points within the first box are determined;

[0197] Based on the motion-correspondence transformation model and the first feature point, determine the background motion flow features generated by the camera motion of the image frame to be analyzed;

[0198] Based on the difference between the flow feature corresponding to the first feature point and the background motion flow feature, the foreground motion flow feature corresponding to the foreground motion of the image frame to be analyzed is determined;

[0199] Filter out feature points in the first bounding box whose foreground motion flow feature magnitude is less than a first magnitude threshold, and use the filtered feature points in the first bounding box as the feature points in the bounding box.

[0200] The calculation formulas for background motion flow features and foreground motion flow features are as follows:

[0201] The extracted regional flow features are subtracted from the background motion flow caused by camera motion. i background This will give you the flow corresponding to the foreground motion in the image. i foreground The calculation formula is shown below:

[0202]

[0203] After obtaining the foreground motion flow features,

[0204] Step cc: Filter out foreground motion flow in BoxInliers (feature points within the bounding box).i foreground For features with a modulus less than 0.0125, the translation, similarity, affine, and homography matrices corresponding to the BoxInliers feature points are estimated sequentially using the IRLS fitting weighted motion vector method described above.

[0205] Step dd: Transform the four vertices of the previous frame according to the priority order of homography, affine transformation, linear similarity transformation and translation transformation to obtain the current target position of the target object in the image frame to be analyzed.

[0206] In this embodiment of the application, the current target position of the target object in the image frame to be analyzed obtained according to the above method can realize 2D image tracking. Each image frame to be analyzed will be backtracked to the feature points of the previous frame according to the optical flow method, and then the flow features are obtained. Therefore, the feature points obtained in each image frame to be analyzed will not gradually decrease, and the error of the current target position of image tracking will not gradually increase. Furthermore, pose calculation is performed based on the target position obtained by image tracking.

[0207] In this embodiment of the application, optionally, determining the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed includes:

[0208] Determine the position scaling amount based on the initial target position and the current target position;

[0209] Based on the position scaling amount, camera parameters, the initial target position, and the current target position, determine the three-dimensional coordinates of the target object in the initial frame and the three-dimensional coordinates of the target object in the image frame to be analyzed;

[0210] Obtain the first rotation amount of the target object in the initial frame and the second rotation amount of the target object in the image frame to be analyzed;

[0211] Based on the first rotation amount and the second rotation amount, determine the rotational change of the target object;

[0212] The translational change of the target object is determined based on the rotational change, the three-dimensional coordinates of the target object in the initial frame, and the three-dimensional coordinates of the target object in the image frame to be analyzed.

[0213] The pose of the target object in the image frame to be analyzed is determined based on the rotation change, the translation change, and the initial pose of the target object in the initial frame.

[0214] Furthermore, this application embodiment provides a method for determining the pose of a target object in an image frame to be analyzed by determining the position scaling amount based on the initial target position and the current target position, and then introducing a rotation change amount, based on the current target position obtained according to the above pose determination method. By introducing the position scaling amount, the amount of computation can be effectively reduced and the accuracy of position estimation can be improved. Introducing the rotation change amount can effectively suppress the error caused by position measurement and improve accuracy and reliability, improve the tracking effect, and help achieve more accurate pose estimation.

[0215] Specifically, taking plants as an example, please refer to... Figure 6-1 and Figure 6-2 In the figure, the dashed boxes represent the initial target position of the plant in the initial frame and the current target position of the target object in the image frame to be analyzed.

[0216] Step aaa: Record the rotation matrix R corresponding to the initial frame. init The rotation data is acquired by the Inertial Measurement Unit (IMU), and the scaling factor (scale) of the image frame to be analyzed relative to the initial frame is calculated based on the initial target object bounding box Box1 in the initial frame (including the center point position (cu1, cv1) and the width and height of the box (w1, h1)) and the current target object bounding box Box2 in the image frame to be analyzed (including the center point position (cu2, cv2) and the width and height of the box (w2, h2)). The calculation formula is shown in the following formula:

[0217]

[0218] Step bbb, using camera parameter f x ,f y ,c x ,c y These parameters are inherent parameters of the camera, generally obtained through calibration. The initial target object position Box1 on the 2D image is calculated, corresponding to its center position in 3D coordinates (X, Y, F). init ,Y init Z init ), calculate the three-dimensional coordinates (X, Y, F) corresponding to the center position of the current target object box Box2. curr ,Y curr Z curr The calculation formula is shown below:

[0219]

[0220]

[0221] Where z represents the manually set model placement position, fx and fy are the camera focal lengths (fx, fy) in the image coordinate system, and cx and cy are the optical centers (cx, cy) of the camera in the image coordinate system.

[0222] Step ccc: Record the IMU rotation matrix R corresponding to the image frame to be analyzed. curr Calculate the rotational change R of the target object in the image frame to be analyzed relative to the target object in the initial frame. relate The calculation formula is shown below:

[0223] R relate =R curr -1 *R init .

[0224] Step ddd: Calculate the translational change t of the target object in three-dimensional space. relate The calculation formula is shown below:

[0225]

[0226] Step eee yields the transformation matrix T of the target object in the image frame to be analyzed relative to the target object in the initial frame. relate The calculation formula is shown below:

[0227]

[0228] Wherein, the transformation matrix T relate The bottom left element is 0, and the bottom right element is 1.

[0229] After step fff, the final 6DOF target object pose is T, calculated using the following formula:

[0230] T = T relate *T init ;

[0231] Among them, T init This is the initial pose.

[0232] This application proposes a pose determination method. First, the initial pose of the target object in the image is obtained by using a preset image detector. Then, a robust tracker (2D image tracker) is used to obtain the position of the 2D image in the image. Second, the 3D rotation of the target object is obtained using the IMU data corresponding to the image frame to be analyzed. Finally, the relative translation and scaling of the target object's position in the image frame to be analyzed compared to its initial position are calculated. The 3D translation of the target object is estimated using camera parameters, thereby obtaining the 6DOF pose of the current target object. This application avoids the problem of feature points becoming increasingly scarce. The feature points obtained in each image frame to be analyzed do not gradually decrease, and the error of the current target position in image tracking does not gradually increase. Therefore, the error in determining the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed will not gradually accumulate. This allows the calculated pose to maintain high accuracy and stability. Furthermore, even if the target image is not detected, image tracking and pose calculation can still be performed on the image frame to be analyzed according to the method described in this application, enhancing robustness.

[0233] Please refer to Figure 7 This application also provides a pose determination system 70, comprising:

[0234] The initial module 71 is used to obtain the initial pose and initial target position of the target object in the initial frame, wherein the initial frame is the first image frame containing the target object;

[0235] Feature module 72 is used to obtain a first feature point of the image frame to be analyzed and the flow feature corresponding to the first feature point, wherein the flow feature is the position difference between the feature point position of the previous frame of the image frame to be analyzed and the first feature point position, and the feature point position of the previous frame of the image frame to be analyzed is obtained by optical flow tracking.

[0236] The position module 73 is used to determine the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow features corresponding to the first feature point;

[0237] The pose module 74 is used to determine the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed.

[0238] In this embodiment of the application, optionally, the pose determination system 70 further includes:

[0239] The judgment module is used to perform target object detection on the current frame and determine whether there is a target object in the current frame;

[0240] If the target object is present in the current frame, the current frame is used as the image frame to be analyzed.

[0241] In this embodiment of the application, optionally, the initial module 71 includes:

[0242] The initial submodule is used to obtain the template feature points of the template to be identified and the first descriptor corresponding to the template feature points, wherein the template to be identified is an image containing only the target object;

[0243] Feature points are extracted from the initial frame to obtain the initial feature points of the initial frame and the second descriptor corresponding to the initial feature points;

[0244] The second descriptor and the first descriptor are matched to obtain the matching relationship between the initial feature points and the template feature points;

[0245] The initial pose of the target object in the initial frame is determined based on the matching relationship between the initial feature points and the template feature points.

[0246] In this embodiment of the application, optionally, the feature module 72 includes:

[0247] The feature submodule is used to extract feature points from the image frame to be analyzed and obtain all feature points of the image frame to be analyzed.

[0248] The optical flow method is used to track the image frame to be analyzed, and the position of the previous feature point corresponding to the previous frame of the image frame to be analyzed is obtained.

[0249] Based on the difference between the position of the previous feature point and the position of the current feature point, determine the flow features corresponding to all feature points;

[0250] The image frame to be analyzed is divided into blocks of at least one size to obtain at least one image frame to be analyzed with multiple image blocks;

[0251] Obtain the number of feature points within all image patches;

[0252] Filter out image blocks in which the number of feature points is less than a threshold, and obtain the first filtered image block;

[0253] Based on the flow feature magnitude corresponding to the feature points in the first image block, the feature points in the first image block are filtered to obtain the internal feature points in the first image block.

[0254] Merge all internal feature points within all image blocks and use the internal feature points as the first feature points. Merging all internal feature points within all image blocks means retaining non-repeating internal feature points in all image blocks and merging and retaining internal feature points that are repeated at the same position in all image blocks.

[0255] In this embodiment of the application, optionally, the initial target position is the position of the initial target object box containing the target object in the initial frame;

[0256] The location module 73 includes:

[0257] The position submodule is used to obtain the in-frame feature points of the first feature points that are located within the initial target object frame, based on the position of the initial target object frame.

[0258] Select the camera motion corresponding transformation model;

[0259] Obtain the flow features corresponding to the feature points within the frame;

[0260] The current target position of the target object in the image frame to be analyzed is obtained by using the camera motion corresponding transformation model and the flow features corresponding to the feature points in the frame.

[0261] In this embodiment of the application, optionally, the location submodule includes:

[0262] The Position 1 submodule is used to determine the straight lines corresponding to the four sides of the initial target object frame based on the initial target object frame;

[0263] Based on the positional relationship between the first feature point and the corresponding straight lines of the four sides of the initial target object box, the feature points within the first box are determined;

[0264] Based on the motion-correspondence transformation model and the first feature point, determine the background motion flow features generated by the camera motion of the image frame to be analyzed;

[0265] Based on the difference between the flow feature corresponding to the first feature point and the background motion flow feature, the foreground motion flow feature corresponding to the foreground motion of the image frame to be analyzed is determined;

[0266] Filter out feature points in the first bounding box whose foreground motion flow feature magnitude is less than a first magnitude threshold, and use the filtered feature points in the first bounding box as the feature points in the bounding box.

[0267] In this embodiment of the application, optionally, the location submodule includes:

[0268] The second position submodule is used to obtain the parameters of the camera motion homography, affine transformation, linear similarity transformation and translation transformation in sequence according to the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, using the iterative weighted least squares method IRLS until the acquisition is successful.

[0269] Based on the transformation parameters of the successfully acquired camera motion, determine the transformation matrix corresponding to the transformation parameters;

[0270] The transformation matrix is ​​used as the transformation model corresponding to camera motion.

[0271] In this embodiment of the application, optionally, the location submodule includes:

[0272] The location submodule is used to obtain the width and height of the image frame to be analyzed;

[0273] Obtain the flow feature corresponding to the first feature point and the width and height of the flow feature corresponding to the first feature point;

[0274] The width and height of the image frame to be analyzed are normalized based on the diagonal length, width, and height of the image frame to be analyzed.

[0275] Based on the diagonal length of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, normalize the width and height of the flow feature corresponding to the first feature point.

[0276] The normalized width and height of the image frame to be analyzed are used as the width and height of the image frame to be analyzed.

[0277] The width and height of the normalized flow feature are used as the width and height of the flow feature corresponding to the first feature point.

[0278] In this embodiment of the application, optionally, the location submodule includes:

[0279] The position 2 sub-module is used to initialize the IRLS weight homography of the flow feature corresponding to the first feature point using a Gaussian distribution based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point.

[0280] The average moving modulus is determined based on the modulus of the flow feature corresponding to the first feature point.

[0281] Based on the average moving modulus and the weights of the flow features corresponding to the first feature point, the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion are obtained sequentially through the IRLS algorithm in the order of priority of homography transformation, affine transformation, linear similarity transformation, and translation transformation until the acquisition is successful.

[0282] In this embodiment of the application, optionally, the pose module 74 includes:

[0283] The pose submodule is used to determine the position scaling amount based on the initial target position and the current target position;

[0284] Based on the position scaling amount, camera parameters, the initial target position, and the current target position, determine the three-dimensional coordinates of the target object in the initial frame and the three-dimensional coordinates of the target object in the image frame to be analyzed;

[0285] Obtain the first rotation amount of the target object in the initial frame and the second rotation amount of the target object in the image frame to be analyzed;

[0286] Based on the first rotation amount and the second rotation amount, determine the rotational change of the target object;

[0287] The translational change of the target object is determined based on the rotational change, the three-dimensional coordinates of the target object in the initial frame, and the three-dimensional coordinates of the target object in the image frame to be analyzed.

[0288] The pose of the target object in the image frame to be analyzed is determined based on the rotation change, the translation change, and the initial pose of the target object in the initial frame.

[0289] The pose determination system provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiment shown in Figure 6 achieve the same technical effect, and will not be described again here to avoid repetition.

[0290] This application provides an electronic device 80, see [link to relevant documentation] Figure 8 As shown, Figure 8 This is a schematic block diagram of an electronic device 80 according to an embodiment of this application, including a processor 81, a memory 82, and a program or instructions stored in the memory 82 and executable on the processor 81. When the program or instructions are executed by the processor, they implement the steps in any pose determination method of this application.

[0291] This application provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements the various processes of the pose determination method embodiment as described above and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0292] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0293] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0294] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0295] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a service classification device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0296] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A pose determination method, characterized in that, include: Obtain the initial pose and initial target position of the target object in the initial frame, wherein the initial frame is the first image frame containing the target object; The first feature point and the flow feature corresponding to the first feature point of the image frame to be analyzed are obtained. The flow feature is the position difference between the feature point position of the previous frame of the image frame to be analyzed and the position of the first feature point. The feature point position of the previous frame of the image frame to be analyzed is obtained by optical flow tracking. Based on the initial target position and the flow features corresponding to the first feature point, the current target position of the target object in the image frame to be analyzed is determined; The pose of the target object in the image frame to be analyzed is determined based on the initial pose and the current target position of the target object in the image frame to be analyzed. Wherein, the initial target position is the position of the initial target object box containing the target object in the initial frame; Determining the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow features corresponding to the first feature point includes: Based on the position of the initial target object box, obtain the in-frame feature points of the first feature points that are located within the initial target object box; Select the camera motion corresponding transformation model; Obtain the flow features corresponding to the feature points within the frame; The current target position of the target object in the image frame to be analyzed is obtained by using the camera motion corresponding transformation model and the flow features corresponding to the feature points in the frame; The step of obtaining the intra-box feature points located within the initial target object box based on the position of the initial target object box includes: Based on the initial target object frame, determine the straight lines corresponding to the four sides of the initial target object frame; Based on the positional relationship between the first feature point and the corresponding straight lines of the four sides of the initial target object box, the feature points within the first box are determined; Based on the motion-correspondence transformation model and the first feature point, determine the background motion flow features generated by the camera motion of the image frame to be analyzed; Based on the difference between the flow feature corresponding to the first feature point and the background motion flow feature, the foreground motion flow feature corresponding to the foreground motion of the image frame to be analyzed is determined; Filter out feature points in the first bounding box whose foreground motion flow feature magnitude is less than a first magnitude threshold, and use the filtered feature points in the first bounding box as the feature points in the bounding box.

2. The method according to claim 1, characterized in that, Before obtaining the initial pose and initial target position of the target object in the initial frame, the following steps are also included: Perform target object detection on the current frame to determine whether there is a target object in the current frame; If the target object is present in the current frame, the current frame is used as the image frame to be analyzed.

3. The method according to claim 1, characterized in that, The process of obtaining the initial pose of the target object in the initial frame includes: Obtain the template feature points of the template to be identified and the first descriptor corresponding to the template feature points, wherein the template to be identified is an image containing only the target object; Feature points are extracted from the initial frame to obtain the initial feature points of the initial frame and the second descriptor corresponding to the initial feature points; The second descriptor and the first descriptor are matched to obtain the matching relationship between the initial feature points and the template feature points; The initial pose of the target object in the initial frame is determined based on the matching relationship between the initial feature points and the template feature points.

4. The method according to claim 1, characterized in that, The first feature point of the image frame to be analyzed includes: Feature point extraction is performed on the image frame to be analyzed to obtain all feature points of the image frame to be analyzed; The optical flow method is used to track the image frame to be analyzed, and the position of the previous feature point corresponding to the previous frame of the image frame to be analyzed is obtained. Based on the difference between the previous feature point position and the current feature point position in the image frame to be analyzed, determine the flow features corresponding to all feature points; The image frame to be analyzed is divided into blocks of at least one size to obtain at least one image frame to be analyzed with multiple image blocks; Obtain the number of feature points within all image patches; Filter out image blocks in which the number of feature points is less than a threshold, and obtain the first filtered image block; Based on the flow feature magnitude corresponding to the feature points in the first image block, the feature points in the first image block are filtered to obtain the internal feature points in the first image block. Merge all internal feature points within all image blocks and use the internal feature points as the first feature points. Merging all internal feature points within all image blocks means retaining non-repeating internal feature points in all image blocks and merging and retaining internal feature points that are repeated at the same position in all image blocks.

5. The method according to claim 1, characterized in that, The selected camera motion corresponding transformation model includes: Based on the width and height of the image frame to be analyzed, and the width and height of the flow feature corresponding to the first feature point, the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion are obtained sequentially using the iterative weighted least squares method IRLS, according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, until the acquisition is successful. Based on the transformation parameters of the successfully acquired camera motion, determine the transformation matrix corresponding to the transformation parameters; The transformation matrix is ​​used as the transformation model corresponding to camera motion.

6. The method according to claim 5, characterized in that, The step involves obtaining the parameters of the camera motion homography transformation, affine transformation, linear similarity transformation, and translation transformation sequentially using iterative weighted least squares (IRLS) based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, until successful acquisition. This includes: Obtain the width and height of the image frame to be analyzed; Obtain the flow feature corresponding to the first feature point and the width and height of the flow feature corresponding to the first feature point; The width and height of the image frame to be analyzed are normalized based on the diagonal length, width, and height of the image frame to be analyzed. Based on the diagonal length of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, normalize the width and height of the flow feature corresponding to the first feature point. The normalized width and height of the image frame to be analyzed are used as the width and height of the image frame to be analyzed. The width and height of the normalized flow feature are used as the width and height of the flow feature corresponding to the first feature point.

7. The method according to any one of claims 5 or 6, characterized in that, The step involves obtaining the parameters of the camera motion homography transformation, affine transformation, linear similarity transformation, and translation transformation sequentially using iterative weighted least squares (IRLS) based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, according to the priority order of homography transformation, affine transformation, linear similarity transformation, and translation transformation, until successful acquisition. Based on the width and height of the image frame to be analyzed and the width and height of the flow feature corresponding to the first feature point, the IRLS weight homography of the flow feature corresponding to the first feature point is initialized using a Gaussian distribution; The average moving modulus is determined based on the modulus of the flow feature corresponding to the first feature point. Based on the average moving modulus and the weights of the flow features corresponding to the first feature point, the parameters of the homography transformation, affine transformation, linear similarity transformation, and translation transformation of the camera motion are obtained sequentially through the IRLS algorithm in the order of priority of homography transformation, affine transformation, linear similarity transformation, and translation transformation until the acquisition is successful.

8. The method according to claim 1, characterized in that, Determining the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed includes: Determine the position scaling amount based on the initial target position and the current target position; Based on the position scaling amount, camera parameters, the initial target position, and the current target position, determine the three-dimensional coordinates of the target object in the initial frame and the three-dimensional coordinates of the target object in the image frame to be analyzed; Obtain the first rotation amount of the target object in the initial frame and the second rotation amount of the target object in the image frame to be analyzed; Based on the first rotation amount and the second rotation amount, determine the rotational change of the target object; The translational change of the target object is determined based on the rotational change, the three-dimensional coordinates of the target object in the initial frame, and the three-dimensional coordinates of the target object in the image frame to be analyzed. The pose of the target object in the image frame to be analyzed is determined based on the rotation change, the translation change, and the initial pose of the target object in the initial frame.

9. A pose determination system, characterized in that, include: An initial module is used to obtain the initial pose and initial target position of the target object in an initial frame, wherein the initial frame is the first image frame containing the target object; The feature module is used to obtain the first feature point of the image frame to be analyzed and the flow feature corresponding to the first feature point. The flow feature is the position difference between the feature point position of the previous frame of the image frame to be analyzed and the first feature point position. The feature point position of the previous frame of the image frame to be analyzed is obtained by optical flow tracking. The location module is used to determine the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow features corresponding to the first feature point; The pose module is used to determine the pose of the target object in the image frame to be analyzed based on the initial pose and the current target position of the target object in the image frame to be analyzed. Wherein, the initial target position is the position of the initial target object box containing the target object in the initial frame; Determining the current target position of the target object in the image frame to be analyzed based on the initial target position and the flow features corresponding to the first feature point includes: Based on the position of the initial target object box, obtain the in-frame feature points of the first feature points that are located within the initial target object box; Select the camera motion corresponding transformation model; Obtain the flow features corresponding to the feature points within the frame; The current target position of the target object in the image frame to be analyzed is obtained by using the camera motion corresponding transformation model and the flow features corresponding to the feature points in the frame; The step of obtaining the intra-box feature points located within the initial target object box based on the position of the initial target object box includes: Based on the initial target object frame, determine the straight lines corresponding to the four sides of the initial target object frame; Based on the positional relationship between the first feature point and the corresponding straight lines of the four sides of the initial target object box, the feature points within the first box are determined; Based on the motion-correspondence transformation model and the first feature point, determine the background motion flow features generated by the camera motion of the image frame to be analyzed; Based on the difference between the flow feature corresponding to the first feature point and the background motion flow feature, the foreground motion flow feature corresponding to the foreground motion of the image frame to be analyzed is determined; Filter out feature points in the first bounding box whose foreground motion flow feature magnitude is less than a first magnitude threshold, and use the filtered feature points in the first bounding box as the feature points in the bounding box.

10. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the pose determination method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the pose determination method as described in any one of claims 1 to 8.

Images