Object tracking method, system, electronic device, and storage medium

Abstract

The application relates to an object tracking method, which comprises the following steps: matching an initial pose of a current frame image with a pre-stored tracking resource to obtain a target 3D point and a pre-stored hierarchical feature of the target 3D point, wherein the pre-stored tracking resource comprises a pose of a target object image, a reference 3D point and a pre-stored hierarchical feature of the reference 3D point; and determining an optimized pose of the current frame image according to the initial pose, the target 3D point and the pre-stored hierarchical feature of the target 3D point, so as to solve the problem of complex implementation of the object tracking method. The pre-stored tracking resource contains a pose of a tracked object image, a reference 3D point and a hierarchical feature of the corresponding 3D point. When object tracking is performed, only the 3D point of a reference frame image is obtained from the pre-stored tracking resource through the initial pose, and the initial pose is optimized according to the 3D point of the reference frame image, so that the pose of the current frame image can be obtained, the complexity of the object tracking method is reduced, and the tracking efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of positioning technology, and in particular to object tracking methods, systems, electronic devices, and storage media. Background Technology

[0002] With the rapid development of computer technology, 3D object pose tracking has been widely used in augmented reality and industrial fields. 3D object tracking obtains the pose of each subsequent frame by acquiring the pose of the first frame.

[0003] In related technologies, common object tracking methods include the following:

[0004] 1. Iterative Closest Point (ICP) analysis using point clouds. This method typically requires a depth camera or multiple sensors, placing high demands on hardware.

[0005] 2. Render the object's CAD model, then compare the rendered image with the rendered image to determine the current image's pose. This method requires a CAD model of the object, and the requirements for the CAD model are relatively high; generally, CAD models of objects are difficult to obtain.

[0006] 3. Regressing pose by training a deep learning network specifically for that object. This method is only applicable to specific objects and lacks versatility.

[0007] Currently, no effective solution has been proposed to address the complexity of implementing object tracking methods in related technologies. Summary of the Invention

[0008] This application provides an object tracking method, system, electronic device, and storage medium to at least solve the problem of complex implementation of object tracking methods in related technologies.

[0009] In a first aspect, embodiments of this application provide an object tracking method, the method comprising:

[0010] Obtain the current frame image and the initial pose of the current frame image;

[0011] The initial pose is matched with the pre-stored tracking resources to obtain the target 3D point and the pre-stored hierarchical features of the target 3D point, wherein the pre-stored tracking resources include the pose of the target object image, the reference 3D point and the pre-stored hierarchical features of the reference 3D point;

[0012] The optimized pose of the current frame image is determined based on the initial pose, the target 3D point, and the pre-stored hierarchical features of the target 3D point.

[0013] In some embodiments, determining the optimized pose of the current frame image based on the initial pose, the target 3D point, and the pre-stored hierarchical features of the target 3D point includes:

[0014] Using the initial pose and the pre-constructed feature extraction network, the hierarchical features of the target 3D point in the current frame image are obtained;

[0015] Based on the pre-stored hierarchical features of the target 3D point and the hierarchical features of the target 3D point in the current frame image, the initial pose is optimized to obtain the optimized pose of the current frame image.

[0016] In some embodiments, the hierarchical features include a first hierarchical feature and a second hierarchical feature, wherein the first hierarchical feature is the original image size hierarchical feature, and the size of the second hierarchical feature is smaller than the size of the first hierarchical feature. Based on the pre-stored hierarchical features of the target 3D point and the hierarchical features of the target 3D point in the current frame image, optimizing the initial pose to obtain the optimized pose of the current frame image includes:

[0017] Based on the pre-stored second-level features of the target 3D point and the second-level features of the target 3D point in the current frame image, the initial pose is optimized to obtain the second-level pose;

[0018] Based on the second-level pose, the target 3D point is projected onto the current frame image to obtain the first-level feature of the target 3D point in the current frame image;

[0019] Based on the pre-stored first-level features of the target 3D point and the first-level features of the target 3D point in the current frame image, the second-level pose is optimized to obtain the optimized pose of the current frame image.

[0020] In some embodiments, matching the initial pose with pre-stored tracking resources to obtain the target 3D point and the pre-stored hierarchical features of the target 3D point includes:

[0021] Based on the pose of the target object image in the pre-stored tracking resources, obtain the relative distance between the initial pose and the corresponding poses of all the target object images;

[0022] The image of the target object with the smallest relative distance is determined as the reference frame image;

[0023] Obtain reference 3D points from the reference frame image to obtain the target 3D point and its hierarchical features.

[0024] In some embodiments, before acquiring the current frame image and the initial pose of the current frame image, the method further includes:

[0025] Acquire images of the target object;

[0026] Obtain the pose of the target object image and the reference 3D points of the target object image;

[0027] Based on the pre-constructed feature extraction network and the target object image, the hierarchical features of the reference 3D points are obtained;

[0028] The pre-stored tracking resources are obtained based on the pose of the target object image, the reference 3D point, and the hierarchical features of the reference 3D point.

[0029] In some embodiments, obtaining the pose of the target object image and reference 3D points of the target object image based on the feature extraction network includes:

[0030] Based on the target object image, reconstruct a three-dimensional model of the target object, and obtain the trajectory length of 3D points in the three-dimensional model and the pose of the target object image;

[0031] The 3D points whose trajectory length is greater than a preset threshold are determined as the reference 3D points.

[0032] In some embodiments, obtaining the hierarchical features of the reference 3D points based on a feature extraction network and the target object image includes:

[0033] The feature extraction network is used to obtain 2D points of the target object image and the hierarchical features of the 2D points.

[0034] Match the 2D points of the target object image with the reference 3D points to obtain the 2D points corresponding to the reference 3D points in each target object image;

[0035] The hierarchical features of the target 3D points are obtained by weighting and averaging the corresponding 2D points of the reference 3D points in each target object image and applying the L2 norm.

[0036] In some embodiments, obtaining the hierarchical features of the target 3D point in the current frame image through the initial pose and a pre-built feature extraction network includes:

[0037] The feature extraction network is used to extract features from the current frame image to obtain the hierarchical features of the current frame image;

[0038] Based on the initial pose, the target 3D point is mapped onto the current frame image to obtain the projection point;

[0039] Based on the hierarchical features of the current frame image, the hierarchical features of the projection point are obtained, and the hierarchical features of the target 3D point in the current frame image are obtained.

[0040] Secondly, embodiments of this application provide an object tracking system, the system comprising an information acquisition module, a matching module, and an optimization module:

[0041] The information acquisition module is used to acquire the current frame image and the initial pose of the current frame image;

[0042] The matching module is used to match the initial pose with the pre-stored tracking resources to obtain the target 3D point and the pre-stored hierarchical features of the target 3D point, wherein the pre-stored tracking resources include the pose of the target object image, the reference 3D point and the pre-stored hierarchical features of the reference 3D point;

[0043] The optimization module is used to determine the optimized pose of the current frame image based on the initial pose, the target 3D point, and the pre-stored hierarchical features of the target 3D point.

[0044] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the object tracking method as described in the first aspect above.

[0045] Compared to related technologies, the object tracking method provided in this application matches the initial pose of the current frame image with pre-stored tracking resources to obtain the target 3D points and pre-stored hierarchical features of the target 3D points. The pre-stored tracking resources include the pose of the target object image, reference 3D points, and pre-stored hierarchical features of the reference 3D points. Based on the initial pose, target 3D points, and pre-stored hierarchical features of the target 3D points, the optimized pose of the current frame image is determined, solving the problem of complex implementation in object tracking methods. The pre-stored tracking resources contain the pose of the tracked object image, reference 3D points, and the corresponding hierarchical features of the 3D points. During object tracking, it is only necessary to obtain the 3D points of the reference frame image from the pre-stored tracking resources using the initial pose, and optimize the initial pose based on the 3D points of the reference frame image to obtain the pose of the current frame image, reducing the complexity of the object tracking method and improving tracking efficiency. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a schematic diagram of the application environment of the object tracking method according to the embodiments of this application;

[0048] Figure 2This is a flowchart of an object tracking method according to an embodiment of this application;

[0049] Figure 3 This is a flowchart of a method for creating pre-stored resources according to an embodiment of this application;

[0050] Figure 4 This is a flowchart of a feature extraction method according to an embodiment of this application;

[0051] Figure 5 This is a flowchart of an object tracking method according to an embodiment of this application;

[0052] Figure 6 This is a structural block diagram of an object tracking system according to an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0055] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0056] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0057] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0058] The object tracking method provided in this application can be applied to, for example... Figure 1 In the application environment shown, Figure 1 This is a schematic diagram illustrating the application environment of the object tracking method according to an embodiment of this application, such as... Figure 1 As shown in the diagram, server 10 and terminal 11 communicate via a network. Terminal 11 acquires the current frame image and sends it to server 10. Server 10 obtains the initial pose of the current frame image based on the pose of the previous frame image, and matches the initial pose with pre-stored tracking resources to obtain a reference frame image and the target 3D points of the reference frame image. Server 10 then determines the optimized pose of the current frame image based on the initial pose and the target 3D points of the reference frame image.

[0059] This embodiment provides an object tracking method. Figure 2 This is a flowchart of an object tracking method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0060] Step S201: Obtain the current frame image and its initial pose. In this embodiment, the initial pose of the current frame image can be the pose of the previous frame image, or it can be an estimated pose obtained by transforming the poses of the previous few frames.

[0061] Step S202 involves matching the initial pose with pre-stored tracking resources to obtain the target 3D point and its pre-stored hierarchical features. The pre-stored tracking resources include the pose of the target object image, reference 3D points, and their pre-stored hierarchical features. In this embodiment, the relative distance between the initial pose and the corresponding poses of all images in the pre-stored tracking resources is calculated. The image with the closest relative distance to the initial pose is selected as the reference image, ensuring that the pose of the obtained reference image is closest to the true pose of the current frame image, thus improving the accuracy of the object tracking method.

[0062] Step S203: Determine the optimized pose of the current frame image based on the initial pose, the target 3D point, and the pre-stored hierarchical features of the target 3D point. In this embodiment, the pose of the current frame image is optimized based on the hierarchical features of the target 3D point in the current frame image and the pre-stored hierarchical features. Specifically, a pose is found such that the difference between the hierarchical features of the same target 3D point in the current frame image and the pre-stored hierarchical features is minimized under this pose, and this pose is taken as the optimized pose of the current frame.

[0063] Through steps S201 to S203, the initial pose of the current frame image is matched with the pre-stored tracking resources to obtain the target 3D points and their pre-stored hierarchical features. The pre-stored tracking resources include the pose of the target object image, reference 3D points, and their pre-stored hierarchical features. Based on the initial pose, target 3D points, and their pre-stored hierarchical features, the optimized pose of the current frame image is determined, solving the problem of complex object tracking implementation. The pre-stored tracking resources contain the pose of the tracked object image, reference 3D points, and their corresponding hierarchical features. During object tracking, it is only necessary to obtain the 3D points of the reference frame image from the pre-stored tracking resources using the initial pose, and optimize the initial pose based on the reference frame image's 3D points to obtain the pose of the current frame image, reducing the complexity of the object tracking method and improving tracking efficiency.

[0064] In some embodiments, determining the optimized pose of the current frame image based on the initial pose, the target 3D point, and the pre-stored hierarchical features of the target 3D point includes:

[0065] By using the initial pose and a pre-built feature extraction network, the hierarchical features of the target 3D point in the current frame image are obtained. Based on the pre-stored hierarchical features of the target 3D point and the hierarchical features of the target 3D point in the current frame image, the initial pose is optimized to obtain the optimized pose of the current frame image.

[0066] This embodiment projects the target 3D point onto the current frame image to obtain the corresponding projected points. A feature extraction network then extracts the hierarchical features of these projected points to obtain the hierarchical features of the target 3D point in the current frame image. The feature distance between the hierarchical features of the target 3D point in the current frame image and its pre-stored hierarchical features is calculated. The ultimate goal is to find a pose that minimizes the sum of the feature distances between the hierarchical features of the target 3D point in the current frame image and its pre-stored hierarchical features; this pose is the optimized pose of the current frame image.

[0067] It should be noted that the target 3D point is the 3D point corresponding to the target object in the reference frame image, where the target object is the object that the user needs to track. In this embodiment, the features of the target 3D point in the current frame image are obtained through pre-stored tracking resources and a feature extraction network. Then, the pose of the current frame image is determined based on the difference between the pre-stored features of the same feature point and the features in the current frame image. The pre-stored tracking resources are also obtained based on the feature extraction network. The feature extraction network can automatically learn the features of the data, avoiding the tedious process of manually extracting features. Moreover, the feature extraction network uses convolution operations, which can utilize the local connectivity and parameter sharing properties of convolution, greatly reducing the number of parameters and computational load in the network, and improving the object tracking efficiency.

[0068] In some embodiments, the hierarchical features include first-level features and second-level features, wherein the first level is the original image size level, and the size of the second level is smaller than the size of the first level. Based on the pre-stored hierarchical features of the target 3D point and the hierarchical features of the target 3D point in the current frame image, the initial pose is optimized to obtain the optimized pose of the current frame image, including:

[0069] Based on the pre-stored second-level features of the target 3D point and the second-level features of the target 3D point in the current frame image, the initial pose is optimized to obtain the second-level pose; based on the second-level pose, the target 3D point is projected onto the current frame image to obtain the first-level features of the target 3D point in the current frame image; based on the pre-stored first-level features of the target 3D point and the first-level features of the target 3D point in the current frame image, the second-level pose is optimized to obtain the optimized pose of the current frame image.

[0070] It should be noted that both the first and second levels are obtained by downsampling and compressing the original image, followed by upsampling to restore it, but the restored sizes are different. The hierarchical features can also include third-level features, which are smaller than the second-level features. In this embodiment, features are first extracted from the current frame image to obtain its corresponding feature map. During the optimization process, after each pose update, the target 3D points are reprojected, and the corresponding features are obtained from the reprojected points on the feature map. The next optimization step is then performed based on these features. When optimizing the pose, optimization starts from the level with the smallest size. The optimization result of each level is used as the initial value for the next level, and finally, the original image size is optimized. Each pyramid level can be iterated N times, or conditions can be set to terminate the optimization early.

[0071] For example, a feature extraction network extracts three levels of features, which are 1 / 16, 1 / 4, and 1 / 1 of the original image size, to construct a feature pyramid. Before pose optimization, the three levels of feature maps of the current frame image are obtained. During pose optimization, optimization starts from the feature level with the lowest resolution, which is the 1 / 16 feature level, and each level is iterated and optimized N times.

[0072] Based on the initial pose, the target 3D point is projected onto the current frame image to obtain the corresponding 2D point. Features of these 2D points are extracted from the 1 / 16 level feature map. Based on these 2D point features and the pre-stored 1 / 16 level features of the target 3D point, the first optimization at the 1 / 16 level is performed to obtain a new pose. Further, based on the new pose, the corresponding 2D point on the current image is redefined, resulting in new 2D point features. Based on these new 2D features and the pre-stored features of the target 3D point, further optimization is performed, updating the pose. After N optimization iterations, the process moves to the 1 / 4 level optimization. The target 3D point is projected from the pose of the previous level to obtain 1 / 4 level features. Based on these 1 / 4 level features and the pre-stored 1 / 4 level features of the target 3D point, the pose obtained from the previous level is optimized, also iterating N times. Finally, the original image size is optimized, similar to the 1 / 16 level optimization process, and will not be elaborated here.

[0073] In this embodiment, extracting multi-scale features through feature pyramids can improve the scale adaptability, accuracy, and robustness of pose estimation.

[0074] Different targets or scenes may exhibit varying scales, and feature extraction at a single scale may not effectively capture these variations. By extracting features at multiple scales through feature pyramids, we can adapt to changes in targets at different scales and provide a more comprehensive feature representation.

[0075] When images are blurred, have low resolution, or are occluded, features at a single scale may be affected, leading to inaccurate pose estimation. By using multi-scale features, feature information from other scales can be used to compensate for the deficiencies of a single scale, thereby improving the robustness of the algorithm.

[0076] Due to the scale variation of the target, features at a single scale may not accurately match the corresponding points on the target. By extracting features at multiple scales, matching points can be searched at different scales, increasing the accuracy and reliability of the matching.

[0077] It's important to note that while feature pyramids can extract features at different scales, this doesn't mean feature extraction needs to be performed on the entire image. Typically, higher-level pyramid features can be computed from lower-level features through downsampling or pooling operations, thus reducing computation and improving algorithm efficiency.

[0078] In some embodiments, matching the initial pose with pre-stored tracking resources to obtain the target 3D point and the pre-stored hierarchical features of the target 3D point includes:

[0079] Based on the pose of the target object image in the pre-stored tracking resources, the relative distance between the initial pose and the corresponding poses of all target object images is obtained. The target object image with the smallest relative distance is determined as the reference frame image. The reference 3D points of the reference frame image are obtained, and the target 3D points and the hierarchical features of the target 3D points are obtained.

[0080] In this embodiment, the target 3D point is the reference 3D point of the reference frame image. This embodiment improves the accuracy of the final current frame image pose by selecting the image with the closest relative distance to the initial pose from the pre-stored tracking resources as the reference frame image.

[0081] In some embodiments, the process further includes obtaining the current frame image and its initial pose before:

[0082] The process involves acquiring images of the target object, obtaining its pose and reference 3D points, and then, based on a pre-constructed feature extraction network and the target object image, obtaining the hierarchical features of the reference 3D points. Pre-stored tracking resources are then derived based on the target object image's pose, the reference 3D points, and their hierarchical features. Specifically, this includes: reconstructing a 3D model of the target object from its image; obtaining the trajectory lengths of the 3D points in the 3D model and the target object image's pose; identifying 3D points with trajectory lengths greater than a preset threshold as reference 3D points; obtaining 2D points and their hierarchical features from the target object image through a feature extraction network; matching the 2D points of the target object image with the reference 3D points to obtain the corresponding 2D points in each target object image; and performing a weighted average and L2 norm on the corresponding 2D points of the reference 3D points in each target object image to obtain the hierarchical features of the target 3D points.

[0083] In this embodiment, before tracking, pre-stored tracking resources need to be created. The specific method includes: acquiring images at 360-degree intervals around the object, then performing 3D reconstruction using an SFM-like algorithm; optionally, the 3D reconstruction process uses Colmap. After obtaining the 3D model of the object, the model is trimmed to obtain the 3D points of the target object. The `model_cropper` command of Colmap or other similar tools can be used to retain only the 3D points of the object, removing background points and noise; alternatively, interactive segmentation (SAM) can be used to segment each image, then only the 2D points within the segmentation mask are retained, thus achieving the purpose of trimming the model. After model trimming, the creation of tracking resources begins.

[0084] Figure 3 This is a flowchart of a method for creating pre-stored resources according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps:

[0085] S31, Acquire image of target object;

[0086] S32, Reconstruct the three-dimensional model of the target object based on the target object image, and obtain the pose of each target object image;

[0087] S33, trim the 3D model of the target object to obtain the target 3D points and the corresponding hierarchical features of the target 3D points;

[0088] S34. Based on the pose of the target object image, the target 3D points, and the hierarchical features of the target 3D points, pre-stored tracking resources are obtained.

[0089] It should be noted that the tracking resources store the pose of each image, the corresponding reference 3D point, and the features of the reference 3D point. The image pose is obtained through SFM (Simplified Function Model), and the reference 3D point is the object's 3D point retained after model trimming, satisfying the condition that the trajectory length of the SFM result is greater than a certain threshold. The features stored for each point are the features of three pyramid levels of the 3D point. Each layer of features is a weighted average of the features of the corresponding 2D points in all images, then subjected to L2 norm. The features of each 2D point are obtained by running a general feature extraction network on each image. In this embodiment, the features of 3D points in each image are stored, rather than the features of 2D points, reducing the storage space occupied by the pre-stored tracking resources.

[0090] In some embodiments, obtaining the hierarchical features of the target 3D point in the current frame image through an initial pose and a pre-built feature extraction network includes:

[0091] The feature extraction network extracts features from the current frame image to obtain the hierarchical features of the current frame image. Based on the initial pose, the target 3D point is mapped to the current frame image to obtain the projection point. Based on the hierarchical features of the current frame image, the hierarchical features of the projection point are obtained to obtain the hierarchical features of the target 3D point in the current frame image.

[0092] In this embodiment, a general feature extraction network needs to be trained before object tracking. The network input is an RGB image, which is downsampled and compressed to 32 or 64 pixels, and then upsampled back to the required size. Image compression can use VGG or ResNet structures, and restoration can use multi-scale fusion strategies such as FPN. The final training objective is to find two images where the features corresponding to the same physical point are similar, and the feature distances between different points are large. Optionally, a superpoint training method can be used, where a reference image is obtained by transforming one image using a known transformation. The correspondence between feature points in the two images is determined by the known transformation, and then distance constraints are applied to the output features of the two images, ultimately outputting features at multiple levels.

[0093] Figure 4 This is a flowchart of a feature extraction method according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps:

[0094] S41, Input the current frame image;

[0095] S42, compress the current frame image based on downsampling technology to obtain an image of 1 / 32 of the original image size, and obtain the hierarchical features of 1 / 32 size;

[0096] S43, by downsampling and size fusion of the original image, layer features of 1 / 16, 1 / 4 and 1 / 1 sizes are obtained respectively based on the layer features of 1 / 32 size.

[0097] It's important to note that traditional machine learning requires manual feature extraction, while feature extraction networks can automatically learn the features of data, avoiding the tedious process of manual feature extraction. Feature extraction networks use Convolutional Neural Networks (CNNs), which can handle input images of any size, thus allowing them to adapt to inputs of different scales. Feature extraction networks can be shared across different tasks and can be fine-tuned to adapt to new tasks, making them highly transferable and usable in various application scenarios. Furthermore, because feature extraction networks use convolutional operations, they can leverage the local connectivity and parameter sharing properties of convolution to reduce the number of parameters and computational cost, thereby improving network efficiency.

[0098] This embodiment utilizes a feature extraction network to extract image features for object tracking, which not only improves tracking efficiency but also adapts to object tracking in various scenarios.

[0099] Figure 5 This is a flowchart of an object tracking method according to an embodiment of this application, such as... Figure 5 As shown, the process includes the following steps:

[0100] S51, take the pose of frame t0 as the initial pose of frame t1;

[0101] S52, Based on the initial pose, obtain the reference frame image and its corresponding target 3D point features;

[0102] S53, extract the features of the t1 frame image through a feature extraction network;

[0103] S54, based on the feature pyramid algorithm, optimizes the pose of the t1-th frame image according to the target 3D point features of the reference frame image and the physical point features of the t1-th frame image, and obtains the optimized pose of the t1-th frame image.

[0104] S55, take the optimized pose of the t1 frame image as the initial pose of the t2 frame image, and optimize the pose of the t2 frame image. The specific pose optimization method is the same as the optimization method of t1, and will not be repeated here.

[0105] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0106] This embodiment also provides an object tracking system. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0107] Figure 6 This is a structural block diagram of an object tracking system according to an embodiment of this application, such as... Figure 6 As shown, the system includes: an information acquisition module 61, a matching module 62, and an optimization module 63.

[0108] The information acquisition module 61 is used to acquire the current frame image and the initial pose of the current frame image;

[0109] The matching module 62 is used to match the initial pose with the pre-stored tracking resources to obtain the reference frame image and the target 3D points of the reference frame image. The pre-stored tracking resources include the pose of the target image, the target 3D points, and the hierarchical features of the target 3D points.

[0110] The optimization module 63 is used to determine the optimized pose of the current frame image based on the initial pose and the target 3D points of the reference frame image.

[0111] This embodiment uses an information acquisition module 61 to acquire the current frame image and its initial pose. A matching module 62 matches the initial pose with pre-stored tracking resources to obtain a reference frame image and the target 3D points within that reference frame image. An optimization module 63 determines the optimized pose of the current frame image based on the initial pose and the target 3D points in the reference frame image. This solves the problem of complex object tracking methods. During object tracking, it is only necessary to acquire the reference frame image from the pre-stored tracking resources using the initial pose and determine the feature distances of the target 3D points in the reference frame image and the current frame image to obtain the pose of the current frame image. This reduces the complexity of the object tracking method and improves tracking efficiency.

[0112] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0113] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0114] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0115] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0116] S1, obtain the current frame image and the initial pose of the current frame image.

[0117] S2, Match the initial pose with the pre-stored tracking resources to obtain the target 3D point and the pre-stored hierarchical features of the target 3D point. The pre-stored tracking resources include the pose of the target object image, the reference 3D point and the pre-stored hierarchical features of the reference 3D point.

[0118] S3. Determine the optimized pose of the current frame image based on the initial pose, the target 3D point, and the pre-stored hierarchical features of the target 3D point.

[0119] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0120] In one embodiment, Figure 7 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 7 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 7 As shown, this electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an object tracking method.

[0121] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0123] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An object tracking method, characterized by, The method comprises: acquiring a current frame image and an initial pose of the current frame image; matching the initial pose with a pre-stored tracking resource to obtain a target 3D point and pre-stored hierarchical features of the target 3D point, wherein the pre-stored tracking resource comprises a pose of a target object image, a reference 3D point and pre-stored hierarchical features of the reference 3D point; determining an optimized pose of the current frame image according to the initial pose, the target 3D point and the pre-stored hierarchical features of the target 3D point, comprising: acquiring hierarchical features of the target 3D point in the current frame image through the initial pose and a pre-constructed feature extraction network; optimizing the initial pose according to the pre-stored hierarchical features of the target 3D point and the hierarchical features of the target 3D point in the current frame image to obtain the optimized pose of the current frame image; the hierarchical features comprise first hierarchical features and second hierarchical features, wherein the first hierarchical level is an original image size level, the size of the second hierarchical level is smaller than the size of the first hierarchical level, and the optimized pose of the current frame image is obtained by optimizing the initial pose according to the pre-stored hierarchical features of the target 3D point and the hierarchical features of the target 3D point in the current frame image, comprising: optimizing the initial pose according to the pre-stored second hierarchical features of the target 3D point and the second hierarchical features of the target 3D point in the current frame image to obtain a second hierarchical pose; projecting the target 3D point to the current frame image according to the second hierarchical pose to obtain the first hierarchical features of the target 3D point in the current frame image; optimizing the second hierarchical pose according to the pre-stored first hierarchical features of the target 3D point and the first hierarchical features of the target 3D point in the current frame image to obtain the optimized pose of the current frame image.

2. The method of claim 1, wherein, matching the initial pose with a pre-stored tracking resource to obtain a target 3D point and pre-stored hierarchical features of the target 3D point comprises: acquiring relative distances between the initial pose and all corresponding poses of the target object images according to the poses of the target object images in the pre-stored tracking resource; determining a target object image with the smallest relative distance as a reference frame image; acquiring a reference 3D point of the reference frame image to obtain the target 3D point and the hierarchical features of the target 3D point.

3. The method of claim 1, wherein, Before the acquiring of the current frame image and the initial pose of the current frame image, the method further comprises: collecting a target object image; acquiring a pose of the target object image and a reference 3D point of the target object image; obtaining hierarchical features of the reference 3D point based on a pre-constructed feature extraction network and the target object image; obtaining the pre-stored tracking resource according to the pose of the target object image, the reference 3D point and the hierarchical features of the reference 3D point.

4. The method of claim 3, wherein, acquiring the pose of the target object image and the reference 3D point of the target object image comprises: reconstructing a three-dimensional model of a target object according to the target object image to acquire a trajectory length of a 3D point in the three-dimensional model and the pose of the target object image; The 3D point with a track length greater than a preset threshold is determined as the reference 3D point.

5. The method of claim 3, wherein, The hierarchical feature of the reference 3D point is obtained based on a feature extraction network and the target object image, and the hierarchical feature of the target 3D point in the current frame image is obtained through the initial pose and the pre-constructed feature extraction network. The 2D point of the target object image is matched with the reference 3D point to obtain a corresponding reference 2D point of the reference 3D point in each target object image. The hierarchical features of the reference 2D points are weighted and averaged and subjected to L2 norm processing to obtain the hierarchical feature of the target 3D point. The hierarchical feature of the target 3D point in the current frame image is obtained through the initial pose and the pre-constructed feature extraction network, and the hierarchical feature of the target 3D point in the current frame image is obtained through the feature extraction network.

6. The method of claim 1, wherein, The hierarchical feature of the target 3D point in the current frame image is obtained through the initial pose and the pre-constructed feature extraction network, and the hierarchical feature of the target 3D point in the current frame image is obtained through the feature extraction network. The system comprises an information acquisition module, a matching module and an optimization module. The information acquisition module is configured to acquire a current frame image and an initial pose of the current frame image. The matching module is configured to match the initial pose with a pre-stored tracking resource to obtain a target 3D point and a pre-stored hierarchical feature of the target 3D point, wherein the pre-stored tracking resource comprises a pose of a target object image, a reference 3D point and a pre-stored hierarchical feature of the reference 3D point.

7. An object tracking system, characterized by The optimization module is configured to determine an optimized pose of the current frame image according to the initial pose, the target 3D point and the pre-stored hierarchical feature of the target 3D point, comprising: The hierarchical feature of the target 3D point in the current frame image is obtained through the initial pose and the pre-constructed feature extraction network. The initial pose is optimized according to the pre-stored hierarchical feature of the target 3D point and the hierarchical feature of the target 3D point in the current frame image to obtain the optimized pose of the current frame image. The hierarchical feature comprises a first hierarchical feature and a second hierarchical feature, wherein the first hierarchical feature is a picture size hierarchical feature, the size of the second hierarchical feature is smaller than that of the first hierarchical feature, and the initial pose is optimized according to the pre-stored hierarchical feature of the target 3D point and the hierarchical feature of the target 3D point in the current frame image to obtain the optimized pose of the current frame image, comprising: The initial pose is optimized according to the pre-stored second hierarchical feature of the target 3D point and the second hierarchical feature of the target 3D point in the current frame image to obtain a second hierarchical pose. The target 3D point is projected into the current frame image according to the second hierarchical pose to obtain the first hierarchical feature of the target 3D point in the current frame image. The second hierarchical pose is optimized according to the pre-stored first hierarchical feature of the target 3D point and the first hierarchical feature of the target 3D point in the current frame image to obtain the optimized pose of the current frame image. ​ ​ ​ 8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the object tracking method as claimed in any one of claims 1 to 6 when executing the computer program.

Images