Pose tracking method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a pose tracking method and device, electronic equipment and storage medium, wherein the method comprises: obtaining initial pose information of a target object in an initial pose image; adjusting the pose of a virtual model of the target object according to the initial pose information, and determining initial edge contour information of the adjusted virtual model; based on the initial edge contour information and at least one pose tracking image containing the target object, determining the tracking pose information of the target object corresponding to the at least one pose tracking image respectively.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a pose tracking method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of image recognition technology, pose tracking technology has gradually matured and has been widely applied. Pose tracking technology can estimate the position and pose of a target object in a continuous video image sequence. Typically, initialization is required before pose tracking. This involves the user adjusting the pose of the target object to be tracked, or using a camera to capture the target object so that the edge contour of the target object in the captured image matches a preset edge contour. Then, the preset edge contour is used to track the pose of the target object. Due to these limitations, the application scenarios for pose tracking are limited, and its application is relatively difficult. Summary of the Invention

[0003] This disclosure provides at least one pose tracking method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of this disclosure provide a pose tracking method, including:

[0005] Obtain the initial pose information of the target object in the initial pose image;

[0006] Based on the initial pose information, the pose of the virtual model of the target object is adjusted, and the initial edge contour information of the adjusted virtual model is determined.

[0007] Based on the initial edge contour information and at least one pose tracking image containing the target object, the tracking pose information of the target object corresponding to the at least one pose tracking image is determined.

[0008] In this regard, by acquiring the initial pose information of the target object in the initial pose image, and adjusting the pose of the virtual model of the target object according to the initial pose information, pose tracking is performed using the initial edge contour information of the adjusted virtual model. This eliminates the need for the user to manually align the target object with the preset contour, thereby reducing the operational difficulty of pose tracking and the conditions for its use, making pose tracking applicable to more scenarios.

[0009] In one possible implementation, the initial pose information of the target object in the initial pose image is obtained as follows:

[0010] Obtain an initial image containing the target object;

[0011] Identify the region of interest corresponding to the target object from the initial image;

[0012] The region of interest is extracted from the initial image, and the extracted region of interest is used as the initial pose image;

[0013] The initial pose information is obtained by estimating the pose information of the target object based on the region of interest.

[0014] This implementation method can reduce the size of the initial pose image by identifying the region of interest corresponding to the target object and then using the region of interest to estimate the initial pose information, thereby reducing the amount of computation required to determine the initial pose information.

[0015] In one possible implementation, determining the tracking pose information of the target object corresponding to each of the at least one pose tracking image based on the initial edge contour information and at least one pose tracking image containing the target object includes:

[0016] The initial edge contour information is used as the edge contour information of the previous pose tracking image in the first pose tracking image among the at least one pose tracking images;

[0017] For the current pose tracking image in the at least one pose tracking image, determine the current edge contour information that matches the previous edge contour information from the current pose tracking image; wherein, the previous edge contour information is the edge contour information corresponding to the previous pose tracking image of the current pose tracking image.

[0018] Determine the pose information corresponding to the current edge contour information, and use the determined pose information as the tracking pose information of the current pose tracking image.

[0019] In this implementation, the current edge contour information of the first pose tracking image among multiple pose tracking images is determined using the initial edge contour information determined from the initial pose image. For other pose tracking images, the current edge contour information is determined using the previous edge contour information, and the tracking pose information is determined using the current edge contour information. Pose tracking can be completed without the user manually aligning the target object and the preset edge contour.

[0020] In one possible implementation, determining the current edge contour information from the current pose tracking image that matches the previous edge contour information of the previous pose tracking image includes:

[0021] Using the previous edge contour information as initial parameter information, a variable parameter curve matching the initial parameter information is established in the current pose tracking image;

[0022] Based on the energy function corresponding to the variable parameter curve, the target parameter information of the variable parameter curve is determined; wherein, the energy function is used to characterize the probability value of each pixel point in the variable parameter curve being a contour pixel point;

[0023] The target parameter information is used as the current edge contour information that matches the previous edge contour information.

[0024] In this implementation, since the previous pose tracking image and the current pose tracking image are two adjacent frames, the previous edge contour information is quite similar to the current edge contour information. Determining the current edge contour information based on the previous edge contour information can improve the accuracy of the current edge contour information and reduce the amount of computation required to determine the current edge contour information to a certain extent.

[0025] In one possible implementation, determining the target parameter information of the variable parameter curve based on the energy function corresponding to the variable parameter curve includes:

[0026] Based on the energy function corresponding to the variable parameter curve, the variable parameter curve is transformed by minimizing energy to obtain the target parameter information corresponding to the variable parameter curve when the energy function is minimized.

[0027] This implementation method performs energy-minimizing deformation on the variable parameter curve to obtain target parameter information, which can improve the accuracy of the current edge contour information and reduce the computational load for determining the current edge contour information to a certain extent.

[0028] In one possible implementation, after establishing the variable parameter curve matching the initial parameter information, the method further includes:

[0029] Based on the energy function, determine the energy value corresponding to the variable parameter curve under the initial parameter information;

[0030] Based on the energy value and the preset energy threshold, determine whether the pose tracking for the target object has failed;

[0031] If pose tracking of the target object fails, update the initial pose image of the target object and proceed to the step of obtaining the initial pose information of the target object in the initial pose image.

[0032] This implementation determines the energy value of the variable parameter curve under the initial parameter information, and judges whether the pose tracking has failed based on the determined energy value and the preset energy threshold. If it fails, the initial pose image is updated, the initial pose information of the initial pose image is re-determined, and pose tracking is performed according to the re-determined initial pose information. When the tracking fails, the user does not need to manually initialize, which helps to improve the smoothness of pose tracking operation and reduce the number of user operation steps.

[0033] In one possible implementation, the initial pose image of the target object is updated:

[0034] The pose tracking image corresponding to the variable parameter curve is used as the initial image containing the target object, and the region of interest corresponding to the target object is identified from the initial image;

[0035] The region of interest is extracted from the initial image, and the region of interest is used as the initial pose image. Then, the process jumps to the step of determining the initial pose information of the target object in the initial pose image.

[0036] This implementation method, by using the pose tracking image corresponding to the failure to determine the initial pose image, and then using the determined initial pose image to re-determine the pose information and perform pose tracking, can achieve seamless connection of pose re-tracking and improve pose tracking efficiency.

[0037] In one possible implementation, the method further includes:

[0038] Based on the tracking pose information of the target object corresponding to the at least one pose tracking image, add virtual effects corresponding to the tracking pose information to the at least one pose tracking image;

[0039] The pose tracking image is displayed after adding virtual effects.

[0040] This implementation adds virtual effects to at least one pose tracking image using the target object's tracking pose information and displays the pose tracking image after adding virtual effects. This can provide users with information related to the tracking pose information, making it easier to understand and use the tracking pose information.

[0041] Secondly, embodiments of this disclosure also provide a pose tracking device, comprising:

[0042] The acquisition module is used to acquire the initial pose information of the target object in the initial pose image;

[0043] An initialization module is used to adjust the pose of the virtual model of the target object according to the initial pose information, and to determine the initial edge contour information of the adjusted virtual model.

[0044] The tracking module is used to determine the tracking pose information of the target object corresponding to the at least one pose tracking image based on the initial edge contour information and at least one pose tracking image containing the target object.

[0045] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0046] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation of the first aspect.

[0047] For a description of the effects of the aforementioned pose tracking device, electronic device, and computer-readable storage medium, please refer to the description of the pose tracking method above; it will not be repeated here.

[0048] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0050] Figure 1 A flowchart of a pose tracking method provided by an embodiment of this disclosure is shown;

[0051] Figure 2 A flowchart of another pose tracking method provided by an embodiment of this disclosure is shown;

[0052] Figure 3 A schematic diagram of a pose tracking device provided in an embodiment of this disclosure is shown;

[0053] Figure 4A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0055] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0056] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0057] Research has found that before performing pose tracking, users need to adjust the pose of the target object to be tracked, or use a camera to photograph the target object so that the edge contour of the target object in the captured image is consistent with a preset edge contour. Then, the preset edge contour is used to perform pose tracking on the target object, which limits the application scenarios of pose tracking and makes it more difficult to use.

[0058] Based on the above research, this disclosure provides a pose tracking method, apparatus, electronic device, and storage medium. The embodiments of this disclosure obtain the initial pose information of the target object in the initial pose image, and adjust the pose of the virtual model of the target object according to the initial pose information. The initial edge contour information of the adjusted virtual model is used for pose tracking. The user does not need to manually align the target object with the preset contour, which can reduce the difficulty of pose tracking operation and reduce the conditions for using pose tracking, thereby making pose tracking applicable to more scenarios.

[0059] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0060] The pose tracking method, apparatus, electronic device, and computer-readable storage medium disclosed herein will be described below through specific embodiments.

[0061] like Figure 1 As shown in the figure, this disclosure provides a pose tracking method that can be applied to electronic devices with computing capabilities, such as terminal devices and servers. Specifically, the pose tracking method may include the following steps:

[0062] S110. Obtain the initial pose information of the target object in the initial pose image.

[0063] The target object mentioned above can be a real-world object. Pose tracking can estimate the position and orientation information of the target object in a continuous video sequence. The position information can be the coordinates of the target object in a three-dimensional coordinate system, and the orientation information can be the three-dimensional orientation of the target object in a three-dimensional coordinate system.

[0064] The initial pose image described above can be obtained using a device with photographic capabilities, such as a camera, smartphone, or augmented reality (AR) device. This initial pose image can be an RGB image or an RGBD depth image. In an RGB image, each pixel has a corresponding RGB value, which is the value of that pixel in the R channel (representing red), G channel (representing green), and B channel (representing blue). In an RGBD depth image, there is an RGB image and its corresponding depth map. The depth map is an image or image channel containing information related to the distance to the surface of scene objects at the viewpoint. In the depth map, the pixel value corresponding to each pixel is the actual distance between the sensor and the target object.

[0065] In some possible embodiments, the initial pose information of the target object in the initial pose image can be obtained through the following steps:

[0066] An initial image containing the target object is obtained; a region of interest corresponding to the target object is identified from the initial image; the region of interest is cropped from the initial image, and the cropped region of interest is used as the initial pose image; the pose information of the target object is estimated based on the region of interest to obtain the initial pose information.

[0067] The initial image can be acquired by a device with photographic capabilities. It may contain the target object and other background objects. Two-dimensional target detection can be performed on the initial image based on the feature information of the target object. The area where the detected target object is located is taken as the region of interest. Then, the RGB image or RGBD image in the region of interest is extracted, and the extracted region of interest is used as the initial pose image.

[0068] After obtaining the initial pose image, the pose information of the target object in the initial pose image can be estimated by the pose recognition model, and then the initial pose information of the target object can be determined based on the estimated pose information.

[0069] In this way, by identifying the region of interest corresponding to the target object and using the region of interest as the initial pose image, the size of the initial pose image can be reduced, thereby reducing the amount of computation required to determine the initial pose information.

[0070] S120. Based on the initial pose information, adjust the pose of the virtual model of the target object, and determine the initial edge contour information of the adjusted virtual model.

[0071] The terminal device can be pre-configured with a virtual model of the target object. The virtual model can be a three-dimensional mesh model. Based on the initial pose information, the pose of the virtual model of the target object can be adjusted so that the pose of the virtual model is consistent with the pose of the target object. Then, the initial edge contour information of the virtual model after the pose adjustment is determined.

[0072] Specifically, after adjusting the pose of the virtual model, the 3D virtual model can be rendered into a 2D edge contour, and the initial edge contour information corresponding to the rendered edge contour can be determined. The initial edge contour information can be represented in the form of an image, where the pixel value of each pixel indicates whether that pixel is part of the edge contour. The size of the image can be the same as the acquired initial image.

[0073] In this way, after obtaining the initial edge contour, the pose of the target object can be tracked based on the initial edge contour without the user having to manually align the target object with the preset contour line. This reduces the difficulty of pose tracking operations and the conditions for using pose tracking, thus making pose tracking applicable to more scenarios.

[0074] S130. Based on the initial edge contour information and at least one pose tracking image containing the target object, determine the tracking pose information of the target object corresponding to each of the at least one pose tracking image.

[0075] The aforementioned pose tracking images and the aforementioned initial images can form a continuous video image sequence, with a typically short time interval between each two frames. The pose tracking images may contain the aforementioned target object. During the capture of the pose tracking images, the target object may be in a static or moving state. If the target object is in motion, its edge contour in the pose tracking image will usually change. The initial pose information can be used to determine the tracking pose information of the target object in the first pose tracking image, and then the determined tracking pose information can be used to determine the tracking pose information of the target object in the next frame of the pose tracking image.

[0076] In some possible embodiments, step S130 may include:

[0077] (1) The initial edge contour information is used as the edge contour information of the previous pose tracking image of the first pose tracking image in the at least one pose tracking image.

[0078] In this step, the initial pose image and the first pose tracking image can be continuous. The initial edge contour information corresponding to the initial pose image can be used as the edge contour information of the pose tracking image preceding the first pose tracking image.

[0079] (2) For the current pose tracking image in the at least one pose tracking image, determine the current edge contour information that matches the previous edge contour information from the current pose tracking image; wherein the previous edge contour information is the edge contour information corresponding to the previous pose tracking image of the current pose tracking image.

[0080] In this step, the current edge contour information corresponding to the current pose tracking image can be determined sequentially according to the time sequence of at least one pose tracking image. The current edge contour information can be determined based on the previous edge contour information and the current pose tracking image. The previous edge contour information can be the edge contour information of the pose tracking image preceding the current pose tracking image in the time sequence.

[0081] For example, the aforementioned previous edge contour information can be used as initial parameter information to establish a variable parameter curve that matches the initial parameter information in the current pose tracking image.

[0082] Among them, the variable parameter curve is a parameter curve with a variable shape. A parameter curve is a curve represented by a parametric equation. The parameter information in the variable parameter curve can be the coordinates of each point in the curve in the pose tracking image. By adjusting the parameter information in the variable parameter curve, the shape of the variable parameter curve can be adjusted.

[0083] After establishing the aforementioned variable parameter curves, the target parameter information of the variable parameter curves can be determined based on the energy function corresponding to the variable parameter curves.

[0084] Here, regarding the energy function in clustering, the objects to be clustered can be viewed as a system, and the degree of dissimilarity between objects can be considered as the energy between system elements. When the energy reaches a certain level, the objects form a new cluster, indicating that the system needs to be reclassified. The clustering process requires that each object belongs to a single cluster, that no system in each cluster has energy greater than a threshold, and that no system in different clusters has energy less than a threshold. Determining whether a point in an image is an edge contour point is essentially an image segmentation problem, which can be transformed into an energy minimization problem.

[0085] Solving a problem using an energy minimization model can involve two main steps. First, an objective function is described, which maps all possible solutions to a set of real numbers and gives the degree of goodness (or badness) of the possible solutions. An objective function typically corresponds to the sum of different constraint terms for the problem. These constraints can be soft or hard constraints. In this embodiment, the objective function is the aforementioned energy function. The constraint terms of the energy function can be determined by the features of the pixels on both sides of the contour. For example, the area outside the contour point is the background region, and the area inside is the region of the target object itself. The similarity of the pixels on the outside should be high, and the similarity of the pixels on the inside should also be high. However, there should be a significant difference between the background region and the region of the target object itself, that is, the similarity of the pixels on both sides of the contour point should be low. The corresponding constraint terms can be generated according to the above rules, thereby forming the energy function.

[0086] Therefore, the energy function described above can represent the degree to which the variable parameter curve is the contour curve in the current pose tracking image, and can be used to characterize the probability value of each pixel point in the variable parameter curve being a contour pixel point.

[0087] The second main step in solving the problem using the energy minimization model is to minimize the energy function, which is the aforementioned energy minimization deformation of the variable parameter curve. Through methods such as gradient descent and simulated annealing, the value of the energy function can be continuously reduced to its lowest value. As the value of the energy function decreases, the parameter information of the variable parameter curve also changes, and the trend of change is towards a direction similar to the contour information of the target object in the current pose tracking image. That is, the smaller the value of the energy function, the closer the variable parameter curve is to the real edge contour.

[0088] Therefore, based on the energy function corresponding to the variable parameter curve, the variable parameter curve can be transformed by minimizing energy to obtain the target parameter information corresponding to the variable parameter curve when the energy function is minimized.

[0089] In this way, by using the previous edge contour information to establish a variable parameter curve that matches the initial parameter information, and performing energy minimization deformation on the variable parameter curve, the target parameter information is obtained. Finally, the target parameter information is used as the current edge contour information. Since the previous pose tracking image and the current pose tracking image are two adjacent frames, the previous edge contour information is quite similar to the current edge contour information. Determining the current edge contour information based on the previous edge contour information can improve the accuracy of the current edge contour information and reduce the computational load of determining the current edge contour information to a certain extent.

[0090] (3) Determine the pose information corresponding to the current edge contour information, and use the determined pose information as the tracking pose information of the current pose tracking image.

[0091] In this step, the conversion relationship between pose information and edge contour information can be used to convert the current edge contour information into pose information, and the determined pose information can be used as the tracking pose information of the current pose tracking image.

[0092] For example, the pose of the virtual model can be continuously adjusted so that the edge contour of the virtual model matches the current edge contour information determined above, and the pose information of the virtual model at the time of matching is used as the tracking pose information of the current pose tracking image.

[0093] In this way, the current edge contour information of the first pose tracking image among multiple pose tracking images is determined by using the initial edge contour information determined from the initial pose image. For other pose tracking images, the current edge contour information is determined by using the previous edge contour information, and the tracking pose information is determined by using the current edge contour information. Pose tracking can be completed without the user having to manually align the target object and the preset edge contour.

[0094] During pose tracking, failures may occur. These failures typically occur because the edge contours of the target object differ significantly between two pose tracking images, making it impossible to determine its accurate pose information using the energy function. Examples include the target object being missing from the pose tracking image or the target object moving too fast. In such cases, it is necessary to re-perform the pose tracking initialization steps and reposition the target object to be tracked.

[0095] In some possible implementations, the following steps can be used to determine whether tracking has failed and to re-initialize:

[0096] Based on the energy function, determine the energy value corresponding to the variable parameter curve under the initial parameter information; based on the energy value and the preset energy threshold, determine whether the pose tracking of the target object has failed; if the pose tracking of the target object fails, update the initial pose image of the target object, and jump to the step of obtaining the initial pose information of the target object in the initial pose image.

[0097] In this step, the value of the energy function of the variable parameter curve under the initial parameter information can be calculated as the energy value of the variable parameter curve. When the energy value is greater than or equal to the preset energy threshold, it can be considered that the variable parameter curve is too different from the real edge contour, and the pose tracking of the target object is determined to be failed. After that, the initial pose image of the target object can be updated, and the process jumps to the step of obtaining the initial pose information of the target object in the initial pose image.

[0098] For example, an image containing the target object can be reacquired using a photographic device and used as the initial pose image.

[0099] In this way, by determining the energy value of the variable parameter curve under the initial parameter information, and judging whether the pose tracking has failed based on the determined energy value and the preset energy threshold, if it fails, the initial pose image is updated, the initial pose information of the initial pose image is re-determined, and then pose tracking is performed according to the re-determined initial pose information. When the tracking fails, the user does not need to manually initialize, which helps to improve the smoothness of the pose tracking operation and reduce the number of steps for the user.

[0100] Furthermore, after determining that the pose tracking of the target object has failed, the pose tracking image corresponding to the variable parameter curve can be used as the initial image containing the target object, and the region of interest corresponding to the target object can be identified from the initial image; then, the region of interest is extracted from the initial image, the region of interest is used as the initial pose image, and the process jumps to the step of determining the initial pose information of the target object in the initial pose image.

[0101] In this way, by using the pose tracking image corresponding to the failed pose tracking to determine the initial pose image in the event of pose tracking failure, and then using the determined initial pose image to re-determine pose information and re-track pose, a seamless connection of pose re-tracking can be achieved, thereby improving pose tracking efficiency.

[0102] The aforementioned pose tracking information is the tracking result obtained from pose tracking. It can be applied to various scenarios. For example, it can be applied to automated production lines for pose tracking of parts during automatic assembly. It can also be applied to intelligent transportation scenarios for tracking the pose information of vehicles on the road. Furthermore, it can be used in the AR platform of smartphones to identify the pose of target objects and generate corresponding AR effects.

[0103] For example, based on the tracking pose information of the target object corresponding to the plurality of pose tracking images, virtual effects corresponding to the tracking pose information can be added to at least one of the plurality of pose tracking images; the pose tracking image after adding the virtual effects can be displayed.

[0104] In this way, by adding virtual effects to at least one pose tracking image in multiple pose tracking highlights using the target object's tracking pose information, and displaying the pose tracking image after adding virtual effects, information related to the tracking pose information can be provided to the user, making it easier to understand and use the tracking pose information.

[0105] like Figure 2 As shown in the embodiments of this disclosure, another pose tracking method is disclosed. This method first acquires an RGB or RGBD image, and then determines whether the object's six-dimensional (6D) pose tracking module has completed initialization. Here, the 6D pose is the pose information mentioned above. If the initialization is not completed, the initialization step is executed, and the object's 6D pose is estimated for the first acquired RGB or RGBD image, and then the object's 6D pose tracking is performed. If the initialization is completed, the object's 6D pose tracking is performed directly (the specific tracking process can be seen in the embodiments of the pose tracking method described above). During the object's 6D pose tracking process, if the energy function value of the edge tracking is less than a threshold, it indicates that the pose tracking is successful, and the 6D pose of the object in the current frame is determined. If it is greater than or equal to the threshold, it indicates that the pose tracking has failed, and the process jumps to the initialization step, re-estimates the object's 6D pose, and executes the subsequent pose tracking steps.

[0106] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0107] Based on the same inventive concept, this disclosure also provides a pose tracking device corresponding to the pose tracking method. Since the principle of the device in this disclosure for solving the problem is similar to the pose tracking method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0108] Reference Figure 3 The diagram shown is a schematic representation of a pose tracking device provided in an embodiment of this disclosure. The device includes:

[0109] The acquisition module 310 is used to acquire the initial pose information of the target object in the initial pose image;

[0110] The initialization module 320 is used to adjust the pose of the virtual model of the target object according to the initial pose information, and to determine the initial edge contour information of the adjusted virtual model.

[0111] The tracking module 330 is used to determine the tracking pose information of the target object corresponding to the at least one pose tracking image based on the initial edge contour information and at least one pose tracking image containing the target object.

[0112] In one possible implementation, the acquisition module 310 is further configured to:

[0113] Obtain an initial image containing the target object;

[0114] The region of interest is extracted from the initial image, and the extracted region of interest is used as the initial pose image;

[0115] The initial pose information is obtained by estimating the pose information of the target object based on the region of interest.

[0116] In one possible implementation, the tracking module 330 is specifically used for:

[0117] The initial edge contour information is used as the edge contour information of the previous pose tracking image in the first pose tracking image among the at least one pose tracking images;

[0118] For the current pose tracking image in the at least one pose tracking image, determine the current edge contour information that matches the previous edge contour information from the current pose tracking image; wherein, the previous edge contour information is the edge contour information corresponding to the previous pose tracking image of the current pose tracking image.

[0119] Determine the pose information corresponding to the current edge contour information, and use the determined pose information as the tracking pose information of the current pose tracking image.

[0120] In one possible implementation, when the tracking module 330 determines the current edge contour information that matches the previous edge contour information of the previous pose tracking image from the current pose tracking image, it is specifically used for:

[0121] Using the previous edge contour information as initial parameter information, a variable parameter curve matching the initial parameter information is established in the current pose tracking image;

[0122] Based on the energy function corresponding to the variable parameter curve, the target parameter information of the variable parameter curve is determined; wherein, the energy function is used to characterize the probability value of each pixel point in the variable parameter curve being a contour pixel point;

[0123] The target parameter information is used as the current edge contour information that matches the previous edge contour information.

[0124] In one possible implementation, when the tracking module 330 determines the target parameter information of the variable parameter curve based on the energy function corresponding to the variable parameter curve, it is used to:

[0125] Based on the energy function corresponding to the variable parameter curve, the variable parameter curve is transformed by minimizing energy to obtain the target parameter information corresponding to the variable parameter curve when the energy function is minimized.

[0126] In one possible implementation, the initialization module 320 is further configured to:

[0127] Based on the energy function, determine the energy value corresponding to the variable parameter curve under the initial parameter information;

[0128] Based on the energy value and the preset energy threshold, determine whether the pose tracking for the target object has failed;

[0129] If pose tracking of the target object fails, update the initial pose image of the target object and proceed to the step of obtaining the initial pose information of the target object in the initial pose image.

[0130] In one possible implementation, the initialization module 320 is further configured to:

[0131] Based on the energy function, determine the energy value corresponding to the variable parameter curve under the initial parameter information;

[0132] Based on the energy value and the preset energy threshold, determine whether the pose tracking for the target object has failed;

[0133] In the event that pose tracking of the target object fails, the pose tracking image corresponding to the variable parameter curve is used as the initial image containing the target object, and the region of interest corresponding to the target object is identified from the initial image.

[0134] The region of interest is extracted from the initial image, and the region of interest is used as the initial pose image. Then, the process jumps to the step of determining the initial pose information of the target object in the initial pose image.

[0135] In one possible implementation, the device further includes a display module for:

[0136] Based on the tracking pose information of the target object corresponding to the at least one pose tracking image, add virtual effects corresponding to the tracking pose information to at least one of the at least one pose tracking images;

[0137] The pose tracking image is displayed after adding virtual effects.

[0138] Corresponding to Figure 1 In addition to the pose tracking method in this disclosure, this embodiment also provides an electronic device 400, such as... Figure 4 The diagram shown is a structural schematic of an electronic device 400 provided in an embodiment of this disclosure, including:

[0139] The system includes a processor 41, a memory 42, and a bus 43. The memory 42 stores execution instructions and includes main memory 421 and external memory 422. The main memory 421, also called internal memory, temporarily stores the computational data in the processor 41, as well as data exchanged with external memory such as a hard disk. The processor 41 exchanges data with the external memory 422 through the main memory 421. When the electronic device 400 is running, the processor 41 communicates with the memory 42 through the bus 43, enabling the processor 41 to execute the following instructions:

[0140] Obtain the initial pose information of the target object in the initial pose image;

[0141] Based on the initial pose information, the pose of the virtual model of the target object is adjusted, and the initial edge contour information of the adjusted virtual model is determined.

[0142] Based on the initial edge contour information and at least one pose tracking image containing the target object, the tracking pose information of the target object corresponding to the at least one pose tracking image is determined.

[0143] The specific execution process of the above instructions can be referred to the steps of the pose tracking method described in the embodiments of this disclosure, and will not be repeated here.

[0144] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the pose tracking method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0145] This disclosure also provides a computer program product, including a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the pose tracking method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0146] The computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A pose tracking method, characterized in that, include: Obtain the initial pose information of the target object in the initial pose image; Based on the initial pose information, the pose of the virtual model of the target object is adjusted, and the initial edge contour information of the adjusted virtual model is determined. Based on the initial edge contour information and at least one pose tracking image containing the target object, determine the tracking pose information of the target object corresponding to the at least one pose tracking image; The step of adjusting the pose of the virtual model of the target object based on the initial pose information and determining the initial edge contour information of the adjusted virtual model includes: Adjust the pose of the virtual model of the target object so that the pose of the virtual model is consistent with the pose of the target object; The adjusted virtual model is rendered as a two-dimensional edge contour, and the initial edge contour information corresponding to the rendered edge contour is determined. The step of determining the tracking pose information of the target object corresponding to each of the at least one pose tracking image based on the initial edge contour information and the pose tracking image containing the target object includes: The initial edge contour information is used as the edge contour information of the previous pose tracking image in the first pose tracking image among the at least one pose tracking images; For the current pose tracking image in the at least one pose tracking image, determine the current edge contour information that matches the previous edge contour information from the current pose tracking image; wherein, the previous edge contour information is the edge contour information corresponding to the previous pose tracking image of the current pose tracking image. Determine the pose information corresponding to the current edge contour information, and use the determined pose information as the tracking pose information of the current pose tracking image; Determining the current edge contour information from the current pose tracking image that matches the previous edge contour information includes: Using the previous edge contour information as initial parameter information, a variable parameter curve matching the initial parameter information is established in the current pose tracking image; Based on the energy function corresponding to the variable parameter curve, the target parameter information of the variable parameter curve is determined; wherein, the energy function is used to characterize the probability value of each pixel point in the variable parameter curve being a contour pixel point; The target parameter information is used as the current edge contour information that matches the previous edge contour information; after establishing the variable parameter curve that matches the initial parameter information, the method further includes: Based on the energy function, determine the energy value corresponding to the variable parameter curve under the initial parameter information; Based on the energy value and the preset energy threshold, determine whether the pose tracking for the target object has failed; If pose tracking of the target object fails, update the initial pose image of the target object and proceed to the step of obtaining the initial pose information of the target object in the initial pose image.

2. The method according to claim 1, characterized in that, The step of obtaining the initial pose information of the target object in the initial pose image includes: Obtain an initial image containing the target object; Identify the region of interest corresponding to the target object from the initial image; The region of interest is extracted from the initial image, and the extracted region of interest is used as the initial pose image; The pose information of the target object is estimated based on the region of interest to obtain the initial pose information.

3. The method according to claim 1, characterized in that, The step of determining the target parameter information of the variable parameter curve based on the energy function corresponding to the variable parameter curve includes: Based on the energy function corresponding to the variable parameter curve, the variable parameter curve is transformed by minimizing energy to obtain the target parameter information corresponding to the variable parameter curve when the energy function is minimized.

4. The method according to claim 1, characterized in that, Updating the initial pose image of the target object includes: The pose tracking image corresponding to the variable parameter curve is used as the initial image containing the target object, and the region of interest corresponding to the target object is identified from the initial image; The region of interest is extracted from the initial image, and the region of interest is used as the initial pose image. Then, the process jumps to the step of determining the initial pose information of the target object in the initial pose image.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Based on the tracking pose information of the target object corresponding to the at least one pose tracking image, add virtual effects corresponding to the tracking pose information to the at least one pose tracking image; The pose tracking image is displayed after adding virtual effects.

6. A pose tracking device, characterized in that, include: The acquisition module is used to acquire the initial pose information of the target object in the initial pose image; An initialization module is used to adjust the pose of the virtual model of the target object according to the initial pose information, and to determine the initial edge contour information of the adjusted virtual model. The tracking module is used to determine the tracking pose information of the target object corresponding to the at least one pose tracking image based on the initial edge contour information and at least one pose tracking image containing the target object. The initialization module is specifically used to: adjust the pose of the virtual model of the target object so that the pose of the virtual model is consistent with the pose of the target object; render the adjusted virtual model as a two-dimensional edge contour, and determine the initial edge contour information corresponding to the rendered edge contour; The tracking module is specifically used to: use the initial edge contour information as the edge contour information of the previous pose tracking image of the first pose tracking image in the at least one pose tracking image; For the current pose tracking image in the at least one pose tracking image, determine the current edge contour information that matches the previous edge contour information from the current pose tracking image; wherein, the previous edge contour information is the edge contour information corresponding to the previous pose tracking image of the current pose tracking image. Determine the pose information corresponding to the current edge contour information, and use the determined pose information as the tracking pose information of the current pose tracking image; Determining the current edge contour information from the current pose tracking image that matches the previous edge contour information includes: Using the previous edge contour information as initial parameter information, a variable parameter curve matching the initial parameter information is established in the current pose tracking image; Based on the energy function corresponding to the variable parameter curve, the target parameter information of the variable parameter curve is determined; wherein, the energy function is used to characterize the probability value of each pixel point in the variable parameter curve being a contour pixel point; The target parameter information is used as the current edge contour information that matches the previous edge contour information; The initialization module is also used for: Based on the energy function, determine the energy value corresponding to the variable parameter curve under the initial parameter information; Based on the energy value and the preset energy threshold, determine whether the pose tracking for the target object has failed; If pose tracking of the target object fails, update the initial pose image of the target object and proceed to the step of obtaining the initial pose information of the target object in the initial pose image.

7. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor executing the machine-readable instructions stored in the memory, wherein when the machine-readable instructions are executed by the processor, the processor performs the steps of the pose tracking method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer device, performs the steps of the pose tracking method as described in any one of claims 1 to 5.

Images