Position determination methods, devices, equipment and media
By acquiring the initial pose and extracting feature points from video frames, the coordinate information and transformation relationships are determined, solving the problems of high computational load and low accuracy in the generation of 3D hand models, and realizing efficient and accurate pose determination in mobile devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-25
- Publication Date
- 2026-03-06
AI Technical Summary
Existing technologies suffer from high computational demands for generating 3D hand models and inaccurate model predictions, especially in mobile devices.
By acquiring the initial pose of the target object and video frames, feature points are extracted to determine coordinate information and transformation relationships. Based on this information, the pose of the target object is calculated, simplifying the computation and improving prediction accuracy.
This reduces the computational load in the pose determination process, avoids discontinuities in inter-frame observations, and improves the accuracy of pose determination.
Smart Images
Figure CN115237254B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of virtual reality, and more particularly to a pose determination method, apparatus, device, and medium. Background Technology
[0002] With the development of Augmented Reality (AR) technology, the interaction between virtual information and the real world is becoming increasingly frequent. Hand Gesture Detection and Tracking is a classic topic in computer vision. In AR and MR, we typically aim to predict a 3D model of the hand to better facilitate virtual-real interaction. However, current technologies often employ the following two methods to generate the 3D model of the hand.
[0003] (i) Using frame-by-frame prediction to generate the 3D model of the hand will significantly increase the amount of computation required for model generation, resulting in extremely high performance requirements for the device, which is not conducive to its use on mobile devices.
[0004] (ii) Using generated data to alleviate the problem of lack of training data in real-world scenarios has significantly improved the accuracy of 3D gesture prediction in real-world datasets. However, these methods often adopt the method of joint training of generated data and real data, ignoring the domain differences between the two. Since the differences between generated data and real data are unavoidable, feature alignment or feature mapping is difficult, and the common features of the two cannot be effectively learned, thus affecting the accuracy.
[0005] In summary, the generation of 3D hand models suffers from problems such as high computational cost and inaccurate model prediction. Summary of the Invention
[0006] This disclosure provides a pose determination method, apparatus, device, and medium to at least solve the problems of high computational complexity and inaccurate model prediction in related technologies. The technical solution of this disclosure is as follows:
[0007] According to a first aspect of the present disclosure, a pose determination method is provided, comprising:
[0008] The initial pose of the target object, a first video frame containing the target object, and a second video frame containing the target object are obtained. The first video frame is the previous video frame adjacent to the second video frame, and the second video frame is the current video frame. The initial pose is determined based on the first video frame.
[0009] Feature points are extracted from the target objects in the first video frame and the second video frame respectively to obtain the first coordinate information of the target objects in the first video frame and the second coordinate information of the target objects in the second video frame.
[0010] Determine the transformation relationship between the first coordinate information and the initial pose;
[0011] Based on the first coordinate information, the transformation relationship, and the second coordinate information, the target pose of the target object in the second video frame is determined.
[0012] In one possible implementation, the method provided in this disclosure extracts feature points from the target object in the first video frame and the second video frame, respectively, including:
[0013] Key point localization processing is performed on the target object in the first video frame and the second video frame respectively, and two sets of key points of the target object are obtained accordingly.
[0014] Determine the feature region based on each set of key points;
[0015] Feature point extraction is performed in the feature region to obtain multiple feature points corresponding to the first video frame and the second video frame, respectively. The feature points are used to characterize the pose features of the target object in the feature region.
[0016] In one possible implementation, the method provided in this disclosure, determining the transformation relationship between the first coordinate information and the initial pose, includes:
[0017] Determine the shape data of the target object based on the initial pose;
[0018] Based on the first coordinate information and shape data, determine the first pose transformation result of the target object in the first video frame relative to the initial pose;
[0019] The transformation relationship is determined based on the first coordinate information and the first pose transformation result.
[0020] In one possible implementation, the method provided in this disclosure determines the target pose of a target object in a second video frame based on first coordinate information, a transformation relationship, and second coordinate information, including:
[0021] Based on the first coordinate information, the second coordinate information, and the first pose transformation result, determine the second pose transformation result of the target object in the second video frame relative to the initial pose;
[0022] Based on the second pose transformation result and shape data, the target pose of the target object in the second video frame is determined.
[0023] In one possible implementation, the method provided in this disclosure, based on first coordinate information and shape data, determines the first pose transformation result of the target object in the first video frame relative to the initial pose, including:
[0024] Based on the first coordinate information and shape data, pose localization processing is performed on the target object in the first video frame to obtain the first pose transformation result of the target object in the first video frame relative to the initial pose.
[0025] Based on the second coordinate information and the first pose transformation result, the second pose transformation result of the target object in the second video frame relative to the initial pose is determined, including:
[0026] Based on the second coordinate information and shape data, pose localization processing is performed on the target object in the second video frame to obtain the second pose transformation result of the target object in the second video frame relative to the first pose transformation result.
[0027] In one possible implementation, after determining the second pose transformation result of the target object relative to the initial pose in the second video frame based on the second coordinate information and the first pose transformation result, the method further includes:
[0028] The second coordinate information is reprojected onto the first video frame, and the second pose transformation result is corrected based on the reprojection result.
[0029] According to a second aspect of the present disclosure, a pose determination apparatus is provided, comprising:
[0030] The acquisition unit is configured to acquire the initial pose of the target object, a first video frame containing the target object, and a second video frame containing the target object. The first video frame is the previous video frame adjacent to the second video frame, and the second video frame is the current video frame. The initial pose is determined based on the first video frame.
[0031] The extraction unit is configured to perform feature point extraction on the target object in the first video frame and the second video frame respectively, and obtain the first coordinate information corresponding to the target object in the first video frame and the second coordinate information corresponding to the target object in the second video frame.
[0032] The first determining unit is configured to perform the transformation relationship between the first coordinate information and the initial pose;
[0033] The second determining unit is configured to determine the target pose of the target object in the second video frame based on the first coordinate information, the transformation relationship, and the second coordinate information.
[0034] In one possible implementation, the extraction unit in the apparatus provided by this disclosure is specifically configured to perform:
[0035] Key point localization processing is performed on the target object in the first video frame and the second video frame respectively, and two sets of key points of the target object are obtained accordingly.
[0036] Determine the feature region based on each set of key points;
[0037] Feature point extraction is performed in the feature region to obtain multiple feature points corresponding to the first video frame and the second video frame, respectively. The feature points are used to characterize the pose features of the target object in the feature region.
[0038] In one possible implementation, the first determining unit in the apparatus provided in this disclosure is specifically configured to perform:
[0039] Determine the shape data of the target object based on the initial pose;
[0040] Based on the first coordinate information and shape data, determine the first pose transformation result of the target object in the first video frame relative to the initial pose;
[0041] The transformation relationship is determined based on the first coordinate information and the first pose transformation result.
[0042] In one possible implementation, the second determining unit in the apparatus provided in this disclosure is specifically configured to perform:
[0043] Based on the first coordinate information, the second coordinate information, and the first pose transformation result, determine the second pose transformation result of the target object in the second video frame relative to the initial pose;
[0044] Based on the second pose transformation result and shape data, the target pose of the target object in the second video frame is determined.
[0045] In one possible implementation, the first determining unit in the apparatus provided in this disclosure is specifically configured to perform:
[0046] Based on the first coordinate information and shape data, pose localization processing is performed on the target object in the first video frame to obtain the first pose transformation result of the target object in the first video frame relative to the initial pose.
[0047] The second determining unit is specifically configured to execute:
[0048] Based on the second coordinate information and shape data, pose localization processing is performed on the target object in the second video frame to obtain the second pose transformation result of the target object in the second video frame relative to the first pose transformation result.
[0049] In one possible implementation, the second determining unit in the apparatus provided by this disclosure is further configured to perform:
[0050] The second coordinate information is reprojected onto the first video frame, and the second pose transformation result is corrected based on the reprojection result.
[0051] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor;
[0052] Memory used to store processor-executable instructions;
[0053] The processor is configured to execute the instructions to implement the pose determination method described in any one of the first aspects of the embodiments of this disclosure.
[0054] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising: a processor;
[0055] Memory used to store processor-executable instructions;
[0056] The processor is configured to execute the instructions to implement the pose determination method described in any one of the first aspects of the embodiments of this disclosure.
[0057] According to a fifth aspect of the present disclosure, a storage medium is provided such that, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is able to perform the pose determination method described in any one of the first aspects of the present disclosure.
[0058] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the pose determination method as described in any one of the first aspects.
[0059] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:
[0060] The method acquires the initial pose of the target object, a first video frame containing the target object, and a second video frame containing the target object. Feature points are extracted from the target object in both the first and second video frames to obtain the first coordinate information and the second coordinate information corresponding to the target object in the first and second video frames, respectively. The transformation relationship between the first coordinate information and the initial pose is determined. Based on the first coordinate information, the transformation relationship, and the second coordinate information, the target pose of the target object in the second video frame is determined. Compared with related technologies, this method eliminates the need for frame-by-frame model prediction, simplifying the computational load in the pose determination process. Furthermore, it performs low-computation prediction of the pose for each frame based on the transformation relationship, avoiding inter-frame observation discontinuities caused by network output errors, thereby improving the accuracy of pose determination.
[0061] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.
[0063] Figure 1 This is a schematic diagram of a 21-bone joint of the hand according to an exemplary embodiment.
[0064] Figure 2 This is a flowchart illustrating a pose determination method according to an exemplary embodiment.
[0065] Figure 3 This is a schematic diagram illustrating a feature point extraction method according to an exemplary embodiment.
[0066] Figure 4 This is a schematic diagram illustrating a specific process of a pose determination method according to an exemplary embodiment.
[0067] Figure 5 This is a schematic diagram illustrating a specific process of another pose determination method according to an exemplary embodiment.
[0068] Figure 6 This is a schematic diagram of the structure of a pose determination device according to an exemplary embodiment.
[0069] Figure 7 This is a schematic diagram of the structure of a pose determination device according to an exemplary embodiment.
[0070] Figure 8 This is a schematic diagram of the structure of a terminal for applying a pose determination method according to an exemplary embodiment. Detailed Implementation
[0071] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.
[0072] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0073] The following are explanations of some of the words that appear in the text:
[0074] 1. In the embodiments of this disclosure, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.
[0075] 2. Augmented Reality (AR) is a technology that cleverly integrates virtual information with the real world. It widely uses various technologies such as multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensing to simulate and apply computer-generated virtual information such as text, images, 3D models, music, and videos to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.
[0076] 3. It combines the real and virtual worlds to create new environments and visualizations (Mixed Reality, MR), where physical entities and digital objects coexist and interact in real time to simulate real objects. It blends real, augmented reality, augmented virtual, and virtual reality technologies.
[0077] 4. In this embodiment of the disclosure, the term "hand bone node" is defined as follows: 21 hand bone nodes. Figure 1 As shown, node 0 is considered to be the root node of the hand.
[0078] 5. In this embodiment of the disclosure, the term "hand mesh point" refers to the definition of 778 hand mesh points. A three-dimensional reconstruction mesh model of the hand can be obtained by connecting the hand mesh points in a certain way.
[0079] 6. In the embodiments of this disclosure, the term "hand coordinate system" refers to a three-dimensional coordinate system established with the root node of the hand as the center, used to represent the coordinates (x, y, z) of key points of the hand in the hand coordinate system. In the embodiments of this disclosure, w is used to represent it uniformly.
[0080] 7. In the embodiments of this disclosure, the term "camera coordinate system" refers to a three-dimensional coordinate system established with the camera as the center, used to represent the coordinates (x, y, z) of map points in the camera coordinate system. In the embodiments of this disclosure, c is used to represent it uniformly.
[0081] 8. In the embodiments of this disclosure, the term "pixel coordinate system" refers to a two-dimensional coordinate system established with the upper left corner of the image as the center, used to represent the position (x, y) of two-dimensional image feature points on the image. In the embodiments of this disclosure, 2d representation will continue to be used uniformly.
[0082] With the development of Augmented Reality (AR) technology, the interaction between virtual information and the real world is becoming increasingly frequent. Hand Gesture Detection and Tracking is a classic topic in computer vision. In AR and MR, we typically aim to predict a 3D model of the hand to better facilitate virtual-real interaction. However, current technologies often employ the following two methods to generate the 3D model of the hand.
[0083] (a) Using frame-by-frame prediction to generate the 3D model of the hand places extremely high demands on the device's performance, which is not conducive to its use on mobile devices.
[0084] (ii) Using generated data to alleviate the problem of lack of training data in real-world scenarios has significantly improved the accuracy of 3D gesture prediction in real-world datasets. However, these methods often adopt the method of joint training of generated data and real data, ignoring the domain differences between the two. Since the differences between generated data and real data are unavoidable, feature alignment or feature mapping is difficult, and the common features of the two cannot be effectively learned, thus affecting the accuracy.
[0085] In summary, the generation of 3D hand models suffers from problems such as high computational cost and inaccurate model prediction.
[0086] Figure 2 This is a flowchart illustrating a pose determination method according to an exemplary embodiment, such as... Figure 2 As shown, the pose determination method includes the following steps.
[0087] In step S201, the initial pose of the target object, the first video frame containing the target object, and the second video frame containing the target object are obtained.
[0088] The first video frame is the preceding video frame adjacent to the second video frame, and the second video frame is the current video frame. Both the first and second video frames contain the target object.
[0089] It should be noted that the initial pose is determined based on the first video frame. It can be obtained by processing the first video frame through a deep learning network, or it can be obtained by using the pose of the first video frame generated by this scheme from the previous frame.
[0090] In step S202, feature points are extracted from the target objects in the first video frame and the second video frame respectively to obtain the first coordinate information corresponding to the target objects in the first video frame and the second coordinate information corresponding to the target objects in the second video frame.
[0091] In specific implementation, such as Figure 3As shown, keypoint localization processing is performed on the target object in the first and second video frames respectively, obtaining two sets of keypoints contained in the target object. Then, the closed region enclosed by each set of keypoints is determined as a feature region. Finally, feature point extraction processing is performed in the feature region to obtain multiple feature points corresponding to the first and second video frames respectively. Then, the first coordinate information and the second coordinate information are obtained based on the feature points. Among them, the feature points are used to represent the pose features of the target object within the feature region.
[0092] Of course, this is not limited to extracting feature points only on the palm. Feature points can also be extracted in locations where there will be little deformation, such as the middle of a finger segment. At the same time, there is no restriction on the type of feature points extracted. They can be feature points or feature lines, etc. This disclosure does not limit this.
[0093] In step S203, the transformation relationship between the first coordinate information and the initial pose is determined.
[0094] In practice, the shape data of the target object, i.e. the hand shape vector, is determined based on the initial pose. Then, based on the first coordinate information and the shape data, the pose localization processing of the target object in the first video frame is performed to obtain the first pose transformation result of the target object in the first video frame relative to the initial pose.
[0095] In step S204, the target pose of the target object in the second video frame is determined based on the first coordinate information, the transformation relationship, and the second coordinate information.
[0096] In practice, based on the second coordinate information and shape data, the target object in the second video frame is subjected to pose localization processing to obtain the second pose transformation result of the target object in the second video frame relative to the first pose transformation result.
[0097] In one possible implementation, the second coordinate information is reprojected onto the first video frame, and the second pose transformation result is corrected based on the reprojection result. Of course, other methods can also be used for error correction.
[0098] Compared with existing technologies, the technical solution provided in this disclosure simplifies the computational load in the pose determination process, avoids inter-frame observation discontinuity caused by network output errors, and improves the accuracy of pose determination.
[0099] The following is combined Figure 4 The specific process of the pose determination method provided in the embodiments of this disclosure will be described in detail.
[0100] In step S401, the initial pose of the target object, the first video frame containing the target object, and the second video frame containing the target object are obtained.
[0101] In practice, the first video frame is the preceding video frame adjacent to the second video frame, the second video frame is the current video frame, and both the first and second video frames contain the target object.
[0102] It should be noted that the initial pose is determined based on the first video frame. It can be obtained by processing the first video frame through a deep learning network, or it can be obtained by using the pose of the first video frame generated by this scheme from the previous frame.
[0103] In step S402, feature points are extracted from the target objects in the first video frame and the second video frame, respectively.
[0104] In specific implementation, such as Figure 3 As shown, keypoint localization is performed on the target object in the first and second video frames respectively, obtaining two sets of keypoints contained in the target object. The closed region enclosed by each set of keypoints is then defined as a feature region. Finally, feature point extraction is performed within the feature region to obtain multiple feature points corresponding to the first and second video frames. Based on these feature points, the first and second coordinate information are obtained. By defining the feature region, the scope of feature extraction is reduced, thereby reducing the computational load during the feature extraction process.
[0105] Of course, this is not limited to extracting feature points only on the palm. Feature points can also be extracted in locations where there will be little deformation, such as the middle of a finger segment. At the same time, there is no restriction on the type of feature points extracted. They can be feature points or feature lines, etc. This disclosure does not limit this.
[0106] In step S403, the first pose conversion result is determined.
[0107] In practice, the shape data of the target object, i.e., the hand shape vector, is determined based on the initial pose. Then, based on the first coordinate information and the shape data, pose localization processing is performed on the target object in the first video frame to obtain the first pose transformation result of the target object in the first video frame relative to the initial pose, which is denoted here as vector T. c1w To express.
[0108] In step S404, the second pose conversion result is determined.
[0109] In practice, the shape data of the target object, i.e., the hand shape vector, is determined based on the initial pose. Then, based on the second coordinate information and the shape data, pose localization processing is performed on the target object in the second video frame to obtain the second pose transformation result of the target object in the second video frame relative to the initial pose. Here, vector T is used. c2w To express.
[0110] In step S405, the second pose conversion result is corrected.
[0111] In practice, the second pose transformation result is corrected by using other auxiliary methods. For example, the motion transformation matrix T from the current frame camera coordinate system to the first frame camera coordinate system can be optimized by constructing the following four error terms and minimizing these error terms. c1c2 And the hand motion vector, thus correcting the second pose transformation result. Where T c2w =T c2c1 *T c1w .
[0112] (a) Reprojection error of feature points converted from 3D to 2D.
[0113] (ii) Difference in hand motion vectors between adjacent frames.
[0114] (iii) Physiological limitations on the range of motion of local human joints.
[0115] (iv) The difference between the 2D bone node coordinates output by the network and the 2D bone node coordinates recovered from the hand motion vector.
[0116] In step S406, the target pose is determined.
[0117] In practice, the hand shape vector, the hand motion vector of the current image frame, and the pose matrix T of the camera pose relative to the hand coordinate system are used. c2w The hand mesh in the camera coordinate system is restored.
[0118] In the above embodiments, it is equivalent to calculating for each video frame and correcting the second pose transformation result by introducing other auxiliary means, thereby improving the accuracy of hand position prediction. However, sometimes due to limitations in device performance, frame-by-frame reconstruction is not performed. The following describes a method that addresses this issue. Figure 5 Taking a two-frame interval as an example, the specific process of another pose determination method provided in this embodiment will be described in detail.
[0119] In step S501, the initial pose of the target object, the first video frame, the second video frame, the third video frame, and the fourth video frame are obtained.
[0120] In specific implementation, the first video frame is the video frame preceding the second video frame, the second video frame is the video frame preceding the third video frame, the third video frame is the video frame preceding the fourth video frame, and the target object is contained in both the first and second video frames.
[0121] It should be noted that the initial pose is determined based on the first video frame. It can be obtained by processing the first video frame through a deep learning network, or it can be obtained by using the pose of the first video frame generated by this scheme from the previous frame.
[0122] In step S502, feature points are extracted from the target object in each video frame.
[0123] In specific implementation, such as Figure 3 As shown, key point localization processing is performed on each target object to obtain four sets of key points contained in the target object. Then, the closed area enclosed by each set of key points is determined as the feature region. Finally, feature point extraction processing is performed in the feature region to obtain multiple sets of feature points. Then, the first coordinate information, second coordinate information, third coordinate information and fourth coordinate information are obtained based on the feature points.
[0124] Of course, this is not limited to extracting feature points only on the palm. Feature points can also be extracted in locations where there will be little deformation, such as the middle of a finger segment. At the same time, there is no restriction on the type of feature points extracted. They can be feature points or feature lines, etc. This disclosure does not limit this.
[0125] In step S503, the first pose conversion result is determined.
[0126] In practice, the shape data of the target object, i.e., the hand shape vector, is determined based on the initial pose. Then, based on the first coordinate information and the shape data, pose localization processing is performed on the target object in the first video frame to obtain the first pose transformation result of the target object in the first video frame relative to the initial pose, which is denoted here as vector T. c1w To express.
[0127] In step S504, the second pose conversion result is determined.
[0128] In practice, the shape data of the target object, i.e., the hand shape vector, is determined based on the initial pose. Then, based on the second coordinate information and the shape data, pose localization processing is performed on the target object in the second video frame to obtain the second pose transformation result of the target object in the second video frame relative to the initial pose. Here, vector T is used. c2w To express.
[0129] In step S505, the second pose conversion result is corrected.
[0130] In practice, the second pose transformation result is corrected by using other auxiliary methods. For example, the motion transformation matrix T from the second video frame camera coordinate system to the first video frame can be optimized by constructing and minimizing the following error terms. c1c2 And the hand motion vector, thus correcting the second pose transformation result. T c2w =T c2c1 *T c1w .
[0131] (I) Reprojection error of 3D-2D feature points
[0132] (ii) Difference in hand motion vectors between adjacent frames
[0133] (III) Physiological limitations on the range of motion of local joints
[0134] In step S506, the third pose conversion result is determined.
[0135] In practice, based on the third coordinate information and shape data, pose localization processing is performed on the target object in the third video frame to obtain the third pose transformation result of the target object in the third video frame relative to the initial pose. Here, vector T is used. c3w To express.
[0136] In step S507, the third pose conversion result is corrected.
[0137] By constructing several error terms similar to those in S505 and minimizing these error terms, the motion transformation matrix T from the camera coordinate system of the third video frame to the second video frame is optimized. c2c3 And the hand motion vector. T c3w =T c3c2 *T c2c1 *T c1w .
[0138] In step S508, the fourth pose conversion result is determined.
[0139] The determination method is the same as described above, and will not be repeated here. Here, we use vector T. c4w To express.
[0140] In step S509, the fourth pose conversion result is corrected.
[0141] By constructing several error terms similar to those in S505 and minimizing these error terms, the motion transformation matrix T from the camera coordinate system of the fourth video frame to the third video frame is optimized. c3c4 And the hand motion vector. T c4w =T c4c3 *T c3c2 *T c2c1 *T c1w .
[0142] In step S510, the target pose is determined.
[0143] In practice, the hand shape vector, the hand motion vector of the fourth video frame, and the pose matrix T of the camera pose relative to the coordinate system in the fourth video frame are used. c4w The hand mesh in the camera coordinate system is recovered to determine the target pose.
[0144] In the above embodiments, by introducing other auxiliary means to correct the second pose transformation results, the third pose transformation results and the fourth pose transformation results, the accuracy of hand position prediction is improved. At the same time, hand position prediction is performed every few frames, which further reduces the amount of computation and reduces the loss of functionality.
[0145] Figure 6 This is a block diagram illustrating a pose determination device according to an exemplary embodiment. (Refer to...) Figure 6 The device includes an acquisition unit 601, an extraction unit 602, a first determination unit 603, and a second determination unit 604.
[0146] The acquisition unit 601 is configured to acquire the initial pose of the target object, a first video frame containing the target object, and a second video frame containing the target object. The first video frame is the previous video frame adjacent to the second video frame, and the second video frame is the current video frame. The initial pose is determined based on the first video frame.
[0147] The extraction unit 602 is configured to perform feature point extraction on the target object in the first video frame and the second video frame respectively, and obtain the first coordinate information corresponding to the target object in the first video frame and the second coordinate information corresponding to the target object in the second video frame.
[0148] The first determining unit 603 is configured to determine the transformation relationship between the first coordinate information and the initial pose;
[0149] The second determining unit 604 is configured to determine the target pose of the target object in the second video frame based on the first coordinate information, the transformation relationship and the second coordinate information.
[0150] In one possible implementation, the extraction unit 602 in the apparatus provided by this disclosure is specifically configured to perform:
[0151] Key point localization processing is performed on the target object in the first video frame and the second video frame respectively, and two sets of key points of the target object are obtained accordingly.
[0152] Determine the feature region based on each set of key points;
[0153] Feature point extraction is performed in the feature region to obtain multiple feature points corresponding to the first video frame and the second video frame, respectively. The feature points are used to characterize the pose features of the target object in the feature region.
[0154] In one possible implementation, the first determining unit 603 in the apparatus provided in this disclosure is specifically configured to perform:
[0155] Determine the shape data of the target object based on the initial pose;
[0156] Based on the first coordinate information and shape data, determine the first pose transformation result of the target object in the first video frame relative to the initial pose;
[0157] The transformation relationship is determined based on the first coordinate information and the first pose transformation result.
[0158] In one possible implementation, the second determining unit 604 in the apparatus provided by this disclosure is specifically configured to perform:
[0159] Based on the first coordinate information, the second coordinate information, and the first pose transformation result, determine the second pose transformation result of the target object in the second video frame relative to the initial pose;
[0160] Based on the second pose transformation result and shape data, the target pose of the target object in the second video frame is determined.
[0161] In one possible implementation, the first determining unit 603 in the apparatus provided in this disclosure is specifically configured to perform:
[0162] Based on the first coordinate information and shape data, pose localization processing is performed on the target object in the first video frame to obtain the first pose transformation result of the target object in the first video frame relative to the initial pose.
[0163] The second determining unit 604 is specifically configured to execute:
[0164] Based on the second coordinate information and shape data, pose localization processing is performed on the target object in the second video frame to obtain the second pose transformation result of the target object in the second video frame relative to the first pose transformation result.
[0165] In one possible implementation, the second determining unit 604 in the apparatus provided by this disclosure is further configured to perform:
[0166] The second coordinate information is reprojected onto the first video frame, and the second pose transformation result is corrected based on the reprojection result.
[0167] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment.
[0168] Processor 710;
[0169] Memory 720 for storing executable instructions of the processor 710;
[0170] The processor 710 is configured to execute the instructions to implement the pose determination method in this embodiment of the present disclosure.
[0171] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 720 including instructions, which can be executed by a processor 710 of device 700 to perform the above method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.
[0172] In the embodiments disclosed herein, such as Figure 8 As shown in the figure, this disclosure provides a terminal 800 applying the pose determination method provided in this disclosure, including: a radio frequency (RF) circuit 810, a power supply 820, a processor 830, a memory 840, an input unit 850, a display unit 860, a camera 870, a communication interface 880, and a wireless Fidelity (Wi-Fi) module 890, etc. Those skilled in the art will understand that... Figure 8 The structure of the terminal shown in the figure does not constitute a limitation on the terminal. The terminal provided in the embodiments of this application may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0173] The following is combined Figure 8 The various components of the terminal 800 will be described in detail below:
[0174] The RF circuit 810 can be used for receiving and transmitting data during communication or a call. Specifically, after receiving downlink data from the base station, the RF circuit 810 sends it to the processor 830 for processing; additionally, it sends uplink data to be transmitted to the base station. Typically, the RF circuit 810 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc.
[0175] In addition, the RF circuit 810 can also communicate wirelessly with networks and other terminals. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).
[0176] Wi-Fi technology is a short-range wireless transmission technology. The terminal 800 can connect to an access point (AP) via the Wi-Fi module 890, thereby enabling access to the data network. The Wi-Fi module 890 can be used for receiving and sending data during communication.
[0177] The terminal 800 can physically connect to other terminals through the communication interface 880. Optionally, the communication interface 880 can be connected to the communication interfaces of other terminals via a cable to enable data transmission between the terminal 800 and other terminals.
[0178] Since the terminal 800 in this embodiment of the application can perform communication services and send information to other contacts, the terminal 800 needs to have data transmission capabilities, that is, the terminal 800 needs to include a communication module. Although Figure 8 The RF circuit 810, the Wi-Fi module 890, and the communication interface 880 are shown, but it is understood that the terminal 800 contains at least one of the above-mentioned components or other communication modules (such as a Bluetooth module) for data transmission.
[0179] For example, when the terminal 800 is a mobile phone, the terminal 800 may include the RF circuit 810 and the Wi-Fi module 890; when the terminal 800 is a computer, the terminal 800 may include the communication interface 880 and the Wi-Fi module 890; when the terminal 800 is a tablet computer, the terminal 800 may include the Wi-Fi module.
[0180] The memory 840 can be used to store software programs and modules. The processor 830 executes various functional applications and data processing of the terminal 800 by running the software programs and modules stored in the memory 840. Furthermore, when the processor 830 executes the program code in the memory 840, it can implement the embodiments of this disclosure. Figure 7 , Figure 8 Part or all of the process.
[0181] Optionally, the memory 840 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, various application programs (such as communication applications), and a pose determination module, etc.; the data storage area may store data created based on the use of the terminal (such as various images, video files, and other multimedia files, as well as pose information templates, etc.).
[0182] In addition, the memory 840 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0183] The input unit 850 can be used to receive numeric or character information input by the user, and to generate key signal inputs related to user settings and function control of the terminal 800.
[0184] Optionally, the input unit 850 may include a touch panel 851 and other input terminals 852.
[0185] The touch panel 851, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 851), and drive corresponding connection devices according to a pre-set program. Optionally, the touch panel 851 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 830, and can also receive and execute commands from the processor 830. Furthermore, the touch panel 851 can be implemented using various types of touch technologies, such as resistive, capacitive, infrared, and surface acoustic wave.
[0186] Optionally, the other input terminal 852 may include, but is not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.
[0187] The display unit 860 can be used to display information input by the user or information provided to the user, as well as various menus of the terminal 800. The display unit 860 is the display system of the terminal 800, used to present the interface and realize human-computer interaction.
[0188] The display unit 860 may include a display panel 861. Optionally, the display panel 861 may be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.
[0189] Furthermore, the touch panel 851 may cover the display panel 861. When the touch panel 851 detects a touch operation on or near it, it transmits the information to the processor 830 to determine the type of touch event. Subsequently, the processor 830 provides corresponding visual output on the display panel 861 according to the type of touch event.
[0190] Although Figure 8 In this embodiment, the touch panel 851 and the display panel 861 are two independent components to realize the input and output functions of the terminal 800. However, in some embodiments, the touch panel 851 and the display panel 861 can be integrated to realize the input and output functions of the terminal 800.
[0191] The processor 830 is the control center of the terminal 800. It connects various components through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 840, and calling data stored in the memory 840, it performs various functions of the terminal 800 and processes data, thereby realizing various services based on the terminal.
[0192] Optionally, the processor 830 may include one or more processing units. Optionally, the processor 830 may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 830.
[0193] The camera 870 is used to enable the terminal 800 to capture images or videos. The camera 870 can also be used to enable the terminal 800 to scan objects (QR codes / barcodes).
[0194] The terminal 800 also includes a power supply 830 (such as a battery) for supplying power to various components. Optionally, the power supply 830 can be logically connected to the processor 830 through a power management system, thereby enabling the power management system to manage functions such as charging, discharging, and power consumption.
[0195] It should be noted that the processor 830 in this embodiment can execute... Figure 7 The processor 710 has functions, and the memory 840 stores the contents of the processor 710.
[0196] In addition, in an exemplary embodiment, this disclosure also provides a storage medium, which, when the instructions in the storage medium are executed by the processor of the pose determination device, enables the pose determination device to implement the pose determination method in the embodiments of this disclosure.
[0197] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0198] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A method of pose determination, the method comprising: The method comprises: obtaining an initial pose of a target object, a first video frame containing the target object, and a second video frame containing the target object, the first video frame being a previous video frame adjacent to the second video frame, the second video frame being a current video frame, and the initial pose being determined based on the first video frame; performing key point positioning processing on the target object in the first video frame and the second video frame respectively, and obtaining two groups of key points of the target object correspondingly; determining a feature region according to each group of key points, the feature region being a region without significant deformation; performing feature point extraction processing in the feature region, and obtaining a plurality of feature points corresponding to the first video frame and the second video frame respectively, wherein the feature points are used to represent the pose features of the target object in the feature region, and further obtaining first coordinate information of the target object in the first video frame and second coordinate information of the target object in the second video frame; determining shape data of the target object based on the initial pose; determining a first pose conversion result of the target object in the first video frame relative to the initial pose based on the first coordinate information and the shape data; determining the conversion relationship based on the first coordinate information and the first pose conversion result; determining a second pose conversion result of the target object in the second video frame relative to the initial pose based on the first coordinate information, the second coordinate information, and the first pose conversion result; determining a target pose of the target object in the second video frame based on the second pose conversion result and the shape data.
2. The method of claim 1, wherein, The method further comprises: performing pose positioning processing on the target object in the first video frame based on the first coordinate information and the shape data, and obtaining the first pose conversion result of the target object in the first video frame relative to the initial pose. The method further comprises: performing pose positioning processing on the target object in the second video frame based on the second coordinate information and the shape data, and obtaining the second pose conversion result of the target object in the second video frame relative to the first pose conversion result.
3. The method of claim 1, wherein, The method further comprises: reprojecting the second coordinate information to the first video frame, and correcting the second pose conversion result based on the reprojecting result.
4. A pose determination apparatus, characterized in that The method comprises: An acquisition unit is configured to acquire an initial pose of a target object, a first video frame containing the target object, and a second video frame containing the target object, the first video frame being a previous video frame adjacent to the second video frame, the second video frame being a current video frame, and the initial pose being determined based on the first video frame; An extraction unit is configured to respectively perform key point positioning processing on the target object in the first video frame and the second video frame, to obtain two groups of key points of the target object; determine a feature region based on each group of key points, the feature region being a region without significant deformation; and perform feature point extraction processing in the feature region, to respectively obtain a plurality of feature points corresponding to the first video frame and the second video frame, wherein the feature points are used to represent the pose feature of the target object in the feature region, and further obtain first coordinate information of the target object in the first video frame and second coordinate information of the target object in the second video frame; A first determination unit is configured to determine shape data of the target object based on the initial pose, determine a first pose conversion result of the target object in the first video frame relative to the initial pose based on the first coordinate information and the shape data, and determine the conversion relationship based on the first coordinate information and the first pose conversion result. A second determination unit is configured to determine a second pose conversion result of the target object in the second video frame relative to the initial pose based on the first coordinate information, the second coordinate information, and the first pose conversion result, and determine a target pose of the target object in the second video frame based on the second pose conversion result and the shape data.
5. The apparatus of claim 4, wherein, The first determination unit is specifically configured to perform: pose positioning processing on the target object in the first video frame based on the first coordinate information and the shape data, to obtain a first pose conversion result of the target object in the first video frame relative to the initial pose. The second determination unit is specifically configured to perform: pose positioning processing on the target object in the second video frame based on the second coordinate information and the shape data, to obtain a second pose conversion result of the target object in the second video frame relative to the first pose conversion result.
6. The apparatus of claim 4, wherein, The second determination unit is further configured to perform: reprojecting the second coordinate information to the first video frame and correcting the second pose conversion result based on the reprojecting result.
7. A pose determination electronic device, comprising: comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the pose determination method of any one of claims 1 to 3.
8. A storage medium, when instructions in the storage medium are executed by a processor of a pose determination electronic device, enabling the pose determination electronic device to perform the pose determination method of any one of claims 1 to 3.
9. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the pose determination method according to any one of claims 1 to 3.
Citation Information
Patent Citations
Pose determination method and device, electronic equipment and storage medium
CN112669381A