Pose estimation method and apparatus, electronic device, and storage medium

Abstract

This invention discloses a pose estimation method, apparatus, electronic device, and storage medium, belonging to the field of computer vision technology. The pose estimation method includes: acquiring two-dimensional images from multiple views of the same user; extracting features from the two-dimensional images in each view to obtain image features corresponding to each view; locating two-dimensional human keypoints based on the image features to obtain two-dimensional human keypoints corresponding to each view; predicting three-dimensional human keypoints based on the two-dimensional human keypoints; using the predicted three-dimensional human keypoints as pseudo-labels to establish a correspondence between two-dimensional points in multiple views; performing feature weighting and fusion on the image features corresponding to the two-dimensional images in each view based on the correspondence between the two-dimensional points to obtain fused features; using the fused features as image features for the next iteration; and after multiple iterations, obtaining the user's three-dimensional pose estimation result.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a pose estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, human pose estimation has gradually become a hot research topic, with broad application prospects in fields such as human-computer interaction, autonomous driving, behavior analysis, and action recognition. Most existing 3D human pose estimation methods only use images acquired by a monocular camera. Monocular images suffer from problems such as occlusion, angular deviation, and depth uncertainty, making it difficult to estimate human key points. A few methods use multi-view images from multiple cameras to supplement 3D information by using images of the same person from multiple angles, which can eliminate the influence of occlusion and overcome some of the problems of monocular methods. However, images from each camera are processed separately, without considering the relationship between multi-view images, affecting the accuracy of human pose estimation. Summary of the Invention

[0003] This invention provides a pose estimation method, apparatus, electronic device, and storage medium to address the problem of insufficient accuracy in current 3D human pose estimation.

[0004] This invention provides an attitude estimation method, comprising:

[0005] Acquire two-dimensional images from multiple views of the same user, extract features from the two-dimensional images in each view, and obtain the image features corresponding to the two-dimensional images in each view;

[0006] Based on the image features, two-dimensional human body key point localization is performed to obtain the two-dimensional human body key points corresponding to the two-dimensional image under each view.

[0007] Based on the two-dimensional human body key points, the three-dimensional human body key points are predicted;

[0008] The predicted 3D human body key points are used as pseudo-labels to establish the correspondence between 2D points in the multiple views.

[0009] Based on the correspondence of two-dimensional points under the multiple views, the image features corresponding to the two-dimensional images under each view are weighted and fused to obtain the fused features.

[0010] The fused features are used as image features for the next iteration. After multiple iterations, the user's 3D pose estimation result is obtained.

[0011] In some embodiments, the step of extracting features from the two-dimensional image under each view to obtain the image features corresponding to the two-dimensional image under each view includes:

[0012] Divide the 2D image under each view into blocks to obtain multiple image blocks corresponding to the 2D image under each view;

[0013] Feature extraction is performed on each image block using a multi-stage cascaded feature extraction network;

[0014] Each stage of the feature extraction network includes an embedding layer and a feature extraction module. The embedding layer is used to introduce a position vector for each image patch. The feature extraction module includes two identical parts, each of which includes two normalization layers, a multi-head self-attention mechanism module, and a multilayer perceptron.

[0015] In some embodiments, the step of locating two-dimensional human key points based on the image features to obtain the two-dimensional human key points corresponding to the two-dimensional image in each view includes:

[0016] Based on the image features, a two-dimensional pose detector is used to locate two-dimensional human key points, and the two-dimensional human key points corresponding to the two-dimensional image under each view are obtained.

[0017] The two-dimensional attitude detector is used to perform multiple downsampling, upsampling, and feature map merging operations.

[0018] In some embodiments, predicting three-dimensional human key points based on the two-dimensional human key points includes:

[0019] Based on the aforementioned two-dimensional human body key points, three-dimensional human body key points are predicted using linear triangulation.

[0020] In some embodiments, the step of using the predicted 3D human body key points as pseudo-labels to establish the correspondence between 2D points in the multiple views includes:

[0021] Based on the predicted 3D human key points, obtain the points on the skeleton connecting the 3D human key points.

[0022] The three-dimensional human body key points and the points on the skeleton connecting the three-dimensional human body key points are respectively mapped to the two-dimensional coordinate system to obtain the two-dimensional position coordinates of each three-dimensional point in multiple views. The three-dimensional point is the three-dimensional human body key point or the point on the skeleton connecting the three-dimensional human body key points.

[0023] Based on the two-dimensional position coordinates of each three-dimensional point in multiple views, the corresponding points of each two-dimensional point in multiple views are obtained, and the correspondence between the two-dimensional points in multiple views is established based on the corresponding points of each two-dimensional point in multiple views.

[0024] Wherein, the two-dimensional point is a point determined based on the two-dimensional position coordinates.

[0025] In some embodiments, the step of performing feature weighted fusion on the image features corresponding to the two-dimensional images in each of the multiple views based on the correspondence between two-dimensional points in the multiple views to obtain the fused features includes:

[0026] For a two-dimensional point in a view, calculate the similarity between the features of the corresponding point of the two-dimensional point and the features of the two-dimensional point;

[0027] The weighted sum of the features of the corresponding points of the two-dimensional point is calculated based on the similarity to obtain the features of the approximate fitted corresponding points.

[0028] Based on the features of the two-dimensional points and the features of the corresponding points of the approximate fitting, the fused features are obtained.

[0029] The present invention also provides an attitude estimation device, comprising:

[0030] The feature extraction module is used to acquire two-dimensional images from multiple views of the same user, extract features from the two-dimensional images in each view, and obtain the image features corresponding to the two-dimensional images in each view.

[0031] The two-dimensional positioning module is used to locate two-dimensional human key points based on the image features, and obtain the two-dimensional human key points corresponding to the two-dimensional image under each view.

[0032] A 3D prediction module is used to predict 3D human key points based on the 2D human key points.

[0033] The relationship establishment module is used to establish the correspondence between two-dimensional points under the multiple views by using the predicted three-dimensional human body key points as pseudo-labels.

[0034] The feature fusion module is used to perform feature fusion on the image features corresponding to the two-dimensional images under each of the multiple views based on the correspondence between two-dimensional points under the multiple views, so as to obtain the fused features.

[0035] The output module is used to take the fused features as the image features for the next iteration. After multiple iterations, the three-dimensional pose estimation result of the user is obtained.

[0036] In some embodiments, the step of extracting features from the two-dimensional image under each view to obtain the image features corresponding to the two-dimensional image under each view includes:

[0037] Divide the 2D image under each view into blocks to obtain multiple image blocks corresponding to the 2D image under each view;

[0038] Feature extraction is performed on each image block using a multi-stage cascaded feature extraction network;

[0039] Each stage of the feature extraction network includes an embedding layer and a feature extraction module. The embedding layer is used to introduce a position vector for each image patch. The feature extraction module includes two identical parts, each of which includes two normalization layers, a multi-head self-attention mechanism module, and a multilayer perceptron.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the attitude estimation methods described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the attitude estimation method as described above.

[0042] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described attitude estimation methods.

[0043] The pose estimation method, apparatus, electronic device, and storage medium provided by this invention extract features from two-dimensional images under multiple views of the same user to obtain corresponding image features. Based on the image features, two-dimensional human key points corresponding to the two-dimensional images under each view are obtained, thereby predicting three-dimensional human key points. The obtained three-dimensional human key points are used as pseudo-labels to establish the correspondence between two-dimensional points under multiple views. The image features corresponding to the two-dimensional images under each view are then fused with feature weighting. After multiple iterations, the three-dimensional pose estimation result is obtained. This invention fully considers the correlation between multiple views and improves the accuracy of three-dimensional pose estimation by establishing correspondence and performing feature fusion. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 A schematic flowchart of the attitude estimation method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the structure of the Swing Transformer provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of key points of a two-dimensional human body provided in an embodiment of the present invention;

[0048] Figure 4 A flowchart illustrating the process of establishing the corresponding positional relationship of two-dimensional points using epipolar geometry in the existing technology;

[0049] Figure 5 This is a flowchart illustrating the process of establishing the corresponding positional relationship of two-dimensional points based on three-dimensional points, as provided in an embodiment of the present invention.

[0050] Figure 6 A schematic diagram illustrating the principle of the attitude estimation method provided in this embodiment of the invention;

[0051] Figure 7 This is a schematic diagram of the attitude estimation device provided in an embodiment of the present invention;

[0052] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0055] Human pose estimation involves depicting human posture in images, videos, and other media, estimating key points. Based on the difference in the dimensions of human pose, human pose estimation can be divided into 2D and 3D human pose estimation. 2D human pose estimation locates and identifies key points and connects them sequentially according to joints to form a projection on a two-dimensional plane of the image. However, 2D human pose estimation cannot describe the spatial relationships and three-dimensional attributes of the human body. 3D human pose estimation, on the other hand, detects the three-dimensional coordinates of key points, representing human posture in three-dimensional space. Its application prospects and research value far exceed those of 2D human pose estimation. 3D human pose estimation can be used for tasks such as human-computer interaction, behavior analysis, and action recognition. For example, by adding a camera, it can detect user posture in real time, assessing postures such as when watching TV, attending class, writing, or observing etiquette. It can also be used to judge movements in full-body sports activities.

[0056] Current 3D human pose estimation methods can be broadly classified into two categories: monocular 3D human pose estimation and multi-view 3D human pose estimation.

[0057] Monocular methods use images from a monocular camera to directly predict 3D human keypoint coordinates or map 2D keypoint coordinates to 3D using a deep neural network. Monocular methods only consider images from a monocular camera and do not consider the relationships between multiple view images. Their advantage lies in the ease and unrestricted availability of monocular images, allowing for training with large amounts of data to obtain relatively accurate predictions. Their disadvantages include the existence of self-occlusion, object occlusion, and depth uncertainty in monocular images, with the lack of 3D information affecting the accuracy of the final 3D coordinates.

[0058] Multi-view methods use multiple cameras to capture images of the same human body from different angles, overcoming occlusion issues and fusing multi-view information within the network. Currently, most of these methods process images from multiple angles separately, failing to consider feature correlations across multiple views, neglecting feature fusion, and not utilizing auxiliary information between viewpoints. The key to multi-view methods lies in determining the positional relationship of the same point in a scene across different viewpoints. For a point in one view, current multi-view methods use epipolar geometry, locating the corresponding point in another view on an epipolar line and representing the corresponding point using the entire epipolar line. However, only one point on the epipolar line is the true corresponding point, thus introducing a lot of irrelevant interference. Furthermore, traditional Convolutional Neural Networks (CNN) structures, because they can only learn pixel relationships within local convolutional kernels, struggle to model the correlations between multiple views.

[0059] To address this, the present invention provides a pose estimation method, apparatus, electronic device, and storage medium. By extracting features from two-dimensional images viewed by the same user across multiple views, corresponding image features are obtained. Based on these image features, two-dimensional human keypoints corresponding to the two-dimensional images in each view are obtained, thereby predicting three-dimensional human keypoints. These three-dimensional human keypoints are used as pseudo-labels to establish a correspondence between two-dimensional points across multiple views. Furthermore, the image features corresponding to the two-dimensional images in each view are weighted and fused. After multiple iterations, the three-dimensional pose estimation result is obtained. This invention fully considers the correlation between multiple views and improves the accuracy of three-dimensional pose estimation by establishing correspondences and feature fusion.

[0060] Figure 1 This is a schematic flowchart illustrating the attitude estimation method provided in an embodiment of the present invention. Figure 1 As shown, an attitude estimation method is provided, including the following steps: step 110, step 120, step 130, step 140, step 150, and step 160. This method flowchart is merely one possible implementation of the present invention.

[0061] Step 110: Obtain two-dimensional images from multiple views of the same user, extract features from the two-dimensional images in each view, and obtain the image features corresponding to the two-dimensional images in each view.

[0062] Two-dimensional images refer to planar images that do not contain depth information. They can be obtained by taking pictures with devices such as cameras, video cameras, and mobile phones.

[0063] Specifically, two-dimensional images under multiple views of the same user refer to two-dimensional images taken from multiple angles for the same user. They can be two-dimensional images under multiple views of the same user at a certain point in time during a dynamic process, or two-dimensional images under multiple views of the same user at a certain time period during a static process.

[0064] Specifically, two-dimensional images under multiple views can be acquired through terminal devices, such as mobile phones, personal computers, and tablets.

[0065] It should be explained that by acquiring two-dimensional images from multiple views of the same user, problems such as occlusion, angle deviation, and uncertainty in depth can be overcome.

[0066] In some embodiments, feature extraction is performed on the two-dimensional image under each view to obtain the image features corresponding to the two-dimensional image under each view, including steps 111 and 112. This method flow is merely one possible implementation of the present invention.

[0067] Step 111: Divide the two-dimensional image under each view into blocks to obtain multiple image blocks corresponding to the two-dimensional image under each view;

[0068] Step 112: Perform feature extraction on each image block using a multi-stage cascaded feature extraction network.

[0069] Each stage of the feature extraction network includes an embedding layer and a feature extraction module. The embedding layer is used to introduce a position vector for each image patch. The feature extraction module consists of two identical parts, each of which includes two normalization layers, a multi-head self-attention mechanism module, and a multilayer perceptron.

[0070] Specifically, the feature extraction module is the Swing Transformer block.

[0071] Figure 2 This is a schematic diagram of the structure of the Swing Transformer provided in an embodiment of the present invention.

[0072] The main technological innovation of Swin Transformer lies in its use of localization and shifted windows. Swin Transformer employs different downsampling scales, from shallow to deep layers, with the granularity of feature extraction continuously increasing. Swin Transformer uses non-overlapping windows for self-attention computation. While this non-overlapping window approach performs localized self-attention computation within the feature map window at each scale, the lack of inter-scale interaction due to the consistently localized computation means it also incorporates shifted windows at different levels. This not only includes window self-attention mechanisms but also shifted window self-attention mechanisms, significantly reducing computational complexity.

[0073] Swin Transformer provides a more general backbone network for Transformer-based computer vision tasks and can be applied to a variety of computer vision tasks, such as image classification, object detection, semantic segmentation, and instance segmentation. In some aspects, its performance even surpasses that of traditional CNNs.

[0074] Specifically, at each stage, each image patch is vectorized through the embedding layer to obtain the embedding vector corresponding to each image patch. The embedding vector corresponding to each image patch is then input into the Swin Transformer block to obtain the output features of the Swin Transformer block.

[0075] The embedding vector for each image patch includes the location vector for each image patch.

[0076] Furthermore, by sequentially passing through multiple cascaded embedding layers and Swing Transformer blocks, image features corresponding to each image block are obtained. Based on the image features corresponding to each image block, the image features corresponding to the two-dimensional image under each view are finally obtained.

[0077] It is understandable that a two-dimensional image {I1,I2,...,I...} is generated based on N views from the same user. N}, using the same Swin Transformer to generate image features {f1,f2,...,f N Compared to traditional CNN networks, the Swing Transformer has a stronger feature extraction capability. Image features generated by the Swing Transformer can be used for subsequent keypoint estimation tasks, and can better establish the correspondence between two-dimensional images in multiple views.

[0078] Step 120: Based on the image features, perform two-dimensional human body key point localization to obtain the two-dimensional human body key points corresponding to the two-dimensional image in each view.

[0079] Figure 3 This is a schematic diagram of key points of a two-dimensional human body provided for an embodiment of the present invention. Figure 3 As shown, the two-dimensional human key points include 16 skeletal points: “0hip”, “1r-hip”, “2r-knee”, “3r-foot”, “4l-hip”, “5l-knee”, “6l-foot”, “7spine”, “8thorax”, “9neck”, “10head”, “11l-shoulder”, “12l-elbow”, “13l-wrist”, “14r-shoulder”, “15r-elbow”, and “16r-wrist”.

[0080] In some embodiments, based on the image features, a two-dimensional pose detector is used to locate two-dimensional human key points, thereby obtaining two-dimensional human key points corresponding to the two-dimensional image in each view.

[0081] The two-dimensional attitude detector is used to perform multiple downsampling, upsampling, and feature map merging operations.

[0082] Specifically, the original image features or fused image features are input into a two-dimensional pose detector to regress multiple heatmaps, thereby generating two-dimensional human key points. Where K represents K 2D human body key points, N represents N views, and i represents the i-th view. This represents the two-dimensional coordinates of the j-th two-dimensional human keypoint in the two-dimensional image of the i-th view.

[0083] Step 130: Based on the two-dimensional human body key points, predict the three-dimensional human body key points.

[0084] In some embodiments, based on the two-dimensional human body key points, the three-dimensional human body key points are predicted using linear triangulation.

[0085] Specifically, the projection matrices {M1, M2, ..., M} of the N cameras corresponding to the two-dimensional images of the N views are... N}, determined by the camera's intrinsic parameter P 1 and camera external parameters P 2 Dot product composition:

[0086]

[0087] Among them, P 1 f in x f y c x and C y These are parameters provided by the camera; camera intrinsic parameter P. 1 Its function is to transform from the camera coordinate system to the pixel coordinate system, P 2 In this context, R and t are the parameters measured during dataset construction, where R is the rotation matrix, t is the translation vector, and P is the camera extrinsic parameter. 2 Its function is to transform from the world coordinate system to the camera coordinate system, M i Let represent the projection matrix of the i-th camera corresponding to the 2D image of the i-th view. This represents the camera intrinsic parameters corresponding to the 2D image of the i-th view. This represents the camera extrinsic parameters corresponding to the 2D image of the i-th view.

[0088] It needs to be explained that the essential process of camera imaging is the transformation from three-dimensional spatial coordinates to two-dimensional image coordinates, i.e., a projection process. The camera matrix establishes this three-dimensional to two-dimensional projection relationship.

[0089] Furthermore, based on the j-th two-dimensional human keypoint of the two-dimensional image with N views Estimate the j-th 3D human keypoint d j The calculation formula is as follows:

[0090]

[0091] The projection matrix M of the 2D image of the ith view onto the ith camera. i Decompose into row vectors Will and the vector representation of the j-th two-dimensional human keypoint in the 2D image of the i-th view. Substituting into the above equation and rearranging, we get:

[0092]

[0093] Find the j-th 2D human key points in N views of a 2D image. And the projection matrices {M1,M2,...,M} of the N cameras corresponding to the two-dimensional images of the N views. N Substituting into the above equation, we get:

[0094]

[0095] Based on K two-dimensional human body key points, singular value decomposition of matrix A yields K three-dimensional human body key points D = {d 1 ,d 2 ,...,d K}, where d j =(x j ,y j ,z j ) represents the three-dimensional coordinates of the j-th three-dimensional human body key point.

[0096] Step 140: Use the predicted 3D human body key points as pseudo-labels to establish the correspondence between 2D points in the multiple views.

[0097] It is understood that pseudo-labels refer to labels that are not real labels. In this embodiment of the invention, the predicted three-dimensional human body key points are used as pseudo-labels to establish the correspondence between two-dimensional points in multiple views, thereby improving the accuracy of pose estimation.

[0098] Specifically, step 140 includes steps 141, 142 and 143.

[0099] Step 141: Based on the predicted 3D human body key points, obtain the points on the skeleton connecting the 3D human body key points;

[0100] Understandably, based on the predicted 3D human key points, sampling can be used to obtain the points on the skeleton connecting the 3D human key points.

[0101] Step 142: Map the three-dimensional human key points and the points on the skeleton connecting the three-dimensional human key points to a two-dimensional coordinate system to obtain the two-dimensional position coordinates of each three-dimensional point in multiple views. The three-dimensional point is the three-dimensional human key point or the point on the skeleton connecting the three-dimensional human key points.

[0102] Specifically, the predicted 3D human key points and the points on the connecting skeletons are set as 3D points D. ′ ={d 1 ,d 2 ,...,d H}, where H represents H three-dimensional points, H≥K, and the projection matrix {M1,M2,...,M} is used with N cameras. N Map the H three-dimensional points to the two-dimensional coordinate systems corresponding to the N views of the two-dimensional image:

[0103]

[0104] It is understandable that on a two-dimensional image with N views, in addition to calculating the two-dimensional position coordinates of K two-dimensional human key points, the two-dimensional position coordinates of points on the skeletal connection line are also calculated. The number of these points is far greater than the number of two-dimensional human key points, which facilitates the subsequent establishment of the correspondence between multiple two-dimensional points.

[0105] Step 143: Based on the two-dimensional position coordinates of each three-dimensional point in multiple views, obtain the corresponding points of each two-dimensional point in multiple views, and establish the correspondence relationship of the two-dimensional points in multiple views based on the corresponding points of each two-dimensional point in multiple views.

[0106] Wherein, the two-dimensional point is a point determined based on the two-dimensional position coordinates.

[0107] Specifically, the N two-dimensional points obtained by mapping the same three-dimensional point represent the two-dimensional position coordinates of the same point in three-dimensional space in N views. In fact, they represent the same point, so they can be corresponding points to each other. In this way, this embodiment finds corresponding points in other views for H two-dimensional points in each view, thereby establishing corresponding positional relationships. Among them, for two-dimensional points... Its corresponding set of points is

[0108] Figure 4 A flowchart illustrating the process of establishing the corresponding positional relationship of two-dimensional points using epipolar geometry, provided for existing technologies. Figure 5 This is a flowchart illustrating the process of establishing the corresponding positional relationship of two-dimensional points based on three-dimensional points, as provided in an embodiment of the present invention. It is understood that for a two-dimensional point, since the coordinates of its corresponding three-dimensional point are unknown, the corresponding two-dimensional point in other views is difficult to locate. Existing technologies can only locate the corresponding point on an epipolar line, while there is only one true point, which introduces irrelevant information. However, this embodiment of the present invention utilizes predicted three-dimensional human body keypoints as pseudo-labels, which can more accurately locate the corresponding point, thereby improving the accuracy of pose estimation.

[0109] Step 150: Based on the correspondence between two-dimensional points under the multiple views, perform feature weighted fusion on the image features corresponding to the two-dimensional images under each view to obtain the fused features.

[0110] It should be noted that the embodiments of the present invention utilize the correspondence between two-dimensional points under multiple views to fuse the image features corresponding to the two-dimensional images of other views into the image features corresponding to the two-dimensional image of a single view, thereby obtaining fused features, which facilitates the introduction of three-dimensional information and the estimation of three-dimensional human pose.

[0111] Specifically, step 150 includes the following steps:

[0112] Step 151: For a two-dimensional point in a view, calculate the similarity between the features of the corresponding point of the two-dimensional point and the features of the two-dimensional point;

[0113] Step 152: Calculate the weighted sum of the features of the corresponding points of the two-dimensional point based on the similarity to obtain the features of the approximate fitted corresponding points;

[0114] Specifically, for a two-dimensional point in a certain view Its corresponding set of points is:

[0115]

[0116] The aforementioned set of corresponding points can be obtained from the correspondence between two-dimensional points in multiple views.

[0117] The features extracted using the same Swin Transformer based on the corresponding point set P of two-dimensional points are:

[0118]

[0119] This embodiment calculates a weighted sum of the features of corresponding points based on the similarity between the features of the corresponding points of the two-dimensional points and the features of the two-dimensional points themselves:

[0120]

[0121] The sim(·) operation is used to calculate the similarity between the features of the corresponding point of the two-dimensional point and the features of the two-dimensional point. These are the features of the corresponding points in the approximate fit.

[0122] Step 153: Based on the features of the two-dimensional points and the features of the corresponding points of the approximate fitting, obtain the fused features.

[0123] Specifically, the fused features are calculated based on the features of the two-dimensional points and the features of the corresponding points of the approximate fit:

[0124]

[0125] Where W is a 1x1 matrix parameter, It is a two-dimensional point Features after fusion.

[0126] Step 160: Use the fused features as the image features for the next iteration. After multiple iterations, obtain the user's 3D pose estimation result.

[0127] Specifically, based on the fused features, the process begins to iteratively execute 2D human keypoint localization, 3D human keypoint prediction, establishment of correspondence between 2D points in multiple views, and feature fusion. That is, the fused features are used as image features for the next iteration, and steps 120 to 150 are re-executed. Through multiple iterations, the accuracy of 3D human pose estimation can be gradually improved. After multiple iterations, the output of step 130 is finally obtained, resulting in a refined 3D pose estimation result.

[0128] Figure 6 This is a block diagram illustrating the principle of the attitude estimation method provided in an embodiment of the present invention.

[0129] In this embodiment of the invention, feature extraction is performed on two-dimensional images under multiple views of the same user to obtain corresponding image features. Based on the image features, two-dimensional human key points corresponding to the two-dimensional images under each view are obtained, thereby predicting three-dimensional human key points. The obtained three-dimensional human key points are used as pseudo-labels to establish the correspondence between two-dimensional points under multiple views. Feature weighted fusion is performed on the image features corresponding to the two-dimensional images under each view. After multiple iterations, the three-dimensional pose estimation result is obtained. This invention fully considers the correlation between multiple views. By establishing correspondence and performing feature fusion, the accuracy of three-dimensional pose estimation is improved. It has strong interpretability and can be widely deployed.

[0130] It should be noted that each embodiment of the present invention can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0131] The attitude estimation device provided in the embodiments of the present invention is described below. The device described below can be referred to in correspondence with the attitude estimation method described above.

[0132] Figure 7 This is a schematic diagram of the attitude estimation device provided by the present invention, as shown below. Figure 7 As shown, the device includes:

[0133] The feature extraction module 710 is used to acquire two-dimensional images under multiple views of the same user, perform feature extraction on the two-dimensional images under each view, and obtain the image features corresponding to the two-dimensional images under each view.

[0134] The two-dimensional positioning module 720 is used to locate two-dimensional human body key points based on the image features, and obtain the two-dimensional human body key points corresponding to the two-dimensional image under each view.

[0135] The three-dimensional prediction module 730 is used to predict three-dimensional human key points based on the two-dimensional human key points.

[0136] The relationship establishment module 740 is used to establish the correspondence between two-dimensional points under the multiple views by using the predicted three-dimensional human body key points as pseudo-labels.

[0137] The feature fusion module 750 is used to perform feature fusion on the image features corresponding to the two-dimensional images under each view based on the correspondence of two-dimensional points under the multiple views, so as to obtain the fused features.

[0138] The output module 760 is used to use the fused features as image features for the next iteration. After multiple iterations, the three-dimensional pose estimation result of the user is obtained.

[0139] Optionally, the step of extracting features from the two-dimensional image under each view to obtain the image features corresponding to the two-dimensional image under each view includes:

[0140] Divide the 2D image under each view into blocks to obtain multiple image blocks corresponding to the 2D image under each view;

[0141] Feature extraction is performed on each image block using a multi-stage cascaded feature extraction network;

[0142] Each stage of the feature extraction network includes an embedding layer and a feature extraction module. The embedding layer is used to introduce a position vector for each image patch. The feature extraction module includes two identical parts, each of which includes two normalization layers, a multi-head self-attention mechanism module, and a multilayer perceptron.

[0143] Optionally, the step of locating two-dimensional human key points based on the image features to obtain the two-dimensional human key points corresponding to the two-dimensional image in each view includes:

[0144] Based on the image features, a two-dimensional pose detector is used to locate two-dimensional human key points, and the two-dimensional human key points corresponding to the two-dimensional image under each view are obtained.

[0145] The two-dimensional attitude detector is used to perform multiple downsampling, upsampling, and feature map merging operations.

[0146] Optionally, the step of predicting three-dimensional human key points based on the two-dimensional human key points includes:

[0147] Based on the aforementioned two-dimensional human body key points, three-dimensional human body key points are predicted using linear triangulation.

[0148] Optionally, the step of using the predicted 3D human body key points as pseudo-labels to establish the correspondence between 2D points in the multiple views includes:

[0149] Based on the predicted 3D human key points, obtain the points on the skeleton connecting the 3D human key points.

[0150] The three-dimensional human body key points and the points on the skeleton connecting the three-dimensional human body key points are respectively mapped to the two-dimensional coordinate system to obtain the two-dimensional position coordinates of each three-dimensional point in multiple views. The three-dimensional point is the three-dimensional human body key point or the point on the skeleton connecting the three-dimensional human body key points.

[0151] Based on the two-dimensional position coordinates of each three-dimensional point in multiple views, the corresponding points of each two-dimensional point in multiple views are obtained, and the correspondence between the two-dimensional points in multiple views is established based on the corresponding points of each two-dimensional point in multiple views.

[0152] Wherein, the two-dimensional point is a point determined based on the two-dimensional position coordinates.

[0153] Optionally, the step of performing feature weighted fusion on the image features corresponding to the two-dimensional images under each of the multiple views based on the correspondence between the two-dimensional points under the multiple views to obtain the fused features includes:

[0154] For a two-dimensional point in a view, calculate the similarity between the features of the corresponding point of the two-dimensional point and the features of the two-dimensional point;

[0155] The weighted sum of the features of the corresponding points of the two-dimensional point is calculated based on the similarity to obtain the features of the approximate fitted corresponding points.

[0156] Based on the features of the two-dimensional points and the features of the corresponding points of the approximate fitting, the fused features are obtained.

[0157] It should be noted that the attitude estimation device provided in this embodiment of the invention can implement all the method steps implemented in the above attitude estimation method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0158] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a pose estimation method, which includes: acquiring two-dimensional images of the same user in multiple views; extracting features from the two-dimensional images in each view to obtain image features corresponding to the two-dimensional images in each view; locating two-dimensional human keypoints based on the image features to obtain two-dimensional human keypoints corresponding to the two-dimensional images in each view; predicting three-dimensional human keypoints based on the two-dimensional human keypoints; using the predicted three-dimensional human keypoints as pseudo-labels to establish a correspondence between the two-dimensional points in the multiple views; performing feature weighted fusion on the image features corresponding to the two-dimensional images in each view based on the correspondence between the two-dimensional points in the multiple views to obtain fused features; using the fused features as image features for the next iteration, and after multiple iterations, obtaining the three-dimensional pose estimation result of the user.

[0159] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the pose estimation method provided in the above-described method embodiments. The method includes: acquiring two-dimensional images under multiple views of the same user; extracting features from the two-dimensional images under each view to obtain image features corresponding to the two-dimensional images under each view; locating two-dimensional human keypoints based on the image features to obtain two-dimensional human keypoints corresponding to the two-dimensional images under each view; predicting three-dimensional human keypoints based on the two-dimensional human keypoints; using the predicted three-dimensional human keypoints as pseudo-labels to establish a correspondence between the two-dimensional points under the multiple views; performing feature weighted fusion on the image features corresponding to the two-dimensional images under each view based on the correspondence between the two-dimensional points under the multiple views to obtain fused features; using the fused features as image features for the next iteration; and after multiple iterations, obtaining the three-dimensional pose estimation result of the user.

[0161] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the pose estimation method provided in the above-described method embodiments. The method includes: acquiring two-dimensional images from multiple views of the same user; extracting features from the two-dimensional images in each view to obtain image features corresponding to the two-dimensional images in each view; locating two-dimensional human keypoints based on the image features to obtain two-dimensional human keypoints corresponding to the two-dimensional images in each view; predicting three-dimensional human keypoints based on the two-dimensional human keypoints; using the predicted three-dimensional human keypoints as pseudo-labels to establish a correspondence between the two-dimensional points in the multiple views; performing feature weighted fusion on the image features corresponding to the two-dimensional images in each view based on the correspondence between the two-dimensional points in the multiple views to obtain fused features; using the fused features as image features for the next iteration; and after multiple iterations, obtaining the three-dimensional pose estimation result of the user.

[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pose estimation method, characterized in that, include: Acquire two-dimensional images from multiple views of the same user, extract features from the two-dimensional images in each view, and obtain the image features corresponding to the two-dimensional images in each view; Based on the image features, two-dimensional human body key point localization is performed to obtain the two-dimensional human body key points corresponding to the two-dimensional image under each view. Based on the two-dimensional human body key points, the three-dimensional human body key points are predicted; Based on the predicted 3D human key points, obtain the points on the skeleton connecting the 3D human key points. The three-dimensional human body key points and the points on the skeleton connecting the three-dimensional human body key points are respectively mapped to the two-dimensional coordinate system to obtain the two-dimensional position coordinates of each three-dimensional point in multiple views. The three-dimensional point is the three-dimensional human body key point or the point on the skeleton connecting the three-dimensional human body key points. Based on the two-dimensional position coordinates of each three-dimensional point in multiple views, the corresponding points of each two-dimensional point in multiple views are obtained, and the correspondence between the two-dimensional points in multiple views is established based on the corresponding points of each two-dimensional point in multiple views; wherein, the two-dimensional point is a point determined based on the two-dimensional position coordinates. Based on the correspondence of two-dimensional points under the multiple views, the image features corresponding to the two-dimensional images under each view are weighted and fused to obtain the fused features. The fused features are used as image features for the next iteration. After multiple iterations, the user's 3D pose estimation result is obtained.

2. The pose estimation method of claim 1, wherein, The step of extracting features from the two-dimensional image under each view to obtain the image features corresponding to the two-dimensional image under each view includes: Divide the 2D image under each view into blocks to obtain multiple image blocks corresponding to the 2D image under each view; Feature extraction is performed on each image block using a multi-stage cascaded feature extraction network; Each stage of the feature extraction network includes an embedding layer and a feature extraction module. The embedding layer is used to introduce a position vector for each image patch. The feature extraction module includes two identical parts, each of which includes two normalization layers, a multi-head self-attention mechanism module, and a multilayer perceptron.

3. The pose estimation method of claim 1, wherein, The step of locating two-dimensional human key points based on the image features to obtain the two-dimensional human key points corresponding to the two-dimensional image in each view includes: Based on the image features, a two-dimensional pose detector is used to locate two-dimensional human key points, and the two-dimensional human key points corresponding to the two-dimensional image under each view are obtained. The two-dimensional attitude detector is used to perform multiple downsampling, upsampling, and feature map merging operations.

4. The pose estimation method of claim 1, wherein, The prediction of three-dimensional human body key points based on the two-dimensional human body key points includes: Based on the aforementioned two-dimensional human body key points, three-dimensional human body key points are predicted using linear triangulation.

5. The pose estimation method of claim 1, wherein, Based on the correspondence between two-dimensional points in the multiple views, the image features corresponding to the two-dimensional images in each view are weighted and fused to obtain the fused features, including: For a two-dimensional point in a view, calculate the similarity between the features of the corresponding point of the two-dimensional point and the features of the two-dimensional point; The weighted sum of the features of the corresponding points of the two-dimensional point is calculated based on the similarity to obtain the features of the approximate fitted corresponding points. Based on the features of the two-dimensional points and the features of the corresponding points of the approximate fitting, the fused features are obtained.

6. An attitude estimation device characterized by comprising: include: The feature extraction module is used to acquire two-dimensional images from multiple views of the same user, extract features from the two-dimensional images in each view, and obtain the image features corresponding to the two-dimensional images in each view. The two-dimensional positioning module is used to locate two-dimensional human key points based on the image features, and obtain the two-dimensional human key points corresponding to the two-dimensional image under each view. A 3D prediction module is used to predict 3D human key points based on the 2D human key points. The relationship establishment module is used to obtain points on the skeleton connecting the predicted 3D human key points. The three-dimensional human body key points and the points on the skeleton connecting the three-dimensional human body key points are respectively mapped to the two-dimensional coordinate system to obtain the two-dimensional position coordinates of each three-dimensional point in multiple views. The three-dimensional point is the three-dimensional human body key point or the point on the skeleton connecting the three-dimensional human body key points. Based on the two-dimensional position coordinates of each three-dimensional point in multiple views, the corresponding points of each two-dimensional point in multiple views are obtained. Based on the corresponding points of each two-dimensional point in multiple views, the correspondence relationship of the two-dimensional points in the multiple views is established; wherein, the two-dimensional point is the point determined based on the two-dimensional position coordinates. The feature fusion module is used to perform feature fusion on the image features corresponding to the two-dimensional images under each of the multiple views based on the correspondence between two-dimensional points under the multiple views, so as to obtain the fused features. The output module is used to take the fused features as the image features for the next iteration. After multiple iterations, the three-dimensional pose estimation result of the user is obtained.

7. The attitude estimation apparatus according to claim 6, characterized by, The step of extracting features from the two-dimensional image under each view to obtain the image features corresponding to the two-dimensional image under each view includes: Divide the 2D image under each view into blocks to obtain multiple image blocks corresponding to the 2D image under each view; Feature extraction is performed on each image block using a multi-stage cascaded feature extraction network; Each stage of the feature extraction network includes an embedding layer and a feature extraction module. The embedding layer is used to introduce a position vector for each image patch. The feature extraction module includes two identical parts, each of which includes two normalization layers, a multi-head self-attention mechanism module, and a multilayer perceptron.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the attitude estimation method as described in any one of claims 1 to 5. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the attitude estimation method as described in any one of claims 1 to 5.

Images