Three-dimensional attitude estimation method and device, storage medium and electronic equipment

Abstract

The invention discloses a three-dimensional attitude estimation method and device, a storage medium and electronic equipment, and relates to the technical field of three-dimensional attitude estimation, and the method comprises the steps: carrying out the joint estimation of a multi-view image of a target object, and obtaining a joint feature map and a joint heat map of each view; performing cross-view-angle alignment analysis on the multi-view-angle joint heat map to obtain an initial three-dimensional joint coordinate of each joint point; carrying out deformable cross-attention feature aggregation on the multi-view joint feature map to obtain multi-view fusion features of the joints; and performing three-dimensional attitude prediction based on the initial three-dimensional joint coordinates of the joints and the multi-view fusion features to obtain target three-dimensional joint coordinates of the joints. According to the invention, the estimation accuracy of the three-dimensional coordinates of the joint points during three-dimensional attitude estimation can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional pose estimation, in particular to a three-dimensional pose estimation method and device, a storage medium and an electronic device. BACKGROUND

[0002] In three-dimensional pose estimation and the like, the coordinates of the joint nodes of a target object are usually estimated according to an image taken of the target object. For example, in the related art, a front-facing camera of a head-mounted device (HMD) of a human body takes an image containing the human body, and three-dimensional pose estimation is performed according to the image to obtain three-dimensional coordinates (3D coordinates) of the joint nodes of the human body. The estimated coordinates can be further used for three-dimensional reconstruction and the like.

[0003] However, in the related art, when performing three-dimensional pose estimation, an independent 2D joint detector is usually relied on, and information across perspectives cannot be effectively integrated and utilized, resulting in poor accuracy of the three-dimensional coordinates of the estimated joint nodes. SUMMARY

[0004] Embodiments of the present application provide a three-dimensional pose estimation scheme, which can effectively improve the estimation accuracy of the three-dimensional coordinates of the joint nodes.

[0005] Embodiments of the present application provide the following technical solutions: According to an embodiment of the present application, a three-dimensional pose estimation method comprises: performing joint estimation on multi-perspective images of a target object to obtain joint feature maps and joint heat maps of each perspective; performing cross-perspective alignment analysis on the joint heat maps of multiple perspectives to obtain initial three-dimensional joint coordinates of each joint node; performing deformable cross-attention feature aggregation on the joint feature maps of multiple perspectives to obtain multi-perspective fusion features of each joint; and performing three-dimensional pose prediction based on the initial three-dimensional joint coordinates of each joint and the multi-perspective fusion features to obtain target three-dimensional joint coordinates of each joint.

[0006] In some embodiments of the present application, the cross-perspective alignment analysis on the joint heat maps of multiple perspectives to obtain initial three-dimensional joint coordinates of each joint comprises: extracting two-dimensional joint coordinates of each joint under each perspective from the joint heat maps of each perspective; constructing a projection function from three-dimensional to two-dimensional for each joint under each perspective based on camera parameters of each perspective and the two-dimensional joint coordinates of each joint under each perspective; calculating a cross-perspective geometric constraint error according to the two-dimensional joint coordinates of each joint under different perspectives and the projection function; and performing minimum solution on the cross-perspective geometric constraint error to obtain the initial three-dimensional joint coordinates of each joint.

[0007] In some embodiments of the present application, the deformable cross-attention feature aggregation of the multi-view joint feature maps obtains multi-view fusion features of each joint, including: generating a query feature vector of a joint based on the joint feature map of a query view, the query view being one of the multi-views; performing sampling offset prediction based on the query feature vector to obtain a sampling offset of the joint; sampling features from the joint feature maps of each view according to the sampling offset to obtain a sampling feature vector of the joint under each view; and predicting cross-attention weights of the joint under each view based on the query feature vector and the sampling feature vector of the joint under each view; and multiplying and summing the sampling feature vector of the joint under each view and the cross-attention weights to obtain multi-view fusion features of the joint under the query view.

[0008] In some embodiments of the present application, the three-dimensional pose prediction based on the initial three-dimensional joint coordinates of each joint and the multi-view fusion features obtains target three-dimensional joint coordinates of each joint, including: concatenating the multi-view fusion features of each joint under the multi-views to obtain multi-view concatenation features corresponding to each joint; performing fully connected processing on the initial three-dimensional joint coordinates and the multi-view fusion features of each joint to obtain predicted three-dimensional joint coordinates of each joint; and obtaining target three-dimensional joint coordinates of each joint according to the predicted three-dimensional joint coordinates of each joint.

[0009] In some embodiments of the present application, before the target three-dimensional joint coordinates of each joint are obtained according to the predicted three-dimensional joint coordinates of each joint, the method further includes: determining a symmetric joint corresponding to a missing joint of the target object; and performing mirror transformation on the predicted three-dimensional joint coordinates of the symmetric joint to obtain the predicted three-dimensional joint coordinates of the missing joint.

[0010] In some embodiments of the present application, the obtaining of the target three-dimensional joint coordinates of each joint according to the predicted three-dimensional joint coordinates of each joint includes: performing velocity smoothing constraint processing on the predicted three-dimensional joint coordinates of each joint according to the historical three-dimensional joint coordinates of each joint at a previous time to obtain the target three-dimensional joint coordinates of each joint.

[0011] In some embodiments of the present application, the joint estimation of the multi-view images of the target object obtains joint feature maps and joint heat maps of each view, including: performing encoding processing on the images of each view respectively to obtain the joint feature maps of each view; and performing convolution and activation processing on the joint feature maps of each view to obtain the joint heat maps of each view.

[0012] In some embodiments of the present application, before the cross-view alignment analysis of the multi-view joint heat maps is performed to obtain the initial three-dimensional joint coordinates of each joint, the method further comprises: obtaining a confidence threshold; and performing mask processing on the joint heat maps of each view according to the confidence threshold to obtain the joint heat maps containing heat map maximum points greater than or equal to the confidence threshold.

[0013] In some embodiments of the present application, before the joint estimation of the multi-view images of the target object is performed to obtain the joint feature maps and joint heat maps of each view, the method further comprises: receiving multi-view images captured by a first group of fisheye cameras and a second group of fisheye cameras arranged overhead of the target object, wherein the field of view directions of the first group of fisheye cameras and the second group of fisheye cameras are opposite, the field of view overlap area of adjacent fisheye cameras in the first group of fisheye cameras and the second group of fisheye cameras accounts for a predetermined proportion of the total field of view, and the predetermined proportion is greater than a predetermined first threshold and less than a predetermined second threshold.

[0014] According to an embodiment of the present application, a three-dimensional pose estimation device comprises: an estimation module configured to perform joint estimation on multi-view images of a target object to obtain joint feature maps and joint heat maps of each view; an alignment analysis module configured to perform cross-view alignment analysis on the multi-view joint heat maps to obtain initial three-dimensional joint coordinates of each joint; an aggregation module configured to perform deformable cross-attention feature aggregation on the joint feature maps of multiple views to obtain multi-view fusion features of each joint; and a regression module configured to perform three-dimensional pose prediction based on the initial three-dimensional joint coordinates of each joint and the multi-view fusion features to obtain target three-dimensional joint coordinates of each joint.

[0015] According to another embodiment of the present application, a storage medium has a computer program stored thereon, and when the computer program is executed by a processor of an electronic device, the electronic device performs the method described in the embodiments of the present application.

[0016] According to another embodiment of the present application, an electronic device can include a memory storing a computer program, and a processor reading the computer program stored in the memory to execute the method described in the embodiments of the present application.

[0017] According to another embodiment of the present application, a computer program product or computer program includes computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the electronic device perform the method provided in various optional implementations of the embodiments of the present application.

[0018] In the embodiments of the present application, joint estimation is performed on multi-view images of a target object to obtain joint feature maps and joint heat maps of each view; cross-view alignment analysis is performed on the joint heat maps of multiple views to obtain initial three-dimensional joint coordinates of each joint; deformable cross-attention feature aggregation is performed on the joint feature maps of multiple views to obtain multi-view fusion features of each joint; and three-dimensional pose prediction is performed based on the initial three-dimensional joint coordinates and the multi-view fusion features of each joint to obtain target three-dimensional joint coordinates of each joint.

[0019] In this way of the embodiments of the present application, the joint feature maps and joint heat maps of each view are estimated based on multi-view images of a target object, then cross-view alignment analysis is performed on the joint heat maps of multiple views to obtain initial three-dimensional joint coordinates of each joint from the perspective of geometric prior, further, deformable cross-attention feature aggregation is performed on the joint feature maps of multiple views to obtain multi-view fusion features of each joint by deeply fusing the joint features of multiple views across views, and finally, three-dimensional pose prediction is performed in combination with the initial three-dimensional joint coordinates of the geometric prior and the multi-view fusion features aggregated across views, so that the target three-dimensional joint coordinates of each joint can be accurately obtained, and the estimation accuracy of the three-dimensional coordinates of the joints in three-dimensional pose estimation can be effectively improved as a whole. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0021] Figure 1 A flowchart of a three-dimensional pose estimation method according to one embodiment of the present application is shown.

[0022] Figure 2 A cross-view alignment analysis flowchart according to one embodiment of the present application is shown.

[0023] Figure 3 A cross-attention feature aggregation flowchart according to one embodiment of the present application is shown.

[0024] Figure 4 A block diagram of a three-dimensional pose estimation device according to one embodiment of the present application is shown.

[0025] Figure 5 A block diagram of an electronic device according to one embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The present disclosure will be further described in details with reference to the drawings and embodiments. It should be understood that the embodiments provided herein are only used to explain the present disclosure and not intended to limit the present disclosure. In addition, the embodiments provided below are used to implement part of the present disclosure, and the technical solutions described in the embodiments of the present disclosure can be combined in any manner without conflict. It should be noted that in the embodiments of the present disclosure, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the method or device comprising a series of elements not only includes the elements explicitly described, but also includes other elements not explicitly listed or inherent to the method or device. Without more limitation, the element defined by the sentence "comprises a" does not exclude the presence of other related elements (such as steps in the method or units in the device, for example, the unit can be part of the circuit, part of the processor, part of the program or software, etc.) in the method or device comprising the element. For example, the three-dimensional pose estimation method provided by the embodiments of the present disclosure comprises a series of steps, but the three-dimensional pose estimation method provided by the embodiments of the present disclosure is not limited to the steps described, and similarly, the three-dimensional pose estimation device provided by the embodiments of the present disclosure comprises a series of units, but the device provided by the embodiments of the present disclosure is not limited to comprising the units explicitly described, and can also comprise units required to be set when obtaining related information or processing based on information. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. It can be understood that in the specific embodiments of the present application, relevant data is involved, and when the embodiments in the present application are applied to specific products or technologies, the permission or consent of the user needs to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0027] In three-dimensional pose estimation, etc., the coordinates of the joints of the target object are usually estimated according to the images taken for the target object. For example, in related technologies, the image containing the human body is taken by the front camera of the head-mounted device (HMD) of the human body, and the three-dimensional coordinates (3D coordinates) of the joints of the human body are obtained by three-dimensional pose estimation according to the image. The estimated coordinates can be further used for three-dimensional reconstruction, etc. However, in the related technologies at present, when performing three-dimensional pose estimation, it usually depends on an independent 2D joint detector, and cannot effectively integrate and utilize the information across the perspective, resulting in poor accuracy of the three-dimensional coordinates of the estimated joints.

[0028] To address the aforementioned technical problems, this application provides a three-dimensional attitude estimation scheme that can effectively improve the accuracy of estimating the three-dimensional coordinates of joints during three-dimensional attitude estimation. The following provides a detailed description of relevant embodiments of the three-dimensional attitude estimation scheme provided in this application.

[0029] Figure 1 A flowchart illustrating one embodiment of a three-dimensional pose estimation method according to this application is shown. The execution entity of this three-dimensional pose estimation method can be an electronic device and / or a server with processing capabilities. Electronic devices include, for example, computers, mobile phones, smartwatches, televisions, virtual reality devices, and home appliances. Servers include, for example, cloud services or physical servers.

[0030] like Figure 1 As shown, the three-dimensional pose estimation method may include steps S110 to S140.

[0031] Step S110: Perform joint estimation on the multi-view images of the target object to obtain joint feature maps and joint heatmaps for each view. Step S120: Perform cross-view alignment analysis on the multi-view joint heatmaps to obtain the initial three-dimensional joint coordinates of each joint point. Step S130: Perform deformable cross-attention feature aggregation on the multi-view joint feature maps to obtain the multi-view fusion features of each joint. Step S140: Based on the initial three-dimensional joint coordinates of each joint and the multi-view fusion features, perform three-dimensional pose prediction to obtain the target three-dimensional joint coordinates of each joint.

[0032] Multi-view images of a target object are captured using cameras with multiple perspectives (such as fisheye cameras). Each camera can capture a portion of the target object, and each image can contain a portion of the target object. For example, the head of the target object can be equipped with multiple fisheye cameras, thus allowing for the capture of multi-view images of the target object centered on itself.

[0033] Using a preset estimation module, joint estimation can be performed on multi-view images of the target object to obtain joint feature maps and joint heatmaps for each view. The joint feature map is a feature matrix obtained by encoding the image, and the joint heatmap is the probability distribution map of the possible two-dimensional positions (2D joint positions) of each joint in the image.

[0034] Furthermore, by employing a preset alignment analysis module, cross-view alignment analysis can be performed on joint heatmaps from multiple perspectives. This cross-view alignment analysis can initially synthesize the 2D joint positions from multiple perspectives to obtain geometric priors and the initial three-dimensional joint coordinates of each joint point. These initial three-dimensional joint coordinates are the three-dimensional joint coordinates of each joint point calculated and analyzed from the geometric priors through cross-view alignment analysis.

[0035] Furthermore, by employing a pre-defined aggregation module, deformable cross-attention feature aggregation can be performed on the joint feature maps from multiple perspectives. Deformable cross-attention feature aggregation is a feature aggregation algorithm that combines deformable convolution and cross-attention mechanisms. Through deformable cross-attention feature aggregation, joint features dynamically shifted across multiple perspectives can be accurately aggregated to obtain the multi-view fusion features of the joint. For example, with joint feature maps from four perspectives, the multi-view fusion features Ak,j of joint j can be obtained by aggregating them at the k-th perspective.

[0036] Furthermore, by employing a pre-defined regression module, three-dimensional pose prediction can be performed based on the initial three-dimensional joint coordinates of each joint and the multi-view fusion features. That is, by combining the initial three-dimensional joint coordinates of geometric priors and the multi-view fusion features aggregated across different viewpoints, the target three-dimensional joint coordinates of each joint can be more accurately predicted. The target three-dimensional joint coordinates are also the three-dimensional coordinates of the joints that are finally estimated.

[0037] Among them, the preset estimation module, the preset alignment analysis module, and the preset regression module can be modules in the preset 3D pose estimation model, which can be a pre-trained deep learning model.

[0038] In summary, in this embodiment of the application, joint feature maps and joint heatmaps for each viewpoint are estimated based on multi-view images of the target object. Then, cross-view alignment analysis is performed on the multi-view joint heatmaps. From the perspective of geometric prior, the initial three-dimensional joint coordinates of each joint point are obtained by comprehensively analyzing the multi-view analysis. Furthermore, deformable cross-attention feature aggregation is performed on the multi-view joint feature maps, and deep cross-view fusion of the multi-view joint features is performed to obtain the multi-view fusion features of each joint. Finally, the three-dimensional pose prediction is performed by combining the initial three-dimensional joint coordinates of the geometric prior and the cross-view fused multi-view features. The target three-dimensional joint coordinates of each joint can be accurately obtained. Overall, the estimation accuracy of the three-dimensional coordinates of the joint points can be effectively improved when estimating the three-dimensional pose.

[0039] The following description Figure 1 Further optional specific embodiments for each step performed during the three-dimensional pose estimation in the example.

[0040] In one embodiment, step S110, performing joint estimation on the multi-view images of the target object to obtain joint feature maps and joint heatmaps for each viewpoint, may include: encoding the images of each viewpoint to obtain joint feature maps for each viewpoint; and performing convolution and activation processing on the joint feature maps of each viewpoint to obtain joint heatmaps for each viewpoint.

[0041] The preset estimation module may include an encoder and a heatmap prediction unit. In the encoder, images from each viewpoint are encoded separately to obtain joint feature maps F for each viewpoint. Further, in the heatmap prediction unit, convolution and activation processing are performed on the joint feature maps of each viewpoint to obtain joint heatmaps for each viewpoint. For example, the joint heatmap for the k-th viewpoint can be... Among them, G conv To represent the convolutional process performed on the convolutional layer, This indicates that the activation function performs activation processing. The heatmap prediction unit can use cross-entropy loss L... heatmap Constraints, among which, Knuckle thermogram, H gt It is a binary variable, H gt It takes the value {0, 1}.

[0042] See Figure 2 In one embodiment, step S120, performing cross-view alignment analysis on the joint heatmaps from multiple perspectives to obtain the initial three-dimensional joint coordinates of each joint, may include: step S210, extracting the two-dimensional joint coordinates of each joint from the joint heatmaps from each perspective; step S220, constructing the three-dimensional to two-dimensional projection function of each joint from each perspective based on the camera parameters of each perspective and the two-dimensional joint coordinates of each joint from each perspective; step S230, calculating the cross-view geometric constraint error based on the two-dimensional joint coordinates and projection function of each joint from different perspectives; and step S240, minimizing the cross-view geometric constraint error to obtain the initial three-dimensional joint coordinates of each joint.

[0043] The preset estimation module may include a maximum point extraction unit, a projection function construction unit, an error calculation unit, and an optimization unit.

[0044] In the maximum point extraction unit, the maximum point of joint j in the joint heatmap can be extracted from the joint heatmap of the k-th viewpoint. This allows us to obtain the two-dimensional joint coordinates of joint j from the k-th viewpoint. Similarly, the two-dimensional joint coordinates of each joint from each viewpoint can be extracted.

[0045] In the projection function construction unit, the camera parameters Ck (which can include camera intrinsic and extrinsic parameters) from the k-th viewpoint and the two-dimensional joint coordinates of joint j from the k-th viewpoint can be used as the basis. Construct the projection function of joint j from three dimensions to two dimensions from the k-th viewpoint. , where p j 3D Let j be the initial 3D joint coordinates, and let the projection function be... This is used to transform the initial 3D joint coordinates from a 3D plane to a 2D plane. Similarly, projection functions from 3D to 2D can be constructed for each joint from each viewpoint.

[0046] Furthermore, in the error calculation unit, the cross-view geometric constraint error E can be calculated based on the two-dimensional joint coordinates and projection functions of each joint under different viewpoints. For example, between the k1th viewpoint and the k2th viewpoint, ,in, Let be the projection function of joint j from three dimensions to two dimensions from the k1-th viewpoint. Let be the projection function of joint j from three dimensions to two dimensions from the k2th viewpoint. Let be the two-dimensional joint coordinates of joint j from the k1-th viewpoint. Let J be the two-dimensional joint coordinates of joint j from the k2th viewpoint, and J be the total number of joints.

[0047] Furthermore, in the optimization unit, the cross-view geometric constraint error E can be minimized to accurately obtain the initial three-dimensional joint coordinates p of joint j. j 3D Among them, the minimization solution can be obtained using error optimization algorithms such as the LM algorithm.

[0048] Furthermore, in one embodiment, before performing cross-view alignment analysis on the joint heatmaps from multiple perspectives to obtain the initial three-dimensional joint coordinates of each joint point, the method may further include: obtaining a confidence threshold; and performing masking processing on the joint heatmaps from each perspective based on the confidence threshold to obtain a joint heatmap containing heatmap maxima points greater than or equal to the confidence threshold.

[0049] A joint heatmap is a probability distribution map of the possible two-dimensional (2D joint position) locations of each joint in an image. By obtaining a preset confidence threshold, the joint heatmaps from each viewpoint are masked according to the confidence threshold. This filters out heatmap maxima points in the initial joint heatmap that are below the confidence threshold, resulting in joint heatmaps that only contain heatmap maxima points greater than or equal to the confidence threshold. This avoids interference from heatmap maxima points below the confidence threshold and further improves the estimation accuracy of the target 3D joint coordinates.

[0050] See Figure 3In one embodiment, step S130, which involves deformable cross-attention feature aggregation of the joint feature maps from multiple perspectives to obtain multi-view fusion features for each joint, may include: step S310, generating a query feature vector for the joint based on the joint feature map of the query perspective, where the query perspective is one of the multiple perspectives; step S320, performing sampling offset prediction based on the query feature vector to obtain the sampling offset of the joint; step S330, sampling features from the joint feature maps of each perspective according to the sampling offset to obtain the sampling feature vector of the joint under each perspective; step S340, predicting the cross-attention weights of the joint under each perspective based on the query feature vector and the sampling feature vectors of the joint under each perspective; and step S350, multiplying and summing the sampling feature vectors of the joint under each perspective with the cross-attention weights to obtain the multi-view fusion features of the joint under the query perspective.

[0051] Each perspective in the multi-view can be used as a query perspective in turn. The preset alignment analysis module can include a vector generation unit, an offset prediction unit, a feature sampling unit, a weight calculation unit, and an aggregation unit.

[0052] In the vector generation unit, a linear transformation can be performed on the joint feature map of the query viewpoint k to generate the query feature vector Q of the joints included in the vector. k In the offset prediction unit, sampling offset prediction can be performed based on the query feature vector using a fully connected network or convolutional network to obtain the sampled offset Δp of the joint. Further, in the feature sampling unit, feature sampling can be performed from the joint feature maps of each viewpoint according to the sampled offset to obtain the sampled feature vector F of the joint at each viewpoint. l (p) k +Δp l ), where F l For the joint feature map of the l-th viewpoint, p k Let Δp be the joint coordinate. l F represents the sampling offset of the joint from the l-th viewpoint. l (p) k +Δp l ) is the sampling feature vector p extracted from the joint feature map of the l-th viewpoint. k +Δp l .

[0053] Furthermore, in the weight calculation unit, the cross-attention weights of joints at each viewpoint can be predicted based on the query feature vector and the sampled feature vectors of the joints at each viewpoint. Specifically, this can be achieved first by using a linear layer based on the query feature vector Q. k The sampled feature vector F of the joint from the l-th viewpoint l (p) k +Δp lPredict the attention weights and obtain the cross-attention weights w of the joints in the l-th viewpoint through SoftMax normalization. l .

[0054] Furthermore, in the aggregation unit, the sampled feature vectors of joints from each viewpoint can be multiplied by the cross-attention weights and summed to obtain the multi-view fusion features of the joints from the query viewpoint. For example, if the total number of viewpoints is L, the sampled feature vectors of joint j from the 1st to the Lth viewpoints can be multiplied by the cross-attention weights and summed to obtain the multi-view fusion feature A of joint j from the query viewpoint k. k,j = w l* F l (p) k +Δp l ).

[0055] In one embodiment, step S140, based on the initial three-dimensional joint coordinates and multi-view fusion features of each joint, performs three-dimensional pose prediction to obtain the target three-dimensional joint coordinates of each joint. This may include: stitching together the multi-view fusion features of each joint under multiple views to obtain the multi-view stitched features corresponding to each joint; performing fully connected processing on the initial three-dimensional joint coordinates and multi-view fusion features of each joint to obtain the predicted three-dimensional joint coordinates of each joint; and obtaining the target three-dimensional joint coordinates of each joint based on the predicted three-dimensional joint coordinates of each joint.

[0056] The pre-defined regression module can include splicing units and fully connected layers f. 3D In the stitching unit, if there are four perspectives, each perspective can be used as the query perspective k to obtain the multi-view fusion features of joint j. The multi-view fusion features of joint j under the first to fourth perspectives are represented as A1, A2, A3, and A4, respectively. A1, A2, A3, and A4 can be stitched together to obtain the multi-view stitching features corresponding to joint j, concat{A1, A2, A3, A4}. Furthermore, in the fully connected layer f... 3D The initial 3D joint coordinates and multi-view fusion features of each joint can be processed by a fully connected layer to obtain the predicted 3D joint coordinates (3D coordinates) of each joint output by the fully connected layer. The preset regression module can be constrained by loss functions such as MPJPE loss. Finally, the target 3D joint coordinates of each joint can be obtained based on the predicted 3D joint coordinates.

[0057] Furthermore, in one embodiment, before obtaining the target three-dimensional joint coordinates of each joint based on the predicted three-dimensional joint coordinates of each joint, the method may further include: determining the symmetrical joints corresponding to the missing joints of the target object; and performing a mirror transformation on the predicted three-dimensional joint coordinates of the symmetrical joints to obtain the predicted three-dimensional joint coordinates of the missing joints.

[0058] The preset regression module may also include a symmetry compensation unit. This unit can identify the missing joints of the target object (i.e., the joints missing after 3D pose prediction) and their corresponding symmetrical joints. For example, if the missing joint is the left hand, the corresponding symmetrical joint could be the right hand. Furthermore, the symmetry compensation unit can mirror the predicted 3D joint coordinates of the symmetrical joints to obtain the predicted 3D joint coordinates of the missing joints that are spatially symmetrical to the symmetrical joints, further reliably improving the completeness of joint pose estimation.

[0059] The symmetric compensation unit can be constrained by a predetermined loss function, which can include a penalty term for applying the symmetric constraint. This penalty term can be... , where P right It can be the predicted three-dimensional joint coordinates of a symmetrical joint, P left It can be the predicted 3D joint coordinates of a missing joint, where R refers to mirror transformation.

[0060] Furthermore, in one embodiment, obtaining the target three-dimensional joint coordinates of each joint based on the predicted three-dimensional joint coordinates of each joint may include: determining the predicted three-dimensional joint coordinates of each joint as the target three-dimensional joint coordinates of each joint.

[0061] Furthermore, in one embodiment, obtaining the target three-dimensional joint coordinates of each joint based on the predicted three-dimensional joint coordinates of each joint may include: performing velocity smoothing constraint processing on the predicted three-dimensional joint coordinates of each joint based on the historical three-dimensional joint coordinates of each joint at a previous time, thereby obtaining the target three-dimensional joint coordinates of each joint.

[0062] For example, the predicted 3D joint coordinates of joint j at the current time t are: The historical three-dimensional joint coordinates of joint j at time t-1 are: The historical three-dimensional joint coordinates of joint j at time t-2 are: Through the velocity smoothing function It can perform velocity smoothing constraint processing on the predicted three-dimensional joint coordinates of joint j, realize the spatiotemporal consistency constraint of three-dimensional joint coordinates, and suppress attitude jump through second-order difference approximation acceleration.

[0063] Furthermore, in one embodiment, before performing joint estimation on the multi-view images of the target object to obtain joint feature maps and joint heat maps for each view, the method may further include: receiving multi-view images captured by a first group of fisheye cameras and a second group of fisheye cameras positioned above the target object's head, wherein the field of view directions of the first group of fisheye cameras and the second group of fisheye cameras are opposite, and the overlapping area of ​​the field of view of adjacent fisheye cameras in the first group of fisheye cameras and the second group of fisheye cameras accounts for a predetermined proportion of the total field of view, the predetermined proportion being greater than a predetermined first threshold and less than a predetermined second threshold.

[0064] A first group of fisheye cameras and a second group of fisheye cameras are positioned above the target object's head. The fields of view of the first and second groups of fisheye cameras are opposite; for example, the field of view of the first group is forward, and the field of view of the second group is backward. The overlap area of ​​the fields of view of adjacent fisheye cameras in the first and second groups constitutes a predetermined proportion of the total field of view. For example, if the first and second groups each contain two fisheye cameras, the predetermined proportion of the overlap area of ​​the fields of view of adjacent fisheye cameras in the total field of view (the combined fields of view of the first and second groups) can be greater than a predetermined first threshold of 40% and less than a predetermined second threshold of 60%. With this configuration, multi-view images effectively covering the target object can be acquired, centered on the target object itself, effectively improving the completeness of the target object's 3D pose estimation.

[0065] Among them, the preset estimation module, preset alignment analysis module, preset aggregation module, and preset regression module can be modules in the preset 3D pose estimation model, which can be a pre-trained deep learning model.

[0066] When training a preset 3D pose estimation model using sample images, some methods can be used to generate adversarial sample images containing occlusion for occluded scenes. The sample images and adversarial sample images are then used to train the preset 3D pose estimation model, forcing the preset 3D pose estimation model to learn occlusion robustness and further improving the performance of 3D pose estimation.

[0067] For example, the Fast Gradient Sign Method (FGSM) can be used to generate adversarial example images containing occlusions from sample images. The Fast Gradient Sign Method can be represented as I... adv =I Where I refers to the sample image, I adv Adversarial sample images, ε refers to the perturbation intensity, L 3D Refers to the adversarial loss function. Refers to calculating the adversarial loss function L 3D The gradient relative to sample image I, Refers to the gradient symbol.

[0068] Furthermore, a mixed-precision model can be used to train and optimize the preset 3D pose estimation model. For example, mixed-precision model can include FP16 and FP32. The loss and gradient can be calculated using FP16, and the gradient can be multiplied by a gradient scaling factor (to prevent FP16 underflow). Then, the gradient is converted back to FP32, the weight parameters of FP32 are updated, and then synchronized back to FP16. Under smooth system conditions, the gradient scaling factor can be S / 2; otherwise, it is 2S, achieving dynamic loss scaling. Overall, this improves training speed while maintaining numerical stability.

[0069] Furthermore, channel pruning can be applied to the trained preset 3D pose estimation model, followed by knowledge distillation loss, to obtain the final preset 3D pose estimation model. This makes the preset 3D pose estimation model more lightweight and easier to deploy at the edge.

[0070] Furthermore, in one embodiment of this application, a pose subspace can be constructed based on multi-view images, so that three-dimensional pose estimation can be performed in the pose subspace to obtain the target three-dimensional joint coordinates of the joint located in the pose subspace. In particular, the three-dimensional pose estimation of a new user can be limited to the kinematically feasible domain of the human body, thereby further ensuring the accuracy of the three-dimensional joint coordinate estimation.

[0071] To facilitate better implementation of the three-dimensional attitude estimation method provided in the embodiments of this application, the embodiments of this application also provide a three-dimensional attitude estimation device based on the above-described three-dimensional attitude estimation method. The meanings of the terms used are the same as in the three-dimensional attitude estimation method described above, and specific implementation details can be found in the descriptions in the method embodiments. Figure 4 A block diagram of a three-dimensional pose estimation apparatus according to an embodiment of this application is shown.

[0072] like Figure 4 As shown, the three-dimensional pose estimation device 400 may include: an estimation module 410 for estimating joints in multi-view images of a target object to obtain joint feature maps and joint heatmaps for each view; an alignment analysis module 420 for performing cross-view alignment analysis on the joint heatmaps from multiple views to obtain initial three-dimensional joint coordinates for each joint; an aggregation module 430 for performing deformable cross-attention feature aggregation on the joint feature maps from multiple views to obtain multi-view fusion features for each joint; and a regression module 440 for performing three-dimensional pose prediction based on the initial three-dimensional joint coordinates and the multi-view fusion features for each joint to obtain target three-dimensional joint coordinates for each joint.

[0073] In some embodiments of this application, the alignment analysis module 420 can be used to: extract the two-dimensional joint coordinates of each joint from the joint heatmaps of each viewpoint; construct the projection function from three dimensions to two dimensions for each joint from each viewpoint based on the camera parameters of each viewpoint and the two-dimensional joint coordinates of each joint from each viewpoint; calculate the cross-view geometric constraint error based on the two-dimensional joint coordinates and projection function of each joint from different viewpoints; and minimize the cross-view geometric constraint error to obtain the initial three-dimensional joint coordinates of each joint.

[0074] In some embodiments of this application, the aggregation module 430 can be used to: generate a query feature vector of a joint based on the joint feature map of the query view, wherein the query view is one of the multiple views; perform sampling offset prediction based on the query feature vector to obtain the sampling offset of the joint; perform feature sampling from the joint feature map of each view according to the sampling offset to obtain the sampling feature vector of the joint under each view; predict the cross-attention weight of the joint under each view based on the query feature vector and the sampling feature vector of the joint under each view; multiply the sampling feature vector of the joint under each view with the cross-attention weight and sum them to obtain the multi-view fusion feature of the joint under the query view.

[0075] In some embodiments of this application, the regression module 440 can be used to: stitch together the multi-view fusion features of each joint under the multi-view perspective to obtain the multi-view stitched features corresponding to each joint; perform fully connected processing on the initial three-dimensional joint coordinates of each joint and the multi-view fusion features to obtain the predicted three-dimensional joint coordinates of each joint; and obtain the target three-dimensional joint coordinates of each joint based on the predicted three-dimensional joint coordinates of each joint.

[0076] In some embodiments of this application, before obtaining the target three-dimensional joint coordinates of each joint based on the predicted three-dimensional joint coordinates of each joint, the device further includes a symmetry compensation unit that can be used to: determine the symmetrical joint corresponding to the missing joint of the target object; and perform a mirror transformation on the predicted three-dimensional joint coordinates of the symmetrical joint to obtain the predicted three-dimensional joint coordinates of the missing joint.

[0077] In some embodiments of this application, the regression module 440 can be used to: perform velocity smoothing constraint processing on the predicted three-dimensional joint coordinates of each joint based on the historical three-dimensional joint coordinates of each joint at a previous time, so as to obtain the target three-dimensional joint coordinates of each joint.

[0078] In some embodiments of this application, the estimation module 410 may be used to: encode the images of each viewpoint to obtain the joint feature map of each viewpoint; and perform convolution and activation processing on the joint feature maps of each viewpoint to obtain the joint heatmap of each viewpoint.

[0079] In some embodiments of this application, before performing cross-view alignment analysis on the joint heatmaps from multiple perspectives to obtain the initial three-dimensional joint coordinates of each joint point, the device further includes a masking module, which can be used to: obtain a confidence threshold; and perform masking processing on the joint heatmaps from each perspective according to the confidence threshold to obtain a joint heatmap containing heatmap maxima points greater than or equal to the confidence threshold.

[0080] In some embodiments of this application, before performing joint estimation on the multi-view images of the target object to obtain joint feature maps and joint heat maps for each view, the device further includes a receiving module for: receiving multi-view images captured by a first group of fisheye cameras and a second group of fisheye cameras positioned above the target object, wherein the field of view directions of the first group of fisheye cameras and the second group of fisheye cameras are opposite, and the overlapping area of ​​the field of view of adjacent fisheye cameras in the first group of fisheye cameras and the second group of fisheye cameras accounts for a predetermined proportion of the total field of view, wherein the predetermined proportion is greater than a predetermined first threshold and less than a predetermined second threshold.

[0081] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0082] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, Figure 5 A block diagram of an electronic device according to an embodiment of this application is shown, specifically: The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 501 is the control center of the electronic device, connecting various parts of the computer device via various interfaces and lines. It executes software programs and / or modules stored in the memory 502, and calls data stored in the memory 502, to perform various functions of the computer device and process data. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and application programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0083] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0084] The electronic device also includes a power supply 503 that supplies power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0085] The electronic device may also include an input unit 504, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0086] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device can load the executable files corresponding to the processes of one or more computer programs into the memory 502 according to the following instructions, and the processor 501 runs the computer programs stored in the memory 502, thereby realizing the various functions in the foregoing embodiments of this application.

[0087] For example, processor 501 can perform the following: perform joint estimation on multi-view images of the target object to obtain joint feature maps and joint heatmaps for each view; perform cross-view alignment analysis on the joint heatmaps from multiple views to obtain initial three-dimensional joint coordinates for each joint point; perform deformable cross-attention feature aggregation on the joint feature maps from multiple views to obtain multi-view fusion features for each joint; and perform three-dimensional pose prediction based on the initial three-dimensional joint coordinates and the multi-view fusion features for each joint to obtain the target three-dimensional joint coordinates for each joint.

[0088] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0089] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the methods provided in embodiments of this application.

[0090] The storage medium can be a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0091] Since the computer program stored in the storage medium can execute the steps of any of the methods provided in the embodiments of this application, the beneficial effects that the methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0092] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0093] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0094] It should be understood that this application is not limited to the embodiments described above and shown in the accompanying drawings, but various modifications and changes can be made without departing from its scope.

Claims

1. A method of three-dimensional pose estimation, the method comprising: The method comprises: joint estimation is performed on multi-view images of a target object to obtain joint feature maps and joint heat maps of each view; cross-view alignment analysis is performed on the joint heat maps of multiple views to obtain initial three-dimensional joint coordinates of each joint; deformable cross-attention feature aggregation is performed on the joint feature maps of multiple views to obtain multi-view fusion features of each joint; three-dimensional pose prediction is performed based on the initial three-dimensional joint coordinates of each joint and the multi-view fusion features to obtain target three-dimensional joint coordinates of each joint.

2. The method of claim 1, wherein, The cross-view alignment analysis of the joint heat maps of multiple views to obtain initial three-dimensional joint coordinates of each joint comprises: two-dimensional joint coordinates of each joint in each view are extracted from the joint heat maps of each view; projection functions from three-dimensional to two-dimensional of each joint in each view are constructed based on camera parameters of each view and two-dimensional joint coordinates of each joint in each view; cross-view geometric constraint errors are calculated according to two-dimensional joint coordinates of each joint in different views and the projection functions; the cross-view geometric constraint errors are minimized to obtain initial three-dimensional joint coordinates of each joint.

3. The method of claim 1, wherein, The deformable cross-attention feature aggregation of the joint feature maps of multiple views to obtain multi-view fusion features of each joint comprises: a query feature vector of a joint is generated based on the joint feature map of a query view, the query view being one of the multiple views; a sampling offset of the joint is predicted based on the query feature vector; sampling feature vectors of the joint in each view are obtained by sampling features from the joint feature maps of each view according to the sampling offset; cross-attention weights of the joint in each view are predicted based on the query feature vector and the sampling feature vectors of the joint in each view; the multi-view fusion features of the joint in the query view are obtained by multiplying and summing the sampling feature vectors of the joint in each view and the cross-attention weights.

4. The method of claim 1, wherein, The three-dimensional pose prediction based on the initial three-dimensional joint coordinates of each joint and the multi-view fusion features to obtain target three-dimensional joint coordinates of each joint comprises: multi-view splicing features corresponding to each joint are obtained by splicing the multi-view fusion features of each joint in the multiple views; predicted three-dimensional joint coordinates of each joint are obtained by fully connecting the initial three-dimensional joint coordinates of each joint and the multi-view fusion features; target three-dimensional joint coordinates of each joint are obtained according to the predicted three-dimensional joint coordinates of each joint.

5. The method of claim 4, wherein, Before the target three-dimensional joint coordinates of each joint are obtained according to the predicted three-dimensional joint coordinates of each joint, the method further comprises: symmetrical joints corresponding to missing joints of the target object are determined; predicted three-dimensional joint coordinates of the missing joints are obtained by mirror transformation of the predicted three-dimensional joint coordinates of the symmetrical joints.

6. The method of claim 4, wherein, The target three-dimensional joint coordinates of each joint are obtained according to the predicted three-dimensional joint coordinates of each joint, comprising: According to the historical three-dimensional joint coordinates of each joint at the previous time, the predicted three-dimensional joint coordinates of each joint are subjected to speed smoothing constraint processing to obtain target three-dimensional joint coordinates of each joint.

7. The method of claim 1, wherein, The joint estimation on the multi-view images of the target object obtains joint feature maps and joint heat maps of each view. The images of each view are respectively subjected to encoding processing to obtain the joint feature maps of each view. The joint feature maps of each view are subjected to convolution and activation processing to obtain the joint heat maps of each view.

8. The method according to any one of claims 1 to 7, characterized in that, Before the cross-view alignment analysis on the multi-view joint heat maps to obtain the initial three-dimensional joint coordinates of each joint, the method further comprises: obtaining a confidence threshold; According to the confidence threshold, the joint heat maps of each view are subjected to mask processing to obtain the joint heat maps containing the maximum value points of the heat maps greater than or equal to the confidence threshold.

9. The method according to any one of claims 1 to 7, characterized in that, Before the joint estimation on the multi-view images of the target object to obtain the joint feature maps and joint heat maps of each view, the method further comprises: receiving multi-view images captured by a first group of fisheye cameras and a second group of fisheye cameras arranged on the head of the target object, wherein the field of view directions of the first group of fisheye cameras and the second group of fisheye cameras are opposite, the field of view overlap area of adjacent fisheye cameras in the first group of fisheye cameras and the second group of fisheye cameras accounts for a predetermined proportion of the total field of view, and the predetermined proportion is greater than a predetermined first threshold and less than a predetermined second threshold.

10. A three-dimensional pose estimation apparatus, characterized by comprising: comprises: an estimation module configured to perform joint estimation on multi-view images of a target object to obtain joint feature maps and joint heat maps of each view; an alignment analysis module configured to perform cross-view alignment analysis on the multi-view joint heat maps to obtain initial three-dimensional joint coordinates of each joint; an aggregation module configured to perform deformable cross-attention feature aggregation on the multi-view joint feature maps to obtain multi-view fusion features of each joint; a regression module configured to perform three-dimensional pose prediction based on the initial three-dimensional joint coordinates of each joint and the multi-view fusion features to obtain target three-dimensional joint coordinates of each joint.

11. A storage medium, characterized by A computer program is stored thereon, and when the computer program is executed by a processor of an electronic device, the electronic device performs the method of any one of claims 1 to 9.

12. An electronic device, comprising: comprises: a memory storing a computer program; a processor reading the computer program stored in the memory to execute the method of any one of claims 1 to 9.

Images