A multi-view three-dimensional human pose estimation method and system

Abstract

The application discloses a multi-view three-dimensional human posture estimation method and system, and specifically comprises the following steps: based on multi-view two-dimensional key point sequences, weighted spatial features of each frame under each view are extracted respectively; based on the weighted spatial features of each view and corresponding two-dimensional detection confidence, fusion is performed through a view adaptive fusion mechanism to generate unified multi-view fusion spatial features; the multi-view fusion spatial features are subjected to time sequence modeling, local time sequence features of each joint are extracted, and time sequence weighted skeleton features are obtained; global time dependence relationship modeling is performed on the time sequence weighted skeleton features to obtain feature sequences with fused space-time dynamic information; and based on the feature sequences, three-dimensional human posture coordinate sequences are predicted through a regression network. While guaranteeing the spatial topology and time semantic modeling capability, the application effectively fuses multi-view features and enhances the stability of time sequence modeling, and improves the accuracy of three-dimensional human posture estimation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-view 3D human pose estimation method and system. Background Technology

[0002] The goal of 3D human pose estimation is to recover the joint coordinates of the human body in 3D space based on the input 2D keypoint sequence. It plays a crucial role in applications such as motion recognition and human-computer interaction.

[0003] In single-person, multi-view scenarios, multi-order GCN structures are often introduced into Graph Convolutional Networks (GCNs) to enable each joint to perceive more distant body parts. However, as the order increases, higher-order structures contain a large amount of node information that is weakly related to the target joint. The model cannot effectively distinguish which structural information is more important, thus affecting the accuracy of the overall body structure representation. Specifically, existing GCNs typically extract human joint features at multiple adjacent orders, but higher orders contain a large number of irrelevant nodes. Furthermore, the importance of dependencies between different orders varies in different action scenarios. Simple averaging and fusion can lead to the over-amplification of irrelevant higher-order information, masking effective features and causing feature redundancy and noise accumulation.

[0004] Secondly, in multi-camera shooting scenarios, images from different perspectives often exhibit differences in lighting, occlusion, and angle. Simply stitching and fusing these perspective features directly can easily be interfered with by features from low-quality perspectives, leading to inaccurate pose estimation. Specifically, in multi-camera human pose estimation scenarios, due to differences in perspective, uneven lighting, and local joint occlusion between different cameras, the features from each perspective often have significant differences in information quality and semantic consistency. Direct stitching is easily affected by low-quality perspectives, especially when some perspectives are severely occluded, making the model's estimation results even more unstable and weakening the overall robustness of the model.

[0005] Furthermore, in action sequences, existing temporal modeling methods often apply equal weights to all frames. However, key frames such as peak frames and transition frames contribute more to 3D estimation, such as the highest and lowest points of a squatting motion. Equal weighting can lead to insufficient understanding of the temporal structure of the action by the model. Specifically, in action sequences for video-based 3D human pose estimation, existing methods often assign the same weight to all frames or select a frame in the middle of the action sequence to represent the entire action process. However, in actual human motion, some key frames (such as peaks and turning points) often carry richer pose information. Ignoring the dynamic features of these frames often leads to inaccurate estimation of action transitions and transition phases, resulting in insufficient dynamic representation of the action. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-view 3D human pose estimation method and system, which, while ensuring the ability to model spatial topology and temporal semantics, effectively integrates multi-view features and enhances the stability of temporal modeling, thereby improving the accuracy of 3D human pose estimation and solving at least one of the aforementioned problems in the prior art.

[0007] In a first aspect, the present invention provides a multi-view three-dimensional human pose estimation method, the method specifically comprising:

[0008] Input a multi-view 2D keypoint sequence, and based on the multi-view 2D keypoint sequence, use a multi-order graph attention mechanism to extract the weighted spatial features of each frame under each view.

[0009] Based on the weighted spatial features of each perspective and the corresponding two-dimensional detection confidence, a perspective adaptive fusion mechanism is used to fuse them and generate unified multi-view fused spatial features.

[0010] Temporal modeling is performed on the multi-view fused spatial features, and local temporal features of each joint are extracted through the joint-weighted attention module to obtain temporal-weighted skeleton features;

[0011] A dual-center multi-head attention mechanism is used to model the global temporal dependency of temporally weighted skeleton features, resulting in a feature sequence that integrates spatiotemporal dynamic information.

[0012] Based on the feature sequence that integrates spatiotemporal dynamic information, a three-dimensional human posture coordinate sequence is obtained by prediction through a regression network.

[0013] Secondly, the present invention provides a multi-view three-dimensional human pose estimation system, the system specifically comprising:

[0014] The spatial feature extraction module is used to input multi-view two-dimensional keypoint sequences. Based on the multi-view two-dimensional keypoint sequences, the weighted spatial features of each frame under each view are extracted using a multi-order graph attention mechanism.

[0015] The spatial feature fusion module is used to fuse the weighted spatial features and corresponding two-dimensional detection confidence scores from each viewpoint through a viewpoint adaptive fusion mechanism to generate unified multi-view fused spatial features.

[0016] The temporal feature extraction module is used to perform temporal modeling of multi-view fused spatial features. It extracts the local temporal features of each joint through the joint weighted attention module to obtain temporal weighted skeleton features.

[0017] The temporal feature fusion module is used to model the global temporal dependency of temporally weighted skeleton features using a dual-center multi-head attention mechanism, so as to obtain a feature sequence that integrates spatiotemporal dynamic information.

[0018] The coordinate sequence output module is used to predict the three-dimensional human posture coordinate sequence through a regression network based on the feature sequence fused with spatiotemporal dynamic information.

[0019] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, and a computer program stored in the memory, wherein when the computer program is executed on the processor, it implements the multi-view three-dimensional human pose estimation method as described in any of the above methods.

[0020] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-view three-dimensional human pose estimation method as described in any of the above methods.

[0021] Compared with the prior art, the present invention has at least one of the following technical effects:

[0022] 1. This invention, while ensuring the ability to model spatial topology and temporal semantics, effectively integrates multi-view features and enhances the stability of temporal modeling, thereby improving the accuracy of 3D human pose estimation.

[0023] 2. This invention introduces a multi-order graph attention mechanism, which, based on the extraction of joint relationship features of different orders, assigns learnable weights to each order feature, enabling the model to automatically adjust the contribution of each order feature according to structural relevance, thereby enhancing the spatial feature representation ability of human body structure.

[0024] 3. This invention introduces a viewpoint adaptive fusion mechanism to calculate the reliability and consistency of each viewpoint at different joints. Reliability is used to suppress occlusion or inaccurate detection of viewpoint features, while consistency is used to improve cross-viewpoint semantic alignment. Then, fusion weights are adaptively generated through learnable parameters. In this way, the final output spatial features retain the complementary information of multiple views while effectively reducing the interference of low-quality views, thereby improving the robustness of the human pose estimation model under multiple views.

[0025] 4. This invention introduces a dual-center multi-head attention mechanism, which simultaneously focuses on the global geometric center frame and the action peak frame in temporal modeling. It calculates the time distance decay weight for the two types of centers respectively, and dynamically balances the two attention paths through gating parameters, so that the model can take into account both the stable and changing stages of the action, thereby capturing the dynamic semantics of human motion more accurately.

[0026] 5. This invention extracts weighted spatial features based on multi-view two-dimensional keypoint sequences using a multi-order graph attention mechanism. It can adaptively allocate weights for features of different orders, enhance the capture of important structural information, and improve the effectiveness of spatial feature extraction.

[0027] 6. This invention generates unified features based on weighted spatial features from each viewpoint and two-dimensional detection confidence through a viewpoint adaptive fusion mechanism, comprehensively considering reliability and consistency, reducing interference from low-quality viewpoints, and enhancing the robustness of the fused features.

[0028] 7. This invention calculates a reliability score based on the feature intensity of normalized spatial features and the two-dimensional detection confidence, and integrates feature intensity and detection confidence to accurately measure the reliability of joint features from various perspectives, providing a reliable basis for subsequent fusion.

[0029] 8. This invention calculates the consistency score at the joints of each viewpoint based on normalized spatial features, and measures the consistency with other viewpoints through cosine similarity, which can accurately reflect the degree of consistency of features between viewpoints and help generate reasonable fusion weights.

[0030] 9. This invention utilizes a dual-center multi-head attention mechanism to model temporally weighted skeleton features to obtain a feature sequence that integrates spatiotemporal dynamic information. By considering the geometric center and action peak frames, it can better capture global temporal dependencies and enhance the expression of action dynamics.

[0031] 10. This invention calculates feature sequences by integrating a dual-center multi-head attention mechanism with a comprehensive scaling vector, and introduces a comprehensive scaling vector to modulate query-key relevance, which can more accurately calculate attention weights and improve the quality of feature sequences fused with spatiotemporal dynamic information.

[0032] 11. This invention predicts the three-dimensional human posture coordinate sequence through a regression network based on the feature sequence of fused spatiotemporal dynamic information. After nonlinear transformation by a multilayer perceptron and mapping by a fully connected layer, the feature sequence can be accurately transformed into three-dimensional spatial coordinates, thereby achieving accurate posture estimation.

[0033] 12. This invention performs temporal modeling on multi-view fusion spatial features and obtains temporal weighted skeleton features through a joint-weighted attention module. It can independently model inter-frame dependencies for each joint, thereby improving the accuracy of temporal feature representation. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a multi-view three-dimensional human pose estimation method provided in an embodiment of the present invention;

[0036] Figure 2This is a schematic diagram of the overall spatial GCN provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of a time modeling architecture provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the structure of a multi-view three-dimensional human pose estimation system provided in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0040] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0041] It should be understood that, when used in this application specification, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0042] It should also be understood that the term “and / or” as used in this application specification means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0043] As used in this application specification, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0044] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0045] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0046] In this application embodiment, the entity executing the process includes a terminal device. This terminal device includes, but is not limited to, devices capable of executing the methods disclosed in this application, such as servers, computers, smartphones, and tablets. Figure 1 A flowchart illustrating a multi-view 3D human pose estimation method disclosed in an embodiment of the present invention is shown below in detail:

[0047] S101, Input a multi-view 2D keypoint sequence, and based on the multi-view 2D keypoint sequence, use a multi-order graph attention mechanism to extract the weighted spatial features of each frame under each view.

[0048] S102, based on the weighted spatial features of each viewpoint and the corresponding two-dimensional detection confidence, a viewpoint adaptive fusion mechanism is used to fuse them and generate unified multi-view fused spatial features.

[0049] S103 performs temporal modeling on multi-view fusion spatial features, extracts local temporal features of each joint through a joint-weighted attention module, and obtains temporal-weighted skeleton features.

[0050] S104 utilizes a dual-center multi-head attention mechanism to model the global temporal dependency of temporally weighted skeleton features, resulting in a feature sequence that integrates spatiotemporal dynamic information.

[0051] S105, based on the feature sequence of fused spatiotemporal dynamic information, obtains the three-dimensional human posture coordinate sequence through regression network prediction.

[0052] In some embodiments, step S101 above, which involves extracting weighted spatial features of each frame from each viewpoint using a multi-view two-dimensional keypoint sequence and a multi-order graph attention mechanism, specifically includes:

[0053] Based on multi-view 2D keypoint sequences, for each frame of each view, multi-level graph convolutional features with different receptive ranges are extracted through a multi-level graph convolutional network.

[0054] Based on multi-order graph convolution features, the concatenation is performed along the order dimension to obtain the order concatenation features of each frame.

[0055] Based on the order-separated features, the attention weights of each order feature are calculated through a graph-order attention mechanism;

[0056] Based on attention weights, weighted summation and nonlinear activation are performed on multi-order graph convolutional features to generate weighted spatial features.

[0057] In this embodiment, the method uses a multi-view 2D human pose sequence as input data. Video frame sequences from multiple camera perspectives are processed by an existing 2D pose detection model to extract the keypoint coordinate sequence of the human 2D skeleton for each frame. For the i-th frame of the v-th perspective, the keypoint coordinate sequence can be represented as... Arrange the keypoint coordinates of each frame in chronological order to form a two-dimensional skeleton sequence from that viewpoint v. , v represents the viewpoint number, T represents the video frame number, and J represents the number of human keypoints. The dimension is The matrix.

[0058] Ultimately, the two-dimensional skeleton sequence obtained from each viewpoint As input to the next module, it is used to extract spatial structure features from various perspectives.

[0059] The spatial GCN introduced in this method consists of a multi-order graph convolutional network (MO-GCN) and a graph-order attention mechanism (GOA). MO-GCN is used to aggregate features from different distances from adjacent joints, while GOA assigns importance to different sequences of joints in each pose.

[0060] (1) In MO-GCN, each viewpoint is calculated separately. The two-dimensional skeleton sequence of the viewpoint v input from the previous module is represented as follows: , Let T represent the sequence of keypoint coordinates for the i-th frame of viewpoint v, where T represents the number of frames. The keypoint coordinates for each frame are... Organized into an undirected graph using a predefined adjacency matrix Describe the physical connections between the joints. The dimension is Given a matrix, if there is a skeletal connection between two joints i and j, then... =1, otherwise 0.

[0061] To capture spatial dependencies at different levels within the human body, this method employs a multi-level graph convolution structure. Multi-level graph convolution progressively expands the adjacency range, allowing the target node to simultaneously receive structural information from its direct neighbors and more distant neighbors. The k-th order adjacency matrix is ​​defined as follows: , represents the connection relationship between nodes with a shortest path of k, where the value of k ranges from 1 to 2. R is the maximum order. For the input skeleton of the i-th frame at the v-th viewpoint. The feature representation vectors of different orders are obtained by multi-order graph convolution, as shown in equation (1):

[0062] (1)

[0063] in It is a skeleton of order k. Feature representation, W is the normalized k-th order adjacency matrix. k It is the learnable parameter matrix of the k-th convolutional layer.

[0064] This process allows us to obtain spatial characteristics of different sensory ranges. Where J is the number of joints and D is the output feature dimension. The dimension is The matrix.

[0065] (2) After extracting spatial features of different orders from multi-order features, a graph order attention (GOA) mechanism is introduced to adaptively weight the features of each order.

[0066] For each viewpoint v, the convolutional features of the skeleton of the i-th frame under that viewpoint are... The concatenation is done along the order dimension, as shown in equation (2). , The graph convolution features of order 0 to order R of the skeleton in the i-th frame at viewpoint v are represented by the symbol. This represents a concatenation operation along the feature dimension, which connects features of different orders in order of their order to form a new feature representation vector. That is, the hierarchical splicing feature.

[0067] (2)

[0068] Define learnable queries within each perspective. AND key The mapping and weight matrices are respectively Calculated independently within each viewpoint v. and Let represent the query and key features obtained by linear transformation at the i-th frame of the v-th viewpoint, respectively, used to characterize the correlation between different graph levels in that frame. The formula is as shown in equation (3):

[0069] , (3)

[0070] Further define a learnable order-weighted attention matrix , Let represent the attention weight of the joint corresponding to the i-th frame in the v-th view, and represent the importance distribution of different orders within the same view, where . ( (A matrix of dimension D) is a learnable linear projection vector. The calculation formula is shown in equation (4):

[0071] (4)

[0072] Finally, the GOA module performs a weighted summation of the convolutional features of each order to obtain the weighted spatial feature representation of the i-th frame under this viewpoint, and the output formula is expressed as Equation (5). It is a skeleton of order k. Feature representation, This represents the attention weights of the joints at order k, obtained as follows: This represents the final spatial feature representation obtained from the v-th viewpoint and the i-th frame after graph-level attention weighting.

[0073] (5)

[0074] Finally, a nonlinear activation function is applied to the result. The weighted spatial features from this perspective are obtained, as shown in equation (6):

[0075] (6)

[0076] This weighted space feature As a spatial representation within the viewpoint, it will be input into the multi-view adaptive fusion module for further processing.

[0077] Reference Figure 2 , Figure 2 This paper demonstrates the core process of multi-view 3D human pose estimation methods, describing the complete process from inputting 2D skeletons from various perspectives to outputting unified multi-view fused spatial features. It mainly includes two key parts: multi-order graph attention mechanism and viewpoint adaptive fusion mechanism.

[0078] Specifically, in the multi-order graph attention mechanism, firstly, based on the 2D skeleton input from each viewpoint (dimension (J,2), where J is the number of joints), for each frame of each viewpoint, multi-order graph convolutional features are extracted using graph convolutional networks (GCNs, such as GCN(Ord=0) to GCN(Ord=R)), with each order's output dimension being (J,D). Then, these features are concatenated along the order dimensions to obtain the order concatenated features (dimension (J,D,R)). Subsequently, the attention weights of each order feature are calculated using the graph order attention mechanism. Based on these weights, the multi-order graph convolutional features are weighted and summed, then non-linearly activated to generate the weighted spatial features (dimension (J,D)) for each frame under each viewpoint.

[0079] In the viewpoint adaptive fusion mechanism, the weighted spatial features of each viewpoint are first normalized to obtain normalized spatial features. Then, based on the normalized spatial features and the corresponding two-dimensional detection confidence, the reliability score of each viewpoint at each joint is calculated. Specifically, the feature strength is obtained by first calculating the L2 norm of the feature vector, and then combining it with the detection confidence through a linear combination and activation function to generate a score. Simultaneously, the consistency score of each viewpoint relative to other viewpoints at each joint is calculated based on the normalized spatial features. This is obtained by calculating the cosine similarity of feature vectors from different viewpoints in the same frame at the same joint and then averaging them. Afterwards, based on the reliability score and consistency score, a learnable function is used to generate a comprehensive weight, which is then normalized to obtain the contribution ratio. Finally, the normalized spatial features of all viewpoints are weighted and summed according to the contribution ratio to generate a unified multi-view fusion spatial feature (dimension (J,D)).

[0080] In some embodiments, in step S102 above, the fusion of weighted spatial features based on each viewpoint and the corresponding two-dimensional detection confidence through a viewpoint adaptive fusion mechanism to generate unified multi-view fused spatial features specifically includes:

[0081] Channel normalization is performed on the weighted spatial features from each perspective to obtain normalized spatial features;

[0082] Based on the normalized spatial features and the corresponding two-dimensional detection confidence, the reliability score of each viewpoint at each joint is calculated.

[0083] Based on normalized spatial features, calculate the consistency score of each viewpoint relative to other viewpoints at each joint.

[0084] Based on reliability and consistency scores, a learnable fusion weight calculation function is used to generate comprehensive weights for each perspective at each joint.

[0085] The overall weights are normalized to obtain the contribution ratio of each perspective at each joint;

[0086] Based on the contribution ratio, the normalized spatial features of all perspectives are weighted and summed to generate unified multi-perspective fused spatial features.

[0087] Furthermore, the calculation of reliability scores for each viewpoint at each joint based on normalized spatial features and corresponding two-dimensional detection confidence specifically includes:

[0088] For the feature vector of each joint in the normalized spatial features, calculate its L2 norm to obtain the feature intensity of the joint under the current view.

[0089] The detection confidence of the corresponding joint in the two-dimensional detection confidence is obtained. Based on the feature intensity and the detection confidence, the results are fused by a linear combination function with learnable coefficients to obtain the fusion result.

[0090] The fusion result is input into the activation function to generate a reliability score.

[0091] Furthermore, the calculation of the consistency score of each viewpoint relative to other viewpoints at each joint based on normalized spatial features specifically includes:

[0092] From the normalized spatial features of the current perspective, obtain the first feature vector at the specified frame and the specified joint;

[0093] Obtain the second feature vector at the same frame and the same joint from the normalized spatial features of any other viewpoint;

[0094] Calculate the cosine similarity between the first feature vector and the second feature vector, and use the cosine similarity as a measure of the consistency between the current viewpoint and any other viewpoint at the specified joint.

[0095] Integrate the consistency metrics of the current viewpoint with all other viewpoints at a specified joint to form a consistency metric group;

[0096] The consistency score at a specified joint is obtained by averaging the consistency metrics group.

[0097] In this embodiment, each viewpoint will use the spatial features obtained by the GOA module. The corresponding two-dimensional detection confidence level generated by the initial 2D pose detection model Both are used as inputs, and the confidence level is... This represents the confidence level of the model in predicting the position of the j-th joint in the i-th frame from the v-th viewpoint. It is obtained by concatenating the confidence scores of all joints within the same frame. Within View-Adaptive Fusion Mechanism (VAF), it is necessary to perform [the necessary steps] for each frame. Integrating into a unified multi-perspective spatial feature This is so that it can be used as input for subsequent time modules.

[0098] Since the feature distribution may be offset from different perspectives, the features of each perspective are first analyzed. Normalized along the channel dimension, represented as Simultaneously, to evaluate the significance of a single viewpoint's response at a particular joint—that is, whether the joint's features are significant and reliable from the current viewpoint—feature strength is defined. , This represents the eigenvector of the j-th joint after normalization, denoted by "". " indicates the L2 norm operation.

[0099] Different viewpoints vary in image quality and joint visibility, and direct fusion may introduce noise from low-quality viewpoints. Therefore, this method evaluates the reliability and consistency of each viewpoint before fusion and adaptively determines the contribution of each viewpoint to the final result.

[0100] (1) Reliability C calculation: A single viewpoint may have occlusion or detection noise, so it is necessary to evaluate whether the viewpoint is reliable at the corresponding joint and calculate the two-dimensional detection confidence. With normalized feature strength Together, they serve as the basis for reliability assessment. The method is shown in equation (7). The learnable coefficients are obtained This indicates the reliability of the observation of the j-th joint from the v-th viewpoint in the i-th frame.

[0101] (7)

[0102] (2) Consistency R calculation: For multi-view problems, the same joint should be expressed consistently from different viewpoints. Therefore, cosine similarity is used. Indicates consistency. This represents the cosine similarity between the u-th and v-th viewpoints at frame i joint j. This represents the normalized feature vector at the j-th joint of the i-th frame from the v-th viewpoint. It is the feature vector at the j-th joint of the i-th frame from another viewpoint u, v is the viewpoint being evaluated, and u is another viewpoint used for comparison. The similarity between viewpoint v and other viewpoint u is calculated as shown in Equation (8):

[0103] (8)

[0104] By summing the similarity of the joint across all other viewpoints, a consistency score can be obtained at that joint, as shown in Equation (9), where V is the number of viewpoints. The cosine similarity between the u-th and v-th viewpoints at frame i joint j is the final result. This represents the consistency score of the j-th joint in the i-th frame for the v-th viewpoint.

[0105] (9)

[0106] Reliability With consistency The final perspective fusion weights are formed by combining the results, as shown in equation (10). The learnable coefficient, To prevent numerical underflow of the stability constant, the obtained This represents the viewpoint weight of the v-th viewpoint at the j-th joint in the i-th frame, used to evaluate the effectiveness and reliability of that viewpoint.

[0107] (10)

[0108] After normalization, the final contribution ratio for each viewpoint is obtained. As in equation (11):

[0109] (11)

[0110] By applying this weight ratio to the joints of all viewpoints and performing a weighted summation, a unified multi-view fusion spatial feature is obtained. This feature will serve as the input to the time modeling module, as shown in equation (12):

[0111] (12)

[0112] In some embodiments, step S103 above, which involves performing temporal modeling on the multi-view fused spatial features and extracting local temporal features of each joint through a joint-weighted attention module to obtain temporally weighted skeleton features, specifically includes:

[0113] For each joint, the multi-view fused spatial features are spliced ​​together in the time dimension to form the time series features of that joint;

[0114] Based on time series features, the inter-frame attention weight matrix for each joint is calculated using a learnable temporal correlation mapping matrix.

[0115] By using the inter-frame attention weight matrix, the features of each frame in the time series are reconstructed with weights to obtain the temporal weighted features of each joint;

[0116] The temporal weighted features of each joint are combined to obtain temporal weighted skeleton features.

[0117] In this embodiment, the method introduces an improved time-aware Transformer structure, which independently assigns weighted attention to each joint in the frame sequence to capture local temporal changes, and introduces a dual-center attention mechanism based on geometric center frames and peak frames at the global level, thereby better capturing the body dynamic features in the time dimension.

[0118] The input sequence is the spatial features obtained by VAF fusion. .

[0119] The Joint-Weighted Attention Mechanism (JWA) employs an inter-frame attention mechanism to independently process the evolution of each joint in the temporal dimension, reducing the noise impact caused by cross-joint feature aliasing.

[0120] After extracting features at the spatial level using MO-GCN and GOA and unifying them using VAF, a continuous spatial feature sequence is obtained. To facilitate subsequent temporal modeling, the feature sequences of all frames are concatenated along the time dimension, denoted as S(j).

[0121] To learn the dependencies between frames, a joint-level inter-frame attention weight matrix is ​​defined. As shown in equation (13), S(j) represents the feature change of the j-th joint throughout the entire frame sequence. It is a learnable temporal correlation mapping matrix. is the inter-frame attention weight matrix, representing the relative importance of each frame at different joints j.

[0122] (13)

[0123] After obtaining the inter-frame attention weight matrix, the features of the joint in the time series are reconstructed by weighted summation to obtain new features that integrate information of the joint in different frames. , expressed as equation (14). This represents the contribution weight of frame t in the time dimension when updating the j-th joint feature in frame i. This represents the feature at joint j in frame t.

[0124] (14)

[0125] Repeat the above process for all joints to obtain the complete temporally weighted skeleton features on frame i. It will be fed as input into the subsequent dual-center multi-head attention module.

[0126] In some embodiments, step S104 above, which involves using a dual-center multi-head attention mechanism to model the global temporal dependency of the temporally weighted skeleton features to obtain a feature sequence that integrates spatiotemporal dynamic information, specifically includes:

[0127] From the time series corresponding to the temporally weighted skeleton features, determine the geometric center frame and the action peak frame, and calculate the time distance between the current frame and the geometric center frame and the action peak frame;

[0128] Based on the time distance and the learnable decay rate parameter, the time decay weights for the geometric center and the peak center are calculated respectively.

[0129] By using learnable gating coefficients, the time decay weights of the geometric center and the peak center are dynamically fused to generate a comprehensive scaling vector;

[0130] By employing a dual-center multi-head attention mechanism that integrates comprehensive scaling vectors, a feature sequence incorporating spatiotemporal dynamic information is calculated.

[0131] Furthermore, the calculation of the feature sequence fused with spatiotemporal dynamic information through the dual-center multi-head attention mechanism of the integrated scaling vector specifically includes:

[0132] Initialize query, key, and value vectors based on time-weighted skeleton features;

[0133] By incorporating a comprehensive scaling vector into multi-head attention computation, the similarity matrix of the query and key is modulated to obtain modulated attention weights.

[0134] The output of each attention head is obtained by calculating based on the modulated attention weights and value vectors;

[0135] By concatenating the outputs of all attention heads and performing linear projection, a feature sequence incorporating spatiotemporal dynamic information is obtained.

[0136] In this embodiment, after the joint-weighted attention module extracts local temporal features, the method further employs a dual-center multi-head attention mechanism (DC-MHA) to capture the dependencies between global body features in the sequence.

[0137] The input is the time-weighted skeleton features output by the JWA module. Set during initialization Calculate the relationship between each position and the geometric center frame. Peak frames The distance between them. This represents the time interval between the current frame and the two types of center frames, the geometric center frame. Midpoint frame and peak frame taken from the sequence Based on the comparison of the change amplitude of each joint feature in adjacent frames, the frame with the largest change amplitude is selected as the peak frame. For the current frame i, the time distance formula is defined as equation (15):

[0138] (15)

[0139] To minimize the weight of frames farther from the two centers in temporal attention, two sets of learnable scaling factor parameters are defined. and These represent the decay rates of the two types of centers. Represents learnable time-mapping parameters that adjust the time distance scale under different sequence rhythms to control the decay rate. Time decay weights. and Defined as in equation (16), it represents the degree of attenuation of each frame relative to its respective center frame in time.

[0140] (16)

[0141] Next, the time scaling weights of the two centers are combined. and Introducing learnable gating coefficients The resulting scaling vector is obtained. , representing how much attention should be given to the i-th frame, as in equation (17):

[0142] (17)

[0143] Based on this, a dual-center attention calculation function is constructed, which is to use a function based on ordinary temporal attention calculation. The zoom-in keyframe is defined during initialization. , where G is the scaling factor for the feature dimension D, used to stabilize the gradient in attention computation. Single-head, dual-center temporal augmentation features. Defined as in equation (18):

[0144] (18)

[0145] Finally, the output features of each attention head are concatenated to obtain the overall output. As shown in equation (19), where, This represents the feature fusion operation, where n is the number of attention heads. This is a multi-head attention weight matrix. This represents a single bicentric attention head.

[0146] (19)

[0147] The output features obtained after the above calculations This feature, as the final output of the time modeling stage, is used to predict the coordinates of key points in the 3D human body.

[0148] Reference Figure 3 This paper demonstrates a detailed process for modeling global temporal dependencies in temporally weighted skeleton features using a dual-center multi-head attention mechanism. First, the input temporally weighted skeleton features undergo a dual-center position vector computation step, considering guiding information from the geometric center frame and the action peak frame. Next, the data dimensions are adjusted through a reshape operation, and query (Q), key (K), and value (V) vectors are generated using multiple linear transformations. Then, the dual-center multi-head attention mechanism modulates the query-key relevance through a comprehensive scaling vector, which combines the temporal decay weights of the geometric center and peak center, and dynamically fuses them using gating coefficients. The modulated attention weights and value vectors are then used to calculate the outputs of each attention head. Finally, these are concatenated and processed by a multilayer perceptron (MLP) to output a feature sequence incorporating spatiotemporal dynamic information. (Appendix) Figure 3 This paper describes how a dual-center multi-head attention mechanism can effectively model global temporal dependencies in temporally weighted skeleton features. By combining guidance information from geometric centers and action peak frames, attention weights are dynamically modulated to generate more accurate and robust spatiotemporal dynamic feature sequences.

[0149] In some embodiments, step S105 above, where the feature sequence based on fused spatiotemporal dynamic information is used to predict the three-dimensional human pose coordinate sequence through a regression network, specifically includes:

[0150] The feature sequence that integrates spatiotemporal dynamic information is input into a multilayer perceptron, and nonlinear transformation and feature reconstruction are performed to obtain the reconstructed feature sequence.

[0151] The reconstructed feature sequence is input into the fully connected layer of the regression network. The fully connected layer maps the reconstructed feature sequence into three-dimensional spatial coordinates and outputs a three-dimensional human pose coordinate sequence in a unified coordinate system.

[0152] In this embodiment, after processing by the DC-MHA module, a feature sequence fusing multi-view spatial features and temporal dynamic information is obtained. Next, the features are nonlinearly transformed and reconstructed using a multilayer perceptron (MLP), and then input into a regression head. The coordinate position of each joint in three-dimensional space is predicted through a fully connected mapping, and the three-dimensional pose sequence of human key points in a unified coordinate system is output.

[0153] Reference Figure 4An embodiment of the present invention provides a multi-view three-dimensional human pose estimation system 4, which specifically includes:

[0154] The spatial feature extraction module 401 is used to input a multi-view two-dimensional keypoint sequence, and based on the multi-view two-dimensional keypoint sequence, to extract the weighted spatial features of each frame under each view using a multi-order graph attention mechanism.

[0155] The spatial feature fusion module 402 is used to fuse the weighted spatial features and corresponding two-dimensional detection confidence based on each viewpoint through a viewpoint adaptive fusion mechanism to generate unified multi-view fused spatial features.

[0156] The temporal feature extraction module 403 is used to perform temporal modeling on multi-view fused spatial features. It extracts the local temporal features of each joint through the joint weighted attention module to obtain temporal weighted skeleton features.

[0157] The temporal feature fusion module 404 is used to model the global temporal dependency of temporally weighted skeleton features using a dual-center multi-head attention mechanism to obtain a feature sequence that integrates spatiotemporal dynamic information.

[0158] The coordinate sequence output module 405 is used to predict the three-dimensional human posture coordinate sequence through a regression network based on the feature sequence of fused spatiotemporal dynamic information.

[0159] It is understandable that, such as Figure 1 The content of the multi-view 3D human pose estimation method embodiments shown are all applicable to the embodiments of this multi-view 3D human pose estimation system. The specific functions implemented by the embodiments of this multi-view 3D human pose estimation system are the same as those shown below. Figure 1 The multi-view 3D human pose estimation method shown in the embodiment is the same, and the beneficial effects achieved are the same as those shown. Figure 1 The beneficial effects achieved by the multi-view 3D human pose estimation method embodiment shown are also the same.

[0160] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0161] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0162] Reference Figure 5 The present invention also provides a computer device 5, including: a memory 502 and a processor 501, and a computer program 503 stored in the memory 502. When the computer program 503 is executed on the processor 501, it implements the multi-view three-dimensional human pose estimation method as described in any of the above methods.

[0163] The computer device 5 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will understand that... Figure 5 The computer device 5 is merely an example and does not constitute a limitation on the computer device 5. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0164] The processor 501 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0165] In some embodiments, the memory 502 may be an internal storage unit of the computer device 5, such as a hard disk or memory of the computer device 5. In other embodiments, the memory 502 may be an external storage device of the computer device 5, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 5. Furthermore, the memory 502 may include both internal and external storage units of the computer device 5. The memory 502 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 502 can also be used to temporarily store data that has been output or will be output.

[0166] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-view three-dimensional human pose estimation method as described in any of the above methods.

[0167] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0168] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0169] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A multi-view three-dimensional human pose estimation method, characterized by, The method specifically includes: Input a multi-view 2D keypoint sequence, and based on the multi-view 2D keypoint sequence, use a multi-order graph attention mechanism to extract the weighted spatial features of each frame under each view. Based on the weighted spatial features of each perspective and the corresponding two-dimensional detection confidence, a perspective adaptive fusion mechanism is used to fuse them and generate unified multi-view fused spatial features. Temporal modeling is performed on the multi-view fused spatial features, and local temporal features of each joint are extracted through the joint-weighted attention module to obtain temporal-weighted skeleton features; A dual-center multi-head attention mechanism is used to model the global temporal dependency of temporally weighted skeleton features, resulting in a feature sequence that integrates spatiotemporal dynamic information. Based on feature sequences that integrate spatiotemporal dynamic information, a 3D human pose coordinate sequence is predicted using a regression network. Specifically, the step of extracting weighted spatial features for each frame from each viewpoint based on a multi-view two-dimensional keypoint sequence and utilizing a multi-order graph attention mechanism includes: Based on multi-view 2D keypoint sequences, for each frame of each view, multi-level graph convolutional features with different receptive ranges are extracted through a multi-level graph convolutional network. Based on multi-order graph convolution features, the concatenation is performed along the order dimension to obtain the order concatenation features of each frame. Based on the order-separated features, the attention weights of each order feature are calculated through a graph-order attention mechanism; Based on attention weights, weighted summation and nonlinear activation are performed on multi-order graph convolutional features to generate weighted spatial features. The method of using a dual-center multi-head attention mechanism to model the global temporal dependency of temporally weighted skeleton features to obtain a feature sequence that integrates spatiotemporal dynamic information specifically includes: From the time series corresponding to the temporally weighted skeleton features, determine the geometric center frame and the action peak frame, and calculate the time distance between the current frame and the geometric center frame and the action peak frame; Based on the time distance and the learnable decay rate parameter, the time decay weights for the geometric center and the peak center are calculated respectively. By using learnable gating coefficients, the time decay weights of the geometric center and the peak center are dynamically fused to generate a comprehensive scaling vector; By employing a dual-center multi-head attention mechanism that integrates comprehensive scaling vectors, a feature sequence incorporating spatiotemporal dynamic information is calculated.

2. The method of claim 1, wherein, The weighted spatial features based on each viewpoint and the corresponding two-dimensional detection confidence are fused through a viewpoint adaptive fusion mechanism to generate unified multi-view fused spatial features, specifically including: Channel normalization is performed on the weighted spatial features from each perspective to obtain normalized spatial features; Based on the normalized spatial features and the corresponding two-dimensional detection confidence, the reliability score of each viewpoint at each joint is calculated. Based on normalized spatial features, calculate the consistency score of each viewpoint relative to other viewpoints at each joint. Based on reliability and consistency scores, a learnable fusion weight calculation function is used to generate comprehensive weights for each perspective at each joint. The overall weights are normalized to obtain the contribution ratio of each perspective at each joint; Based on the contribution ratio, the normalized spatial features of all perspectives are weighted and summed to generate unified multi-perspective fused spatial features.

3. The method of claim 2, wherein, The method of calculating the reliability score of each viewpoint at each joint based on normalized spatial features and corresponding two-dimensional detection confidence levels specifically includes: For the feature vector of each joint in the normalized spatial features, calculate its L2 norm to obtain the feature intensity of the joint under the current view. The detection confidence of the corresponding joint in the two-dimensional detection confidence is obtained. Based on the feature intensity and the detection confidence, the results are fused by a linear combination function with learnable coefficients to obtain the fusion result. The fusion result is input into the activation function to generate a reliability score.

4. The method of claim 2, wherein, The calculation of consistency scores for each viewpoint relative to other viewpoints at each joint, based on normalized spatial features, specifically includes: From the normalized spatial features of the current perspective, obtain the first feature vector at the specified frame and the specified joint; Obtain the second feature vector at the same frame and the same joint from the normalized spatial features of any other viewpoint; Calculate the cosine similarity between the first feature vector and the second feature vector, and use the cosine similarity as a measure of the consistency between the current viewpoint and any other viewpoint at the specified joint. Integrate the consistency metrics of the current viewpoint with all other viewpoints at a specified joint to form a consistency metric group; The consistency score at a specified joint is obtained by averaging the consistency metrics group.

5. The method of claim 1, wherein, The process of performing temporal modeling on multi-view fused spatial features, and extracting local temporal features of each joint through a joint-weighted attention module to obtain temporally weighted skeleton features, specifically includes: For each joint, the multi-view fused spatial features are spliced ​​together in the time dimension to form the time series features of that joint; Based on time series features, the inter-frame attention weight matrix for each joint is calculated using a learnable temporal correlation mapping matrix. By using the inter-frame attention weight matrix, the features of each frame in the time series are reconstructed with weights to obtain the temporal weighted features of each joint; The temporal weighted features of each joint are combined to obtain temporal weighted skeleton features.

6. The method of claim 5, wherein, The feature sequence fused with spatiotemporal dynamic information is calculated through a dual-center multi-head attention mechanism that integrates the comprehensive scaling vector, specifically including: Initialize query, key, and value vectors based on time-weighted skeleton features; By incorporating a comprehensive scaling vector into multi-head attention computation, the similarity matrix of the query and key is modulated to obtain modulated attention weights. The output of each attention head is obtained by calculating based on the modulated attention weights and value vectors; By concatenating the outputs of all attention heads and performing linear projection, a feature sequence incorporating spatiotemporal dynamic information is obtained.

7. The method of claim 1, wherein, The feature sequence based on fused spatiotemporal dynamic information is used to predict the three-dimensional human pose coordinate sequence through a regression network, specifically including: The feature sequence that integrates spatiotemporal dynamic information is input into a multilayer perceptron, and nonlinear transformation and feature reconstruction are performed to obtain the reconstructed feature sequence. The reconstructed feature sequence is input into the fully connected layer of the regression network. The fully connected layer maps the reconstructed feature sequence into three-dimensional spatial coordinates and outputs a three-dimensional human pose coordinate sequence in a unified coordinate system.

8. A multi-view three-dimensional human pose estimation system, characterized by, The system specifically includes: The spatial feature extraction module is used to input multi-view two-dimensional keypoint sequences. Based on the multi-view two-dimensional keypoint sequences, the weighted spatial features of each frame under each view are extracted using a multi-order graph attention mechanism. The spatial feature fusion module is used to fuse the weighted spatial features and corresponding two-dimensional detection confidence scores from each viewpoint through a viewpoint adaptive fusion mechanism to generate unified multi-view fused spatial features. The temporal feature extraction module is used to perform temporal modeling of multi-view fused spatial features. It extracts the local temporal features of each joint through the joint weighted attention module to obtain temporal weighted skeleton features. The temporal feature fusion module is used to model the global temporal dependency of temporally weighted skeleton features using a dual-center multi-head attention mechanism, so as to obtain a feature sequence that integrates spatiotemporal dynamic information. The coordinate sequence output module is used to predict the three-dimensional human pose coordinate sequence through a regression network based on the feature sequence fused with spatiotemporal dynamic information. Specifically, the step of extracting weighted spatial features for each frame from each viewpoint based on a multi-view two-dimensional keypoint sequence and utilizing a multi-order graph attention mechanism includes: Based on multi-view 2D keypoint sequences, for each frame of each view, multi-level graph convolutional features with different receptive ranges are extracted through a multi-level graph convolutional network. Based on multi-order graph convolution features, the concatenation is performed along the order dimension to obtain the order concatenation features of each frame. Based on the order-separated features, the attention weights of each order feature are calculated through a graph-order attention mechanism; Based on attention weights, weighted summation and nonlinear activation are performed on multi-order graph convolutional features to generate weighted spatial features. The method of using a dual-center multi-head attention mechanism to model the global temporal dependency of temporally weighted skeleton features to obtain a feature sequence that integrates spatiotemporal dynamic information specifically includes: From the time series corresponding to the temporally weighted skeleton features, determine the geometric center frame and the action peak frame, and calculate the time distance between the current frame and the geometric center frame and the action peak frame; Based on the time distance and the learnable decay rate parameter, the time decay weights for the geometric center and the peak center are calculated respectively. By using learnable gating coefficients, the time decay weights of the geometric center and the peak center are dynamically fused to generate a comprehensive scaling vector; By employing a dual-center multi-head attention mechanism that integrates comprehensive scaling vectors, a feature sequence incorporating spatiotemporal dynamic information is calculated.

Images