Occlusion-aware vertex reasoning and semantic enhancement: a 3D pose estimation method and system

Abstract

This application belongs to the field of computer vision technology and discloses a method and system for 3D pose estimation based on occlusion-aware vertex inference and semantic enhancement. The method includes: receiving a 2D pose sequence containing occluded joints; using an occlusion-aware vertex inference network to complete the position of the occluded joints; and generating adaptive confidence scores for each joint based on the joint motion trajectory and initial scores using a confidence score generator to obtain the completed 2D pose sequence and corresponding confidence scores; using a semantic feature enhancement network to denoise and optimize the 3D pose assumption sampled from a Gaussian distribution. During the training phase, this network extracts joint semantic features in parallel through a Transformer encoder and a Transformer-like convolutional network, and uses a semantic feature matching module to align the two types of features to enhance the semantic representation capability of the Transformer encoder; during the testing phase, only the trained Transformer encoder is used to iteratively optimize the 3D pose assumption, and finally outputs an accurate and complete 3D human pose.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a three-dimensional pose estimation method and system with occlusion-aware vertex reasoning and semantic enhancement. Background Technology

[0002] 3D human pose estimation (3DHPE) aims to recover the 3D spatial coordinates of human joints from 2D images or videos, and is a fundamental and crucial task in the field of computer vision. It has broad application prospects in human-computer interaction, action recognition, virtual reality, and medical rehabilitation. Currently, most mainstream 3D human pose estimation methods adopt a "2D-to-3D uplifting" framework, which involves first obtaining a 2D keypoint sequence using a mature 2D pose detector, and then uplifting this sequence to 3D space.

[0003] However, in real-world applications, the human body is often occluded by objects, other people, or its own limbs, causing 2D keypoint detectors to fail to accurately locate all relevant nodes, resulting in joint loss or coordinate errors. This incomplete and noisy 2D pose input poses a significant challenge to the subsequent 3D pose enhancement process. Although some temporal information-based methods (such as those using the Transformer architecture) can infer occluded joints using inter-frame correlations, they often implicitly handle occlusion, lacking explicit modeling of joint visibility. Furthermore, these methods typically treat all joints equally, failing to establish a reliable scoring mechanism to distinguish visible joints from inferred occluded joints. This allows low-quality inferred joints to negatively impact feature learning, leading to decreased model performance. On the other hand, existing methods lack sufficient deep semantic information mining of human joint motion, such as joint co-movement patterns and motion smoothness, which limits the model's ability to generate high-quality, natural motion sequences.

[0004] Therefore, how to design a method that can explicitly handle occlusion, assign reasonable confidence levels to joints of different qualities, and make full use of joint semantic information to enhance feature representation is an urgent problem to be solved in the field of 3D human pose estimation. Summary of the Invention

[0005] The main objective of this invention is to provide a 3D pose estimation method and system based on occlusion-aware vertex reasoning and semantic enhancement, aiming to solve the problem of low 3D pose estimation accuracy in the prior art when dealing with severely occluded scenes due to the lack of a reliability assessment mechanism for occluded joints and insufficient utilization of joint semantic information.

[0006] In a first aspect, the present invention provides a three-dimensional pose estimation method with occlusion-aware vertex reasoning and semantic enhancement, comprising the following steps:

[0007] Obtain the initial 2D pose sequence of the occluded joints;

[0008] The initial 2D pose sequence is processed by an occlusion-aware vertex inference network. Based on temporal context information, the position inference of occluded joints in the initial 2D pose sequence is performed to generate a preliminary completed 2D pose sequence. An initial confidence score is assigned to each joint in the preliminary completed 2D pose sequence. The preliminary completed 2D pose sequence and the initial confidence score are input into a confidence score generator to generate a final confidence score that reflects the true quality of each joint, so as to obtain a complete 2D joint input containing the coordinates of each joint and the final confidence score.

[0009] Based on the complete two-dimensional joint input, the semantic feature enhancement network is used to denoise and optimize the noisy three-dimensional pose assumption. The semantic feature enhancement network includes a Transformer encoder branch and a convolutional network encoder branch during the training phase. The first joint semantic features and the second joint semantic features are extracted in parallel through the two branches. The semantic alignment loss between the first joint semantic features and the second joint semantic features is calculated using a semantic feature matching mechanism to supervise and enhance the semantic feature representation capability of the Transformer encoder.

[0010] Using the trained Transformer encoder, the sampled three-dimensional pose assumptions are iteratively denoised to output the final complete three-dimensional human pose sequence.

[0011] As an optional implementation of the first aspect of this application, the step of performing position reasoning on occluded joints in the initial two-dimensional pose sequence based on temporal context information to generate a preliminarily completed two-dimensional pose sequence specifically includes: traversing each frame and each joint in the initial two-dimensional pose sequence; determining whether the current joint is an occluded joint; if the current joint is an occluded joint, then performing the following sub-steps: searching bidirectionally along the time dimension, with the current frame as the center, towards past and future frames for the nearest unoccluded joint corresponding to the occluded joint; and determining the position coordinates of the occluded joint according to one of the following rules based on the search results. The completion process is as follows: Rule 1: If a corresponding unoccluded joint is found in both past and future frames equidistant from the current frame, the position coordinates of the occluded joint are determined as the midpoint between the coordinates of these two unoccluded joints. Rule 2: If a corresponding unoccluded joint is found only in a past frame, the position coordinates of the occluded joint are determined as the position coordinates of the corresponding joint in the past frame. Rule 3: If a corresponding unoccluded joint is found only in a future frame, the position coordinates of the occluded joint are determined as the position coordinates of the corresponding joint in the future frame. All completed joints are combined with the original unoccluded joints to form the preliminary completed two-dimensional pose sequence.

[0012] As an optional implementation of the first aspect of this application, an initial confidence score is assigned to each joint in the initially completed two-dimensional pose sequence, and the initially completed two-dimensional pose sequence and the initial confidence scores are input to a confidence score generator to generate a final confidence score reflecting the true quality of each joint, so as to obtain a complete two-dimensional joint input containing the coordinates of each joint and the final confidence score. Specifically, this includes: assigning an initial confidence score to each joint based on the inter-frame distance between the searched unoccluded joints and the current frame. The initial confidence score for unoccluded joints is 1.00, while the initial confidence score for occluded joints is reduced based on the search distance; the preliminarily completed two-dimensional pose sequence is then... and initial confidence scores for all joints As input, a confidence score generator consisting of two convolutional functions is fed in. This confidence score generator models the correlation between the joint coordinates and the initial confidence score, and outputs the confidence score. , is defined by the following formula: ;in, For confidence score generator, For the first n Frame number i Confidence scores for each joint; the preliminarily completed two-dimensional pose sequence With the confidence score By splicing the components together, the complete two-dimensional joint input is obtained. .

[0013] As an optional implementation of the first aspect of this application, the convolutional network encoder branch is a Transformer-like convolutional network Conv2DNet, which is composed of multiple cascaded Conv2D blocks; before inputting features into the Conv2DNet, the temporal features are first divided into G groups along the time dimension and reshaped into a two-dimensional feature map resembling an image; the structure of each Conv2D block is defined by the following formula: ; ;in, Indicates feature input, Indicates intermediate variables. Indicates batch standardization. This represents a multilayer perceptron. This represents the multi-scale convolutional attention mechanism.

[0014] As an optional implementation of the first aspect of this application, the core components of the multi-scale convolutional attention mechanism are grouped double convolutional (GDC) and double normalized densities (DN). The specific implementation process of the multi-scale convolutional attention mechanism includes: designing a learnable kernel function. and The key K and value V are required for the convolutional attention mechanism; the grouped dual convolutional GDC is implemented based on strip convolution, including learning the correlation between adjacent frames through horizontal convolution and learning the correlation between distant frames through vertical convolution; to achieve multi-scale context capture, the strip convolution uses multiple sets of convolution kernels of different sizes, with kernel sizes of respectively , , The computational process of the multi-scale convolutional attention mechanism is defined by the following formula: ;in, , , , Indicates the kernel size. Indicates a channel. Indicates the number of learnable parameters. For activation function, This represents matrix multiplication.

[0015] As an optional implementation of the first aspect of this application, the step of calculating the semantic alignment loss between the first joint semantic features and the second joint semantic features using a semantic feature matching mechanism specifically includes: obtaining the first joint semantic features output by the Transformer encoder branch. and the second joint semantic features output by the encoder branch of the convolutional network ; through a hyperparameter Channel distillation function Process the semantic features of the first joint respectively and the second joint semantic features The feature activation values ​​are converted into a probability distribution along the channel dimension to eliminate the scale difference between the two branches. The channel distillation function is defined as follows: ;in, For channel indexing, Let N be the spatiotemporal position of the joint, and N be the number of input frames. It refers to the number of joints in the human body. This represents the characteristics of the c-th channel. The feature represents the c-th channel and the th i The values ​​at each position are calculated; the KL divergence between the two probability distributions is used as the semantic alignment loss. Its definition is as follows: ;in This indicates that the feature of the Transformer encoder branch output contains all features in the c-th channel. i The value of the position, This indicates that the features output by the encoder branch of the convolutional network contain all features in the c-th channel. i The value of the position.

[0016] As an optional implementation of the first aspect of this application, the method's total loss function during the training phase Loss from 3D pose reconstruction and the semantic alignment loss The weighted composition is defined as follows: ;in, , For true 3D pose, The denoised 3D pose reconstructed by the Transformer encoder. , These are the weighting coefficients.

[0017] As an optional implementation of the first aspect of this application, the step of using the trained Transformer encoder to iteratively denoise multiple sampled 3D pose hypotheses and output the final complete 3D human pose sequence specifically includes: during the testing phase, the convolutional network encoder branch is no longer used; H initial 3D pose hypotheses are randomly sampled; and within the total time step T, [the following steps are performed]. Each iteration optimizes the current 3D pose hypothesis by denoising it using the Transformer encoder to generate a better hypothesis; The time step of each iteration decreases from T to 1, following the following decreasing pattern: ;Finish After each iteration, the final optimized complete three-dimensional human pose sequence is output.

[0018] Secondly, embodiments of this application provide a 3D pose estimation system with occlusion-aware vertex reasoning and semantic enhancement, comprising:

[0019] The data acquisition module is used to acquire the initial two-dimensional pose sequence of joints that are occluded;

[0020] The occlusion-aware vertex inference module is used to process the initial two-dimensional pose sequence through the occlusion-aware vertex inference network, and perform position inference on the occluded joints in the initial two-dimensional pose sequence based on temporal context information to generate a preliminary completed two-dimensional pose sequence.

[0021] The confidence score generation module is used to assign an initial confidence score to each joint in the initially completed two-dimensional pose sequence, and input the initially completed two-dimensional pose sequence and the initial confidence score to the confidence score generator to generate a final confidence score that reflects the true quality of each joint, so as to obtain a complete two-dimensional joint input containing the coordinates of each joint and the final confidence score.

[0022] The semantic feature enhancement module is used to denoise and optimize the noisy 3D pose assumption based on the complete 2D joint input through the semantic feature enhancement network. The semantic feature enhancement network includes a Transformer encoder branch and a convolutional network encoder branch during the training phase. The two branches extract the semantic features of the first joint and the semantic features of the second joint in parallel, and use the semantic feature matching mechanism to calculate the semantic alignment loss between the semantic features of the first joint and the semantic features of the second joint, so as to supervise and enhance the semantic feature representation ability of the Transformer encoder.

[0023] The three-dimensional pose output module is used to perform iterative denoising processing on multiple sampled three-dimensional pose assumptions using the trained Transformer encoder, and output the final complete three-dimensional human pose sequence.

[0024] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0025] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] 1. An occlusion-aware vertex inference network is proposed, which can explicitly infer and complete occluded joints. Through an adaptive confidence score generation mechanism, the reliability of each joint is quantified, effectively reducing the negative impact of unreliable joint data on model learning.

[0028] 2. A semantic feature enhancement network is proposed. Through an innovative dual-branch (Transformer+Conv2DNet) training architecture and semantic feature matching module, it effectively integrates the advantages of Transformer in global context modeling and CNN in local detail capture, significantly enhancing the model's ability to learn semantic information of human joints and improving the accuracy of the final pose estimation.

[0029] 3. The designed Transformer-like Conv2DNet can efficiently process time-series data, forms a good synergy with the Transformer branch, and can be removed during the testing phase, ensuring high efficiency of model inference. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall framework of the occlusion-aware vertex reasoning and semantic enhancement three-dimensional pose estimation method (VRSFE-Net) proposed in this invention;

[0031] Figure 2 This is a schematic diagram of the occlusion reasoning (OR) mechanism in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the confidence score generation mechanism (CSG) in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the input feature grouping method and multi-scale convolutional attention mechanism (MCA) of Conv2DNet in this embodiment of the invention;

[0034] Figure 5 This is a schematic diagram of the denoiser and Conv2DNet structure in the Semantic Feature Enhancement Network (SFE-Net) in an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of the structure of a 3D pose estimation system with occlusion-aware vertex reasoning and semantic enhancement provided in an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0038] Example 1

[0039] Please see Figure 1 This invention proposes a 3D pose estimation method based on occlusion-aware vertex reasoning and semantic enhancement, the core of which is the VRSFE-Net network. This network receives an initial 2D pose sequence containing N frames. As input, where The pose is estimated by a 2D pose estimator, where N is the number of input frames. This refers to the number of joints in the human body. To effectively simulate occlusion scenarios, multiple joints are randomly selected and their coordinates are set to (0, 0). This method aims to output a complete 3D human pose sequence. ,in VRSFE-Net consists of two core components: the occlusion-aware vertex reasoning network (VR-Net) and the semantic feature enhancement network (SFE-Net). Specifically: (i) The occlusion-aware vertex reasoning network (VR-Net) includes an occlusion reasoning mechanism (OR), which uses temporal context to complete occluded joints and proposes an occlusion-aware confidence score generation mechanism (CSG) to generate a confidence score for each joint, thereby reducing the impact of incorrect joints on the model; (ii) The semantic feature enhancement network (SFE-Net) consists of a Transformer-based denoiser and a Transformer-like convolutional network (Conv2DNet), aiming to improve the representational ability of features by aligning the semantic features extracted by these two components.

[0040] (a) Occlusion-aware vertex inference network (VR-Net)

[0041] In 2D-to-3D pose enhancement methods, the quality of the 2D pose sequence significantly impacts the performance of the 3D pose estimation model. This is especially true in scenes with severe occlusion, where 2D keypoints may be lost or incorrectly estimated. To address this issue, this invention proposes the VR-Net model, such as... Figure 1 As shown in the upper part, the model uses the occlusion reasoning mechanism (OR) to fill in the missing key points and proposes an occlusion-aware confidence score generation mechanism (CSG) to generate a score for each key point, thereby reducing the impact of key points with reasoning errors on the model.

[0042] 1. Occlusion reasoning mechanism (OR)

[0043] Since the joint information in the context can provide strong clues for determining the position of joints in the current frame, this invention proposes to use the nearest corresponding unoccluded joints in past and future frames to complete the position of occluded joints in the current frame, such as... Figure 2 As shown.

[0044] Iterate through each frame and each joint in the initial 2D pose sequence, and determine whether the current joint is an occluded joint. If the current joint is an occluded joint... Then execute the following sub-steps:

[0045] Along the time dimension, with the current frame as the center, a bidirectional search is performed towards past and future frames for the nearest unoccluded joint corresponding to the occluded joint. Based on the search results, one of the following rules is used to determine the position coordinates of the occluded joint to complete it:

[0046] Rule 1: If a corresponding unoccluded joint is found in both past and future frames equidistant from the current frame (e.g., ... Figure 2 If the red joint is obscured, then the position coordinates of the obscured joint are determined as the midpoint of the coordinates of the two unobscured joints.

[0047] Rule 2: If the corresponding unoccluded joint is only found in past frames (e.g.) Figure 2 If the yellow joint is obscured, then the position coordinates of the obscured joint are determined as the position coordinates of the corresponding joint in the past frame.

[0048] Rule 3: If the corresponding unoccluded joint is only found in future frames (e.g., ...) Figure 2 If the blue joint is obscured, then the position coordinates of the obscured joint are determined as the position coordinates of the corresponding joint in the future frame.

[0049] Finally, all the completed joints are combined with the original unoccluded joints to form the preliminary completed two-dimensional pose sequence.

[0050] 2. Confidence Score Generation Mechanism (CSG)

[0051] While the interpolation methods described above can infer the locations of occluded joints, the reliability of these locations varies. Therefore, a confidence score needs to be assigned to each joint to reduce the impact of erroneous joints on model performance. To this end, the inter-frame distance between the searched unoccluded joints and the current frame is used to assign an initial confidence score to each joint. .

[0052] Specifically, the scheme first presets a time-series window of a fixed size, covering... A historical frame and The system searches for the nearest corresponding unoccluded joint within this window of future frames. For each historical frame searched, the confidence score is reduced from 1.00. Each time a future frame is searched, a deduction is made. Points. It is worth noting that if it involves Figure 2 The "yellow joint" or "blue joint" situations shown will result in an additional deduction from the calculated score. or In addition, the confidence score for all unoccluded joints is fixed at 1.00.

[0053] However, the above scheme has three inherent flaws: First, using the same scoring rules for fast and slow movements significantly weakens the joint correlation between context frames when movement changes drastically. Second, body parts that are far apart may exhibit similar movement states, making it unreasonable to assign scores solely based on distance. Finally, the reasoning for occluded joints is an iterative process; using joints already completed in previous iterations to deduce the currently occluded joint leads to error accumulation, but the existing scoring mechanism does not consider the cumulative effect of scores on the searched joints.

[0054] To address these issues, this invention proposes a method based on initial confidence scores. An occlusion-aware (CSG) mechanism, incorporating contextual temporal information, generates final confidence scores for each joint. These scores serve as the basis for joint weights in model optimization, effectively reducing the impact of faulty joints on model performance.

[0055] The architecture of CSG is as follows Figure 3 As shown, for the first n The first frame i Each joint, CSG completes the 2D keypoint sequence. and their corresponding initial confidence scores As input, where It may contain a significant amount of error. CSG includes a confidence score generator that not only captures the joint-level spatiotemporal dependencies in 2D pose from the input data, but also implicitly models the correlation between 2D keypoints and confidence scores under the constraint of initial confidence scores, thereby generating new scores that accurately reflect the quality of the keypoints. Specifically, the confidence score generator first uses a linear projection function... The keypoint coordinates are mapped to a high-dimensional feature space, and then a two-layer convolution function is used to learn the spatiotemporal features and the correlation between keypoints and scores, finally outputting the confidence score. This process can be formally represented by formula (1):

[0056]

[0057] in, It is a confidence score generator. It is the first n Frame number i Confidence scores for each joint. (Through splicing) and confidence score This can yield a new 2D joint input. .

[0058] (ii) Semantic Feature Enhancement Network (SFE-Net)

[0059] This invention employs MixSTE, a Transformer-based network, as the backbone network for denoising to learn joint motion trajectories. While Transformer excels at extracting long-range contextual information, it has limitations in capturing fine-grained local details. Existing methods typically address this limitation by adding extra branches or modules, which increases computational cost and reduces inference speed. Therefore, this invention proposes a Semantic Feature Enhancement Network (SFE-Net), the structure of which is as follows: Figure 1 The lower half of the diagram is shown. SFE-Net utilizes two branches—a denoiser (Transformer encoder) and a convolutional network (CNN) encoder—to learn spatiotemporal features. Its core component, the Semantic Feature Matching (SFM) module, facilitates the learning of rich semantic information by aligning the spatiotemporal features extracted by both. To ensure that the model's inference speed is not affected by the added branches, only the denoiser is used to optimize the hypotheses during the testing phase. Given that Transformer and CNN excel at capturing global and local features, respectively, forcibly aligning the features extracted by the standard CNN and Transformer encoders would degrade the learning quality. Therefore, this invention designs the convolutional network encoder as a Transformer-like Conv2DNet.

[0060] 1. Conv2DNet, a Transformer-like class

[0061] To ensure that Conv2DNet can learn rich global motion structure features and local details, Conv2DNet is designed based on the following two considerations:

[0062] (a) The Conv2D blocks that make up Conv2DNet are structurally similar to the Transformer encoder, which helps to better learn semantic information similar to that learned by the Transformer branch. In addition, a multi-scale convolutional attention mechanism (MCA) is used instead of a self-attention mechanism to ensure that multi-level features can be learned effectively. The Conv2D blocks can be described by Equations (2) and (3). (b) Starting from the structure of the input data of Conv2DNet, the input temporal data is adjusted to a feature map similar to an image to ensure that the multi-scale convolutional attention mechanism can capture long-distance dependencies between joints and local details at the same time.

[0063]

[0064]

[0065] in Indicates feature input, Indicates intermediate variables. Indicates batch standardization. It is a multilayer perceptron. Representing the multi-scale convolutional attention mechanism, Conv2DNet is composed of... It is composed of cascaded Conv2D blocks.

[0066] For a completed 2D keypoint sequence The system generates H noisy 3D pose hypotheses, which are then optimized using 2D keypoint sequences as conditions. First, input embeddings are used to map the 2D keypoints and 3D hypotheses to a high-dimensional feature space to obtain high-dimensional features. .Then, The semantic features of the first joint are obtained after processing by a Transformer encoder (denoiser) and a Convolutional Network encoder (Conv2DNet). Second joint semantic features For the Conv2DNet branch, to ensure that the multi-scale convolutional attention mechanism (MCA) can capture both details and long-range dependencies, features are... Divide the data into G groups along the time-series dimension and abstract them into image-like features. ,like Figure 4 As shown in (a) in the figure.

[0067] The structure diagram of MCA is as follows: Figure 4 As shown in (b), its core modules are Group Double Convolution (GDC) and Double Normalization (DN). To preserve the spatial structure of the feature map while controlling the number of parameters, GDC uses a learnable kernel to represent the features. Grouping features into feature blocks (local regions) fully considers the correlation within these local regions. To simultaneously capture global motion structure features and local details, GDC is implemented based on strip convolution: firstly, in... Horizontal convolution is performed in the horizontal direction to learn the correlation between adjacent frames, followed by vertical convolution in the vertical direction to learn the correlation between distant frames (the frame spacing is determined by the number of groups G). A softmax function is applied along the N / G dimensions, and L2 normalization is performed along the G dimension. To obtain multi-scale context from local to global, multi-scale convolutional kernels are used to implement strip convolution, with kernel sizes of [missing information]. , , Therefore, MCA can be defined by formula (4):

[0068]

[0069] in, , , , Indicates the kernel size. Indicates a channel. and It can be viewed as a key K and a value V in a Transformer. Indicates the number of learnable parameters. It is a convolution operation.

[0070] 2. Semantic Feature Matching Module (SFM)

[0071] To ensure that the model's inference speed is not affected by the Conv2DNet branch, only the denoiser (Transformer branch) is used to optimize noisy 3D poses during the testing phase. To enhance the denoiser's ability to extract global contextual information and local joint details, this invention proposes a semantic feature matching module (SFM). This module supervises the denoiser to learn high-quality global and local features by aligning the semantic features extracted by Conv2DNet and the denoiser.

[0072] The specific structure of Conv2DNet and the denoiser is as follows: Figure 5 As shown, the semantic features (spatiotemporal features) of the first joint are output respectively. Second joint semantic features To learn more complete detailed features, Conv2DNet employs a residual learning mechanism; simultaneously, to achieve more accurate semantic alignment, this invention is based on the channel distillation function. Define semantic alignment loss As shown in formulas (5) and (6):

[0073]

[0074]

[0075] Where the channel distillation function By converting feature activation values ​​into a probability distribution along the channel dimension, the scale difference between the Transformer encoder and Conv2DNet is effectively eliminated, thus facilitating subsequent alignment operations. It is a channel index. It refers to the spatiotemporal location of the joint. This represents the characteristics of the c-th channel. The feature represents the c-th channel and the th i The value at each position, This indicates that the feature of the Transformer encoder branch output contains all features in the c-th channel. i The value of the position, This indicates that the features output by the encoder branch of the convolutional network contain all features in the c-th channel. i The value of the position. As a hyperparameter controlling the smoothness of the feature distribution, its value is positively correlated with the degree of distribution softening: The larger the value, the flatter the feature distribution. Expectation It can achieve the minimum value, therefore, regardless of Is it big or small? These should all be small values. This effectively prevents the denoiser from being dominated by salient features, allowing it to learn subtle features more fully.

[0076] (III) Overall Training and Testing Process

[0077] 1. Overall training

[0078] This method employs the standard Denoising Diffusion Probability Model (DDPM) framework to perform end-to-end training of VRSFE-Net. For a given pair of samples... :

[0079] (a) to Randomly occlude m joints to obtain an occluded 2D pose. ,in Then, a new 2D pose is obtained using OR and CSG mechanisms. To ensure code reproducibility, a random seed is used to determine the occluded joints.

[0080] (b) From a time range A time step t is sampled, where T represents the total diffusion time step;

[0081] (c) Add Gaussian noise arrive Capable of generating noisy 3D poses. ;

[0082] (d) Extract joint-level semantic features using a denoiser and Conv2DNet respectively. and Moreover, using a noise denoiser to Reconstructing the noise-removed 3D pose using the current time step t as a condition. ;

[0083] (e) Calculation and Semantic alignment loss between ,as well as and The MSE loss between the parameters is used to optimize the model parameters. Defined as in Equation (7), the overall loss for training the model is shown in Equation (8):

[0084]

[0085]

[0086] in, For true 3D pose, The denoised 3D pose reconstructed by the Transformer encoder. , These are the weighting coefficients.

[0087] 2. Testing Process

[0088] The testing process employs the standard Denoising Diffusion Implicit Model (DDIM) framework to accelerate the sampling process.

[0089] (a) Sampling H initial 3D pose assumptions ;

[0090] (b) Refining noisy 3D hypotheses using a denoiser Generate reasonable 3D hypotheses ;

[0091] (c) Input into DDIM to regenerate the pose with noise. ;

[0092] (d) Utilize ,pass The next iteration optimizes the current 3D pose assumption. The time step decreases from T to 1, following the following decreasing pattern: ,in ;

[0093] (e) Complete After several iterations, the final optimized complete 3D human pose sequence is output.

[0094] Although the denoiser only needs to be trained once during the training phase, researchers can define H (the number of 3D pose hypotheses) and [other parameters] independently during the testing phase. The value of (optimization iteration count) can be flexibly adjusted to improve the quality of the final estimation result. This design maintains the model's training efficiency while providing the ability to balance accuracy and speed for different scenarios in practical applications.

[0095] Example 2

[0096] Please see Figure 6 The diagram shown is a schematic representation of a 3D pose estimation system with occlusion-aware vertex reasoning and semantic enhancement proposed in the second embodiment of this application. The system includes the following key modules:

[0097] The data acquisition module 100 is used to acquire the initial two-dimensional pose sequence of the occluded joints;

[0098] The occlusion-aware vertex inference module 200 is used to process the initial two-dimensional pose sequence through the occlusion-aware vertex inference network, perform position inference on the occluded joints in the initial two-dimensional pose sequence based on temporal context information, and generate a preliminary completed two-dimensional pose sequence.

[0099] The confidence score generation module 300 is used to assign an initial confidence score to each joint in the initially completed two-dimensional pose sequence, and input the initially completed two-dimensional pose sequence and the initial confidence score to the confidence score generator to generate a final confidence score that reflects the true quality of each joint, so as to obtain a complete two-dimensional joint input containing the coordinates of each joint and the final confidence score.

[0100] The semantic feature enhancement module 400 is used to denoise and optimize the noisy 3D pose assumption based on the complete 2D joint input through the semantic feature enhancement network. The semantic feature enhancement network includes a Transformer encoder branch and a convolutional network encoder branch during the training phase. The two branches extract the semantic features of the first joint and the semantic features of the second joint in parallel, and use the semantic feature matching mechanism to calculate the semantic alignment loss between the semantic features of the first joint and the semantic features of the second joint, so as to supervise and enhance the semantic feature representation ability of the Transformer encoder.

[0101] The 3D pose output module 500 is used to perform iterative denoising processing on multiple sampled 3D pose hypotheses using the trained Transformer encoder, and output the final complete 3D human pose sequence.

[0102] The occlusion-aware vertex reasoning and semantic enhancement 3D pose estimation system in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), etc. This application embodiment does not specifically limit the specific implementation.

[0103] The occlusion-aware vertex reasoning and semantic enhancement 3D pose estimation system in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0104] The occlusion-aware vertex reasoning and semantic enhancement 3D pose estimation system provided in this application embodiment can achieve... Figure 1 The various processes of the occlusion-aware vertex reasoning and semantic enhancement 3D pose estimation method implemented in the method embodiment are not described in detail here to avoid repetition.

[0105] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a 3D pose estimation method for occlusion-aware vertex reasoning and semantic enhancement, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0106] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of the occlusion-aware vertex reasoning and semantic enhancement three-dimensional pose estimation method and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0107] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0108] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0110] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for occlusion-aware top vertex reasoning and semantic-enhanced 3D pose estimation, the method comprising: Includes the following steps: Obtain the initial 2D pose sequence of the occluded joints; The initial 2D pose sequence is processed by an occlusion-aware vertex inference network. Based on temporal context information, the position inference of occluded joints in the initial 2D pose sequence is performed to generate a preliminary completed 2D pose sequence. Specifically, this includes: traversing each frame and each joint in each frame of the initial 2D pose sequence; determining whether the current joint is an occluded joint; if the current joint is an occluded joint, then performing the following sub-steps: searching bidirectionally along the time dimension, centered on the current frame, towards past and future frames for the nearest unoccluded joint corresponding to the occluded joint; based on the search results, determining the position coordinates of the occluded joint using one of the following rules to... The completion process is as follows: Rule 1: If a corresponding unoccluded joint is found in both past and future frames equidistant from the current frame, the position coordinates of the occluded joint are determined as the midpoint between the coordinates of these two unoccluded joints; Rule 2: If a corresponding unoccluded joint is found only in a past frame, the position coordinates of the occluded joint are determined as the position coordinates of the corresponding joint in the past frame; Rule 3: If a corresponding unoccluded joint is found only in a future frame, the position coordinates of the occluded joint are determined as the position coordinates of the corresponding joint in the future frame; All completed joints are combined with the original unoccluded joints to form the preliminary completed two-dimensional pose sequence. An initial confidence score is assigned to each joint in the initially completed 2D pose sequence, and the initially completed 2D pose sequence and the initial confidence score are input to a confidence score generator to generate a final confidence score that reflects the true quality of each joint, so as to obtain a complete 2D joint input containing the coordinates of each joint and the final confidence score. Based on the complete two-dimensional joint input, a semantic feature enhancement network is used to denoise and optimize the noisy three-dimensional pose assumption. During the training phase, the semantic feature enhancement network includes a Transformer encoder branch and a convolutional network encoder branch. The convolutional network encoder branch is a Transformer-like convolutional network, Conv2DNet, which is composed of multiple cascaded Conv2D blocks. Before inputting features into the Conv2DNet, the temporal features are first divided into G groups along the time dimension and reshaped into a two-dimensional feature map resembling an image. The structure of each Conv2D block is defined by the following formula: ; ;in, Indicates feature input, Indicates intermediate variables. Indicates batch standardization. This represents a multilayer perceptron. This represents a multi-scale convolutional attention mechanism, the core components of which are grouped dual convolutional (GDC) and dual normalized DN. The first joint semantic features and the second joint semantic features are extracted in parallel by two branches, and the semantic alignment loss between the first joint semantic features and the second joint semantic features is calculated by using a semantic feature matching mechanism, so as to supervise and enhance the semantic feature representation capability of the Transformer encoder. Using the trained Transformer encoder, the sampled three-dimensional pose assumptions are iteratively denoised to output the final complete three-dimensional human pose sequence.

2. The method according to claim 1, characterized in that, The steps of assigning an initial confidence score to each joint in the initially completed 2D pose sequence, and inputting the initially completed 2D pose sequence and the initial confidence scores into a confidence score generator to generate a final confidence score reflecting the true quality of each joint, to obtain a complete 2D joint input containing the coordinates of each joint and the final confidence score, specifically include: Based on the inter-frame distance between the found unoccluded joints and the current frame, an initial confidence score is assigned to each joint. The initial confidence score for unoccluded joints is 1.00, while the initial confidence score for occluded joints is reduced based on the search distance. The initially completed two-dimensional pose sequence and the initial confidence scores of all joints As input, a confidence score generator consisting of two convolutional functions is fed in. This confidence score generator models the correlation between the joint coordinates and the initial confidence score, and outputs the confidence score. , is defined by the following formula: ； in, For confidence score generator, For the first n Frame number i Confidence score for each joint; The initially completed two-dimensional pose sequence With the confidence score By splicing the components together, the complete two-dimensional joint input is obtained. .

3. The method according to claim 1, characterized in that, The specific implementation process of the multi-scale convolutional attention mechanism includes: By designing learnable kernel functions and Keys K and values ​​V required for the convolutional attention mechanism; The grouped dual convolutional GDC is implemented based on strip convolution, including learning the correlation between adjacent frames through horizontal convolution and learning the correlation between distant frames through vertical convolution. To achieve multi-scale context capture, the strip convolution employs multiple sets of convolution kernels of different sizes, with kernel sizes of respectively... , , ; The computation process of the multi-scale convolutional attention mechanism is defined by the following formula: ； in, , , , Indicates the kernel size. Indicates a channel. Indicates the number of learnable parameters. For activation function, This represents matrix multiplication.

4. The method according to claim 1, characterized in that, The steps for calculating the semantic alignment loss between the semantic features of the first joint and the semantic features of the second joint using a semantic feature matching mechanism specifically include: Obtain the first joint semantic features output by the Transformer encoder branch. and the second joint semantic features output by the encoder branch of the convolutional network ; Through a hyperparameter Channel distillation function Process the semantic features of the first joint respectively and the second joint semantic features The feature activation values ​​are converted into a probability distribution along the channel dimension to eliminate the scale difference between the two branches. The channel distillation function is defined as follows: ； in, For channel indexing, Let N be the spatiotemporal position of the joint, and N be the number of input frames. It refers to the number of joints in the human body. This represents the characteristics of the c-th channel. The feature represents the c-th channel and the th i The value of each position; The KL divergence between the two probability distributions is calculated as the semantic alignment loss. Its definition is as follows: ； in This indicates that the feature of the Transformer encoder branch output contains all features in the c-th channel. i The value of the position, This indicates that the features output by the encoder branch of the convolutional network contain all features in the c-th channel. i The value of the position.

5. The method according to claim 1 or 4, characterized in that, The total loss function of the method during the training phase Loss from 3D pose reconstruction and the semantic alignment loss The weighted composition is defined as follows: ； in, , For true 3D pose, The denoised 3D pose reconstructed by the Transformer encoder. , These are the weighting coefficients.

6. The method according to claim 1, characterized in that, The steps of using the trained Transformer encoder to iteratively denoise the sampled multiple 3D pose hypotheses and output the final complete 3D human pose sequence specifically include: During the testing phase, the convolutional network encoder branch is no longer used; Randomly sample H initial 3D pose assumptions; Performed within the total time step T Each iteration optimizes the current 3D pose hypothesis by using the Transformer encoder to denoise the current hypothesis in order to generate a better hypothesis. The The time step of each iteration decreases from T to 1, following the following decreasing pattern: ; Finish After each iteration, the final optimized complete three-dimensional human pose sequence is output.

7. A 3D pose estimation system with occlusion-aware vertex reasoning and semantic enhancement, characterized in that, include: The data acquisition module is used to acquire the initial two-dimensional pose sequence of joints that are occluded; The occlusion-aware vertex inference module is used to process the initial two-dimensional pose sequence through an occlusion-aware vertex inference network. Based on temporal context information, it performs position inference on occluded joints in the initial two-dimensional pose sequence to generate a preliminary completed two-dimensional pose sequence. Specifically, it includes: traversing each frame and each joint in the initial two-dimensional pose sequence; determining whether the current joint is an occluded joint; if the current joint is an occluded joint, performing the following sub-steps: searching bidirectionally along the time dimension, centered on the current frame, towards past and future frames for the nearest unoccluded joint corresponding to the occluded joint; and determining the occluded joint based on the search results using one of the following rules. The position coordinates are used to complete the sequence: Rule 1: If the corresponding unoccluded joint is found in both past and future frames equidistant from the current frame, the position coordinates of the occluded joint are determined as the midpoint between the coordinates of these two unoccluded joints; Rule 2: If the corresponding unoccluded joint is found only in past frames, the position coordinates of the occluded joint are determined as the position coordinates of the corresponding joint in the past frame; Rule 3: If the corresponding unoccluded joint is found only in future frames, the position coordinates of the occluded joint are determined as the position coordinates of the corresponding joint in the future frame; All completed joints are combined with the original unoccluded joints to form the initially completed two-dimensional pose sequence. The confidence score generation module is used to assign an initial confidence score to each joint in the initially completed two-dimensional pose sequence, and input the initially completed two-dimensional pose sequence and the initial confidence score to the confidence score generator to generate a final confidence score that reflects the true quality of each joint, so as to obtain a complete two-dimensional joint input containing the coordinates of each joint and the final confidence score. The semantic feature enhancement module is used to denoise and optimize the noisy 3D pose assumption based on the complete 2D joint input through a semantic feature enhancement network. During the training phase, the semantic feature enhancement network includes a Transformer encoder branch and a convolutional network encoder branch, wherein the convolutional network encoder branch is a Transformer-like convolutional network Conv2DNet, which is composed of multiple cascaded Conv2D blocks. Before inputting features into the Conv2DNet, the temporal features are first divided into G groups along the time dimension and reshaped into an image-like 2D feature map. The structure of each Conv2D block is defined by the following formula: ; ;in, Indicates feature input, Indicates intermediate variables. Indicates batch standardization. This represents a multilayer perceptron. The multi-scale convolutional attention mechanism is described, with its core components being grouped double convolutional (GDC) and double normalized DN. The first joint semantic features and the second joint semantic features are extracted in parallel through two branches, and the semantic alignment loss between the first joint semantic features and the second joint semantic features is calculated using a semantic feature matching mechanism to supervise and enhance the semantic feature representation capability of the Transformer encoder. The three-dimensional pose output module is used to perform iterative denoising processing on multiple sampled three-dimensional pose assumptions using the trained Transformer encoder, and output the final complete three-dimensional human pose sequence.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of a three-dimensional pose estimation method for occlusion-aware vertex reasoning and semantic enhancement as described in any one of claims 1-6.

Images