Multi-person attitude estimation method and system based on topology perception dynamic association

Abstract

The invention provides a multi-person posture estimation method and system based on topology perception dynamic association, and relates to the technical field of computer vision, and the method comprises the steps: synchronously obtaining a plurality of initial human body instance sets in an image and a corresponding initial key point set in each initial human body instance set based on a multi-scale feature map, generating a preliminary topological association between the human body instance and the key point; for each key point in the initial key point set, dynamically adjusting the spatial range of the key point based on the global context feature of the human body instance to which the key point belongs and the local semantic feature of the key point, and generating a dynamic key point frame; based on key point features extracted by the dynamic key point frame, key point relation reasoning is carried out based on a topology perception correlation model, key point positioning is optimized, grouping from key points to human body instances is completed, and a final multi-person posture estimation result is output. According to the method, the technical problems of poor robustness and insufficient precision of the existing multi-person attitude estimation method in a complex scene are solved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for estimating the pose of multiple people based on topology-aware dynamic association. Background Technology

[0002] Multi-person pose estimation is a fundamental task in computer vision. Its goal is to detect all body keypoints (such as elbows and knees) of individuals from a given image or video and correctly correlate and combine these keypoints into a complete, independent human pose instance. This technology has broad application prospects in many fields such as human-computer interaction, intelligent monitoring, sports analysis, and virtual reality. With the development of deep learning technology, the performance of multi-person pose estimation has been significantly improved; however, in complex scenes (such as dense crowds, severe occlusion, and complex interactions), existing methods still face significant challenges, and their robustness and accuracy need further improvement.

[0003] Currently, the main methods for multi-person pose estimation are: (1) Top-down approach. This approach first uses an independent human detector to locate the bounding boxes of all human instances in the image, and then performs single-person pose estimation within each cropped bounding box. Although this type of approach performs well on datasets with relatively simple scenes, its performance is highly dependent on the quality of the preceding human detector. Errors in human detection (such as missed detections, false detections, or inaccurate box localization) will directly propagate to the pose estimation stage, forming cascaded errors. In addition, in crowded scenes, its computational complexity will increase linearly with the number of people in the image, resulting in low efficiency. Furthermore, fixed rectangular bounding boxes are difficult to flexibly adapt to various complex human poses (such as stretching, curling up) and partial occlusion, and are prone to introducing too much background noise or cropping out key parts.

[0004] (2) Bottom-up approach. This method first detects all possible human keypoints across the entire image, and then groups and clusters these keypoints to different individual instances using a pre-defined association strategy. In scenarios where human bodies highly overlap or have close interactions, the ambiguity of the association between keypoints and individual instances increases dramatically, making it easy to generate incorrect associations, incorrectly grouping keypoints belonging to different people together, or incorrectly splitting keypoints belonging to the same person; moreover, the keypoint grouping strategy often relies on manually designed heuristic rules or complex post-processing optimization, which has limited generalization ability and is difficult to adapt to diverse scenarios; the ambiguity problem of keypoint matching is particularly prominent in dense crowds.

[0005] (3) Single-stage method. This method unifies human detection and keypoint estimation into a single-stage method in a single end-to-end network. Although this type of method improves efficiency, it still relies on a fixed keypoint topology, i.e., a pre-set human skeleton model. If all human poses conform to this static structure, it cannot effectively handle topological changes caused by non-rigid deformation, occlusion, etc. In addition, the network usually adopts a static receptive field design, such as a fixed-size convolutional kernel or attention window, which cannot adapt to human instances of different scales and different degrees of occlusion. Moreover, the association patterns between keypoints are relatively fixed, lacking the ability to model dynamic topological relationships that are sensitive to image context, resulting in poor performance in complex scenes.

[0006] It is evident that existing technologies generally suffer from insufficient ability to model dynamic topological relationships, rigid receptive field design, and susceptibility to occlusion and scale changes when dealing with complex real-world scenes. Therefore, there is an urgent need for a multi-person pose estimation method that can perceive dynamic topology, adaptively adjust keypoint associations, and maintain high robustness in complex environments. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a multi-person pose estimation method and system based on topology-aware dynamic association, which solves the technical problems of poor robustness and insufficient accuracy of existing multi-person pose estimation methods when facing complex scenes due to their reliance on fixed topology, static receptive field and rigid association.

[0008] In a first aspect, the present invention provides a multi-person pose estimation method based on topology-aware dynamic association, the method comprising: Feature extraction and multi-scale fusion are performed on the input image to generate a multi-scale feature map; based on the multi-scale feature map, multiple initial human instance sets and the initial key point sets corresponding to each initial human instance set are obtained simultaneously to generate a preliminary topological association between human instances and key points. For each key point in the initial key point set, the spatial range of the key point is dynamically adjusted based on the global context features of the human instance to which the key point belongs and the local semantic features of the key point itself, thereby generating a dynamic key point bounding box. Based on the key point features extracted from the dynamic key point bounding box, key point relationship reasoning is performed based on the topology-aware association model, key point localization is optimized, and key points are grouped into human instances, outputting the final multi-person pose estimation result.

[0009] Furthermore, feature extraction and multi-scale fusion are performed on the input image, specifically including: Convolutional neural networks are used to extract multi-level features from the input image; The multi-level features are input into the local perceptual feature alignment module, processed through parallel feature extraction and feature alignment branches, and fused under the premise of spatial alignment to generate a multi-scale feature map with local features and spatial consistency.

[0010] Furthermore, the feature extraction branch is a rotationally symmetric convolutional structure used to capture fine-grained local features; the feature alignment branch is used to align the spatial coordinates between feature maps of different levels and output the multi-scale feature map.

[0011] Furthermore, generating preliminary topological associations between human instances and key points specifically includes: The multi-scale feature map is input into the encoder of the Transformer to perform global context modeling, thereby obtaining global encoded features; A set of learned human instance queries are interacted with global encoded features to generate an embedding vector and corresponding location coordinates for each human instance; The embedding vector of each human body instance is expanded into a set of key point queries to obtain the center coordinates and spatial range of each key point; Based on the location coordinates of each human instance and the center coordinates of each key point, a preliminary topological association between the human instance and the key point is obtained.

[0012] Furthermore, the center coordinates and spatial extent of each key point are obtained, specifically including: Based on the embedding vector of the human instance associated with the key point, the center coordinates of the key point are predicted by a lightweight regression head. The spatial range of key points is predicted based on human prior heuristic initialization.

[0013] Furthermore, dynamic keyframes are generated, specifically including: The global context features and the local semantic features are fused to generate a unified fused feature representation; Based on the fused feature representation, a scaling factor is obtained for adjusting the size of the key point bounding box; The spatial extent of the key points is adjusted using the scaling factor to generate dynamic key point boxes with optimized width and height.

[0014] Furthermore, the establishment of the topology-aware association model includes: establishing spatial dependencies between different key points within the same human body instance based on preliminary topology association, and establishing semantic consistency relationships between similar key points between different human body instances.

[0015] Furthermore, the topology-aware association model achieves relational reasoning through a cross-keypoint learning module. Specifically, the cross-keypoint learning module performs the following operations: Global information is aggregated from spatial dependencies and semantic consistency relationships to generate global attention weights for recalibrating key point features; Convolutional operations are used to extract local spatial features around keypoints, enhancing the perception of the relative positions between keypoints.

[0016] Furthermore, the cross-keypoint learning module employs a group attention mechanism to process information from different relational dimensions in parallel.

[0017] In a second aspect, the present invention provides a multi-person pose estimation system based on topology-aware dynamic association, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0018] In summary, this invention provides a method and system for multi-person pose estimation based on topology-aware dynamic association. Compared with existing technologies, the technical solution conceived in this invention can achieve the following beneficial effects: (1) By adopting a joint detection paradigm that simultaneously acquires initial human instances and key points, this invention avoids the problem of cascading error propagation caused by the reliance on external detectors in traditional top-down methods. In particular, by combining dynamic key point box adjustment and topology-aware association model inference, it can adaptively handle complex situations such as occlusion, crowding, and scale changes, significantly improving the estimation robustness and accuracy in complex scenes.

[0019] (2) This invention dynamically predicts the optimal detection region size for each keypoint by fusing the global contextual features of human instances and the local semantic features of keypoints. This overcomes the inherent defects of fixed-size detection boxes, avoiding false detections caused by background noise introduced by excessively large boxes, and preventing false associations of keypoints caused by insufficient contextual information by excessively small boxes. By introducing dynamic keypoint boxes, it can flexibly cope with different keypoint types and different human scales, and shows strong and stable performance on various datasets. It achieves adaptive optimization of the receptive field of keypoint detection and demonstrates excellent generalization ability.

[0020] (3) This invention goes beyond static association strategies that rely on fixed skeleton models, and instead explicitly constructs a topology-aware association model during the fine-tuning stage. This model can simultaneously learn the topological relationships of bones within the human body (such as the connection between the wrist and elbow) and the semantic consistency of similar key points between people (such as the right shoulder of everyone). This dual-spatial relationship modeling, combined with global attention recalibration and local spatial feature enhancement, enables the model to have powerful key point disambiguation and correct grouping capabilities in densely intersecting and severely occluded scenes, fundamentally reducing the probability of incorrectly associating key points of different individuals, and effectively solving the problem of key point association ambiguity in dense crowds.

[0021] (4) The multi-stage architecture from coarse to fine in this invention is an efficient end-to-end system. On the one hand, by using parallel feature extraction branches and feature alignment branches, while ensuring the consistency of the multi-scale feature space, it fully preserves the fine-grained details necessary for accurate localization, providing a high-quality feature foundation for subsequent stages. In addition, the query mechanism based on Transformer combines human detection and key point initialization, reducing redundant calculations. It achieves the goal of maintaining high accuracy while having good computational efficiency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the method steps of a multi-person pose estimation method and system based on topology-aware dynamic association provided by the present invention; Figure 2 This is a schematic diagram illustrating the multi-stage architecture principle of a multi-person pose estimation method and system based on topology-aware dynamic association provided by the present invention. Figure 3 This is a schematic diagram of the principle of the local sensing feature alignment module of a multi-person pose estimation method and system based on topology-aware dynamic association provided by the present invention. Figure 4 This is a schematic diagram illustrating the principle of the cross-keypoint learning module of a multi-person pose estimation method and system based on topology-aware dynamic association provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of additional identical elements in the method, step, or apparatus that includes the element.

[0026] To address the shortcomings of existing multi-person pose estimation methods in complex scenes, which suffer from poor robustness and insufficient accuracy due to reliance on fixed topology, static receptive fields, and rigid associations, and considering that extracting global contextual features of human instances before keypoint inference can improve robustness and reduce error propagation, and that keypoint association is inherently dynamic and dependent on the global contextual features of human instances, this invention provides a multi-person pose estimation method and system based on topology-aware dynamic association. It employs a coarse-to-fine multi-stage architecture (TopoDA), progressively optimizing pose estimation results through hierarchical design. This mimics the human visual process of perceiving keypoints from a global instance level to a refined local level, enabling adaptive handling of complex situations such as occlusion, crowding, and scale variations, significantly improving estimation robustness and accuracy in complex scenes.

[0027] Specifically, such as Figure 1 and Figure 2 As shown, the method includes: S100: Perform feature extraction and multi-scale fusion on the input image to generate a multi-scale feature map; based on the multi-scale feature map, simultaneously acquire multiple initial human instance sets and the corresponding initial key point sets in each initial human instance set to generate a preliminary topological association between human instances and key points.

[0028] The first stage is human-to-keypoint detection. This invention no longer completely decouples human detection and keypoint estimation, but instead synchronously generates the initial positions of human instances and their corresponding keypoints. This effectively avoids the cascading errors caused by human detection errors in traditional top-down methods.

[0029] like Figure 2As shown in (a), the first stage starts with the original input image. First, the image passes through a feature extraction and multi-scale fusion network (e.g., a convolutional neural network incorporating a local perceptual feature alignment module), outputting a set of spatially aligned and detail-rich multi-scale feature maps; these multi-scale feature maps serve as the shared feature base for all subsequent processing. Then, based on the multi-scale feature maps, a query-based mechanism (e.g., a Transformer-based encoder-decoder architecture) is used to synchronously detect human bodies and keypoints, generating preliminary topological associations between human instances and keypoints, thus avoiding error accumulation caused by serial processing in traditional methods.

[0030] It should be noted that multi-scale feature fusion is crucial for detecting human bodies at different scales, but simple fusion methods are prone to spatial misalignment and loss of detail. To overcome this limitation, this invention proposes a local perceptual feature alignment module, which, while maintaining spatial accuracy, fully preserves the fine-grained local details necessary for accurate keypoint detection in occluded scenes. This is particularly critical for robust pose estimation under occlusion and scale changes.

[0031] As an example, feature extraction and multi-scale fusion of the input image specifically include: Convolutional neural networks are used to extract multi-level features from the input image; Multi-level features are input into the local perceptual feature alignment module, processed through parallel feature extraction and feature alignment branches, and fused under the premise of spatial alignment to generate a multi-scale feature map with local features and spatial consistency.

[0032] Furthermore, the feature extraction branch is a rotationally symmetric convolutional structure used to capture fine-grained local features; the feature alignment branch is used to align the spatial coordinates between feature maps of different levels, outputting multi-scale feature maps. The rotationally symmetric convolutional structure can be a windmill-shaped convolutional kernel.

[0033] In other words, the feature extraction branch is used to capture fine-grained local features and improve the detection effect of small-scale human bodies; the feature alignment branch is used to ensure the accurate spatial alignment of features from different resolutions, so as to preserve the spatial details that are crucial to accurate localization to the greatest extent while utilizing multi-scale contextual information.

[0034] like Figure 3 The diagram shown is a schematic of the local perceptual feature alignment module, which aims to solve the problems of feature space misalignment and loss of detail information in the process of multi-scale feature fusion.

[0035] First, it receives convolutional feature maps from different levels of the backbone network as input, with the maps representing features from different levels, such as F. i-1 F iF i+1 The symbols represent global features from different resolutions. The core task of the local perceptual feature alignment module is to effectively align and fuse these feature maps with different spatial resolutions and semantic levels.

[0036] The local perceptual feature alignment module uses a parallel dual-branch structure to process the input features in order to achieve synergistic optimization of feature enhancement and spatial alignment.

[0037] The feature extraction branch focuses on capturing and enhancing fine-grained local information in the feature maps. Specifically, the input features are first channel-adjusted through a 1×1 convolution, and then fed into a rotationally symmetric convolutional structure for deep feature extraction to enhance the perception of local details. This process also incorporates a learnable factor γ, which is adjusted to adaptively control the feature flow.

[0038] The feature alignment branch aims to achieve cross-resolution feature alignment, ensuring spatial consistency between semantic representations and detailed representations. Specifically, it uses lower-level features F... i-1 and the high-level features F after preliminary processing i For example, firstly, the high-level features are upsampled to match the spatial resolution of the low-level features; then, a learnable offset prediction sub-network is used to compute the low-level feature F. i-1 The spatial offset offset relative to the target alignment position is used; finally, using the spatial offset offset, operations such as deformable convolution or bilinear sampling are performed on the low-level features E. i-1 Perform precise spatial warping and resampling to ensure accurate spatial alignment with higher-level features.

[0039] After parallel dual-branch processing, the feature map rich in detail output from the feature extraction branch is fused with the spatially aligned feature map output from the feature alignment branch through element-wise addition or channel concatenation. The final output is a set of multi-scale feature maps that retain rich local detail while maintaining strict spatial alignment.

[0040] To bridge the gap between global human-level semantics and local keypoint-level representation, this invention also introduces human-to-keypoint detection, explicitly modeling the preliminary topological relationships between human instances and their joints. For example... Figure 2 As shown in (a), the multi-scale feature map processed by the local perception feature alignment module is flattened and input into the Transformer encoder. The location encoding of human features is obtained by the instance decoder, and then the key points are obtained to realize cross-resolution information interaction.

[0041] As one example, generating a preliminary topological association between human instances and key points specifically includes: S1001: Input multi-scale feature maps into the Transformer encoder for global context modeling to obtain global encoded features. For example, the input image passes through a feature extraction network (such as ResNet-50) and a subsequent local perceptual feature alignment module to obtain a set of features with spatial dimensions of the original... Figure 1 Multi-scale feature maps of 1 / 32, 1 / 16, and 1 / 8 are denoted as F. 32 F 16 F8; After flattening and stitching these multi-scale feature maps, they are input into a standard Transformer encoder. The encoder performs operations through a multi-layer self-attention mechanism, so that each pixel position in the feature map can interact with all positions in the whole map, thereby completing global context modeling and outputting a global encoded feature Z rich in global semantic information.

[0042] S1002: A set of learned human instance queries interacts with global encoded features to generate the embedding vector and corresponding location coordinates of each human instance. In other words, a set of learnable instance queries interacts with global encoded features through self-attention and cross-attention mechanisms to decode the bounding box coordinates (location coordinates) and embedding vector of each initial human instance.

[0043] For example, at the decoder end, a set of learnable human instance query vectors Q is initialized. person ={q1,q2,...,q N} where N is the preset maximum possible number of human instances, for example, N=100. These human instance query vectors interact with the global encoded features Z output by the encoder through a cross-attention mechanism. Specifically, each human instance query q i The most relevant global information is aggregated from the global encoded feature Z. Subsequently, each updated human instance query is processed through two independent feedforward networks: an instance classification head and a bounding box regression head. The instance classification head predicts the probability that the human instance query corresponds to a real human instance; while the bounding box regression head predicts the bounding box coordinates (x, y, w, h) of the human instance query. In this way, the embedding vectors and corresponding location coordinates of all possible human instances in the image are obtained.

[0044] S1003: Expand the embedding vector of each human body instance into a set of key point queries to obtain the center coordinates and spatial range of each key point.

[0045] For each confirmed human instance query, for example, a query with a classification probability higher than the threshold of 0.7, its corresponding embedding vector e is... iThis is expanded to generate keypoint queries specific to this instance. Specifically, the human body is defined with K keypoint types, such as K=17, following the COCO dataset standard. This is achieved through a learnable linear projection matrix W. expand ∈R D×K , embed vector e i Expand to a set of key point queries: Q i keypoints =e i ·W expand In this way, all K key point queries originate from the same human body instance and are naturally bound to that instance.

[0046] Query each key point q i k Similarly, using a lightweight regression head, the center coordinates (x, y) of each keypoint are obtained. k ,y k ) and spatial range (w k ,h k ).

[0047] Furthermore, the center coordinates and spatial extent of each key point are obtained, specifically including: Based on the embedding vector of the human instance associated with the keypoint, the center coordinates of the keypoint are predicted by a lightweight regression head. For example, the human instance embedding vector is input into a lightweight regression head, which is usually composed of one or more fully connected layers. The regression head directly outputs the normalized center coordinates (x, y, y) of each keypoint of the human instance relative to the image coordinate system. k ,y k ).

[0048] The spatial range of key points is obtained based on human prior heuristic initialization prediction, so as to perform adaptive optimization in the next stage through dynamic key point bounding box adjustment.

[0049] This structured initialization provides a robust topological prior for the downstream optimization process, effectively improving the model's adaptability in occlusion and interaction scenarios.

[0050] S1004: Based on the position coordinates of each human instance and the center coordinates of each key point, obtain the initial topological perception association model between the human instance and the key points.

[0051] Keypoint queries are extended using shared instance embedding vectors, explicitly associating each keypoint with its corresponding human instance. This association is structured, manifested as follows: for each valid human instance, there exists a unique instance identifier, represented by its query embedding, and all keypoints of that instance are logically grouped using this identifier. This one-to-many mapping relationship forms a solid foundation for the subsequent dynamic optimization phase.

[0052] S200: For each keypoint in the initial keypoint set, based on the global context features of the human instance to which the keypoint belongs and the local semantic features of the keypoint itself, the spatial range of the keypoint is dynamically adjusted to generate a dynamic keypoint bounding box.

[0053] The second stage is Dynamic Keypoint Box Modulation. This stage refines each initial keypoint output from the previous stage. Its core is not to use a fixed-size detection region for all keypoints, but to adaptively modulate the size of the detection box for each keypoint. Human sequence expansion and keypoint sequence expansion are performed based on the human context encoder and keypoint semantic encoder, respectively. A scaling factor is obtained through a multilayer perceptron. The size of the initial keypoint box generated in the previous stage is dynamically adjusted by the scaling factor to output the dynamic keypoint box.

[0054] The success of keypoint detection largely depends on the appropriate receptive field size for each keypoint query. However, existing methods typically use fixed-size bounding boxes, which have several inherent drawbacks: excessively large bounding boxes introduce background noise and increase the probability of false detections; conversely, excessively small bounding boxes fail to capture sufficient disambiguation context information for keypoints, potentially leading to misclassifications of semantically similar keypoints from different individuals; furthermore, the optimal bounding box size varies significantly across different keypoint types and human scales. Therefore, static methods are inherently difficult to optimize in robust multi-person pose estimation tasks.

[0055] Therefore, to address the shortcomings of fixed receptive fields, this invention adaptively adjusts the detection area size for each key point based on global human context features and local key point semantic features, thereby effectively suppressing background noise and capturing sufficient discriminative information.

[0056] As one example, generating dynamic keyframes specifically includes: Global contextual features are fused with local semantic features to generate a unified fused feature representation; Based on the fused feature representation, a scaling factor is obtained for adjusting the size of the key point box; By using scaling factors to adjust the spatial extent of key points, dynamic key point boxes with optimized width and height are generated.

[0057] In other words, firstly, the global contextual features (including overall information such as human pose and scale) of the human instance to which the current keypoint belongs, as well as the local semantic features of the keypoint itself, are extracted separately. Next, the human features representing global information and the keypoint features representing local details are aligned and concatenated to form a unified fusion representation, thereby establishing the association between the human body and the keypoint. Finally, a small network (such as a fully connected layer) predicts a pair of scaling factors δ used to adjust the initial bounding box width of the keypoint. w and the scaling factor δ of height h Using the predicted scaling factor, the size of the initial keyframe generated in the previous stage is dynamically adjusted, and a dynamic keyframe with optimized size that is more compatible with the keyframe type and human scale is output.

[0058] After obtaining the scaling factor, the spatial extent of the key points is dynamically adjusted, that is, the width and height of the key point bounding box are adjusted, specifically including: Get the initial width w of the spatial range of the key points init and initial height h init Using scaling factors, the initial width w is adjusted accordingly. init and initial height h init Scaling is performed to calculate the optimized dynamic keypoint bounding box size. The calculation method can be as follows: The optimized width is: w adjusted =w init ×δ w ; The optimized height is: h adjusted =h init ×δ h ; During the adjustment process, the center coordinates (x, y) of the key points are... k ,y k () Remain unchanged.

[0059] This mechanism allows for the intelligent allocation of an appropriate perception range to each key point. For example, for a large and feature-rich torso, it might be possible to predict δ. w and δ h Slightly greater than 1, the detection box is appropriately enlarged to capture more context; for small, blurry, and easily confused wrists or ankles, the model may predict a scaling factor less than 1 to reduce the scope of the box to exclude interference from nearby key points or background; thus significantly improving robustness in detecting drastic scale changes and edge joints, fundamentally overcoming the limitations of static receptive fields.

[0060] S300: Based on the key point features extracted by the dynamic key point bounding box, it performs key point relationship reasoning based on the topology-aware association model, optimizes key point localization, completes the accurate grouping of key points to human instances, and outputs the final multi-person pose estimation results.

[0061] The third stage is Topology-Aware Keypoint Refinement. Based on the dynamic keypoint bounding boxes output from the previous stage, the refined features of each keypoint are precisely cropped and extracted from the shared multi-scale feature map. Subsequently, these keypoint features are fed into the Topology-Aware Association Model (Keypoint Decoder) for relationship reasoning and optimization.

[0062] The final stage of TopoDA focuses on achieving fine-grained pose adjustment through topology-aware inference. Although the preceding stages have provided coarse human detection and adaptive keypoint localization, in complex scenes, especially in crowded situations with overlapping or blurred keypoints, achieving accurate pose estimation still requires explicit modeling of the joint structures within the human body and the relationships between keypoints.

[0063] Therefore, based on the coarse pose provided in the first two stages, this stage introduces a cross-keypoint learning module to finely adjust the complex relationships between keypoints, and uses a topology-aware association model to infer keypoint relationships, optimize keypoint localization, complete the accurate grouping of keypoints to human instances, and output the final multi-person pose estimation results.

[0064] It should be noted that the core of the topology-aware association model is to perform information interaction across key points. As an example, the establishment of the topology-aware association model includes: establishing spatial dependencies between different key points within the same human instance based on preliminary topological association (i.e., the human skeleton topology, such as the association between the left wrist and the left elbow), and establishing semantic consistency relationships between similar key points between different human instances (such as the right elbow of all people), thereby achieving more robust key point allocation in crowded and occluded scenarios.

[0065] As an example, the topology-aware association model achieves relational reasoning through a cross-keypoint learning module. Specifically, the cross-keypoint learning module performs the following operations: Global information is aggregated from spatial dependencies and semantic consistency relationships to generate global attention weights for recalibrating key point features; Convolutional operations are used to extract local spatial features around keypoints, enhancing the perception of the relative positions between keypoints.

[0066] Furthermore, the cross-keypoint learning module employs a group attention mechanism to process information from different relational dimensions in parallel.

[0067] Keypoint self-attention mechanism is used to enhance the representation ability of individual keypoints; deformable attention mechanism is used to adaptively aggregate spatial features within dynamic keypoint boxes for accurate localization.

[0068] like Figure 4 As shown, the input to the cross-keypoint learning module is the set of all human keypoint features obtained from the previous stage, denoted as Q. kd ∈R N×K×C , where N represents the number of human instances in the image, K represents the number of keypoints in each instance, and C is the number of feature channels; this input includes the initial position and appearance information of all keypoints of all human bodies.

[0069] The cross-keypoint learning module employs a dual-spatial relationship modeling strategy. By simultaneously processing spatial relationships at both global and local levels, it captures long-range semantic connections and fine-grained spatial constraints from the dependencies between keypoints, modeling them from two complementary dimensions: global cross-spatial relationships and local joint relationships. The model aggregates global information from both the human instance dimension and the keypoint type dimension, generating corresponding attention weights. These weights are used to recalibrate the original features, highlighting global responses with strong semantic connections. Convolutional operations are used to extract local spatial features around keypoints, thereby enhancing the perception of subtle relative positional relationships between joints.

[0070] The global cross-space branch aims to model the semantic consistency relationships between different human body instances and between similar keypoints, i.e., the global topology across instances, and the overall associations within human body instances. The main process is as follows: First, input feature Q... kd The data are averaged and pooled separately along two dimensions: human instance dimension and keypoint type dimension, to obtain feature representations that aggregate the global context from different dimensions (Q). kd h ∈R N×1×C and Q kd k ∈R 1×K×C Then, the aggregated features are passed through a 1×1 convolutional layer for cross-channel information interaction and fusion, and then processed by an activation function σ (such as Sigmoid or Softmax function) to generate a global attention weight map (A). h ∈R N×1×C and A k ∈R 1×K×C A h and A k Combined with the obtained feature Q, which includes global semantic consistency information kd gIt can highlight key features that have strong semantic consistency globally (e.g., all people's right shoulders should have similar feature patterns) and suppress outlier or noisy features, thereby enhancing the model's ability to distinguish similar joints between different individuals in dense crowds.

[0071] The local joint relationship branch aims to model fine-grained spatial dependencies between adjacent or related keypoints within the same human instance, i.e., the skeletal topology within the human body. The main process is as follows: Input feature Q... kd The data is fed into one or more 3×3 convolutional layers for processing to obtain Q. kd l ∈R N×K×C Q kd l Then, after normalization operations such as average pooling and softmax, the small-sized convolutional kernels can operate within the local neighborhood of keypoints, effectively capturing subtle spatial relationship patterns such as relative positions and orientations between joints, and outputting feature Q that includes local spatial context information. kd wl .

[0072] The features output from the global cross-spatial branch and the local joint relationship branch are respectively passed to the feature integration submodule. On the one hand, the feature Q of the global semantic consistency information is... kd g The feature Q is obtained after normalization operations such as average pooling and softmax. kd wg Q kd wg With Q kd l The first fusion feature is obtained after fusion; on the other hand, Q kd wl With Q kd g After fusion, a second fusion feature is obtained. Finally, the first and second fusion features are deeply integrated and fused through weighted summation, concatenation, or further feature modulation operations to output a recalibrated keypoint feature representation enhanced with topological relationships. These optimized features have stronger discriminative power for the attribution relationships between keypoints and will be directly used for subsequent refined regression of keypoint locations and final grouping of human instances.

[0073] This dual-space relationship modeling, on the one hand, uses an attention mechanism to aggregate information from two dimensions—human instances and key points—to calculate attention weights. This means that it can learn both the skeletal connections within a single human body (internal topology) and the semantic consistency of similar key points between different human bodies (inter-human topology), highlighting strong semantic associations and enabling a full understanding of the context of key points in complex scenes. On the other hand, it extracts subtle local spatial features around key points through convolution, enhancing the perception of the relative positional relationships between subtle joints.

[0074] This approach, combining global and local perspectives, ensures that both the overall structure and local details are grasped, enabling the deduction of flexible and dynamic topological relationships rather than relying on fixed templates. It allows for fine-tuning of the position of each keypoint, ensuring that keypoints can be correctly grouped to their corresponding individuals even in severely occluded or human interaction scenarios. This significantly reduces the probability of incorrect associations and ultimately solves the problem of accurate grouping of keypoints to human instances, outputting high-precision final multi-person pose estimation results.

[0075] On the other hand, the present invention also provides a multi-person pose estimation system based on topology-aware dynamic association, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0076] Specifically, the system includes: The feature extraction and fusion module is used to extract features from the input image and fuse them at multiple scales to generate a multi-scale feature map. The joint detection module, connected to the feature extraction and fusion module, is used to simultaneously acquire multiple initial human instance sets and the corresponding initial key point sets in the image based on multi-scale feature maps, and generate preliminary topological associations between human instances and key points. The dynamic bounding box adjustment module, connected to the joint detection module, is used to dynamically adjust the spatial range of each keypoint in the initial keypoint set based on the global context features of the human instance to which the keypoint belongs and the local semantic features of the keypoint itself, thereby generating a dynamic keypoint bounding box. The topology-aware optimization module, connected to the dynamic bounding box adjustment module, is used to perform key point relationship reasoning based on the key point features extracted from the dynamic key point bounding box and the topology-aware association model, optimize key point localization, complete the grouping of key points to human instances, and output the final multi-person pose estimation results.

[0077] To verify the effectiveness and superiority of the present invention, it was validated on two standard datasets.

[0078] For complex scenarios including severe occlusion and dense crowds, this invention was evaluated on the CrowdPose dataset, which is specifically designed for complex scenarios.

[0079] Table 1. Comparison results of this invention with other methods on the CrowdPose dataset.

[0080] As shown in Table 1, the present invention achieved the best performance, with an AP of 73.5, which is significantly better than other top-down, bottom-up and single-stage methods. This shows that the present invention has excellent performance in complex scenarios such as severe occlusion and overlapping poses.

[0081] For complex scenarios including severe occlusion and dense crowds, this invention was also evaluated on the COCO dataset, which is specifically designed for complex scenarios.

[0082] Table 2 Comparison results of the present invention with other methods on the COCO dataset.

[0083] As shown in Table 2, the present invention achieved the best performance, with an AP of 73.8, which is also significantly better than other top-down, bottom-up and single-stage methods. This shows that the present invention not only has excellent performance in complex scenarios such as severe occlusion and overlapping poses, but also maintains strong performance in conventional scenarios, which fully verifies that the method of the present invention has good generalization ability and wide applicability.

[0084] In summary, this invention employs a coarse-to-fine multi-stage architecture, addressing the core issues of rigid topological structures and insufficient robustness of associations in existing methods under complex scenarios. The first stage utilizes a joint human instance-keypoint detection approach, simultaneously estimating human instances and their corresponding keypoints, thereby alleviating the decoupling problem common in traditional top-down processes and effectively avoiding cascading errors. Building upon the first stage, the second stage introduces a dynamic keypoint bounding box adjustment mechanism, adaptively adjusting the spatial extent of each keypoint detection region based on global contextual features and local semantic features, enhancing the model's ability to handle scale changes and edge joints, thus overcoming the limitations of fixed receptive fields. Finally, the third stage incorporates cross-keypoint learning, capturing dependencies within and between human bodies through grouped attention, enabling the topology-aware association model to infer flexible topological relationships, thereby optimizing the accurate localization and correct allocation of keypoints in complex scenarios such as occlusion and crowding. This architecture ensures a smooth transition from global instance-level understanding to local keypoint-level refinement in the inference process, achieving a balance between accuracy and efficiency. It also enables adaptive handling of complex situations such as occlusion, congestion, and scale changes, significantly improving the robustness and accuracy of estimation in complex scenarios.

[0085] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0086] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0087] In the several embodiments provided by this invention, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0088] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This invention is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0089] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for multi-person pose estimation based on topology-aware dynamic association, characterized in that, The method comprises: feature extraction and multi-scale fusion on an input image to generate a multi-scale feature map; based on the multi-scale feature map, synchronously obtaining a plurality of initial human instance sets in the image and an initial key point set corresponding to each initial human instance set to generate a preliminary topological association of human instances and key points; for each key point in the initial key point set, based on the global context feature of the human instance to which the key point belongs and the local semantic feature of the key point itself, dynamically adjusting the spatial range of the key point to generate a dynamic key point box; based on the key point feature extracted by the dynamic key point box, based on a topologically aware association model, key point relationship reasoning is performed to optimize key point positioning and complete grouping of key points to human instances, and finally output multi-person pose estimation results.

2. The method of claim 1, wherein, The feature extraction and multi-scale fusion on the input image specifically comprises: extracting multi-level features of the input image using a convolutional neural network; inputting the multi-level features into a local perception feature alignment module, processing through a feature extraction branch and a feature alignment branch operating in parallel, and fusing under the premise of spatial alignment to generate a multi-scale feature map with local features and consistent space.

3. The method of claim 2, wherein, The feature extraction branch is a rotationally symmetric convolutional structure for capturing fine-grained local features; the feature alignment branch is used to realize spatial coordinate alignment between different level feature maps, and outputs the multi-scale feature map.

4. The method of claim 1, wherein, The preliminary topological association of human instances and key points is generated, specifically comprising: inputting the multi-scale feature map into the encoder of the Transformer for global context modeling to obtain global encoding features; interacting a set of learned human instance queries with the global encoding features to generate an embedding vector and corresponding position coordinates of each human instance; extending the embedding vector of each human instance into a set of key point queries to obtain the center coordinates and spatial range of each key point; based on the position coordinates of each human instance and the center coordinates of each key point, obtaining the preliminary topological association of human instances and key points.

5. The method of claim 4, wherein, The center coordinates and spatial range of each key point are obtained, specifically comprising: based on the embedding vector of the human instance associated with the key point, the center coordinates of the key point are predicted through a lightweight regression head; based on the heuristic initialization of human prior, the spatial range of the key point is predicted.

6. The method of claim 1, wherein, The dynamic key point box is generated, specifically comprising: fusing the global context feature and the local semantic feature to generate a unified fusion feature representation; based on the fusion feature representation, a scaling factor for adjusting the size of the key point box is obtained; using the scaling factor to adjust the spatial range of the key point to generate a dynamic key point box with optimized width and height.

7. The method of claim 1, wherein, The establishment of the topologically aware association model comprises: based on the preliminary topological association, establishing the spatial dependency relationship between different key points in the same human instance, and establishing the semantic consistency relationship between the same type of key points in different human instances.

8. The method of claim 7, wherein, The topologically aware association model realizes relationship reasoning through a cross-key point learning module, specifically, the cross-key point learning module performs the following operations: The global information is aggregated from spatial dependency and semantic consistency to generate global attention weights to recalibrate the key point features; Local spatial features around the key points are extracted by convolution operation to enhance the perception of relative positions between the key points.

9. The method of claim 8, wherein, The cross-keypoint learning module adopts a group attention mechanism to process information in different relationship dimensions in parallel.

10. A topology-aware dynamic association based multi-person pose estimation system comprising a memory, a processor and a computer program stored on the memory, wherein, The processor executes the computer program to implement the steps of the method of any one of claims 1-9.

Images