Human posture estimation method, electronic device, storage medium, and program product

Abstract

The application discloses a human pose estimation method, an electronic device, a storage medium and a program product, and relates to the technical field of image recognition. The method comprises the following steps: inputting a target image frame and an auxiliary image frame into a backbone convolution network to obtain a target frame feature and a first auxiliary frame feature; inputting the target frame feature and the first auxiliary frame feature into a motion alignment module to obtain a second auxiliary frame feature; inputting the target frame feature and the second auxiliary frame feature into an adaptive space-time fusion module to obtain a fused space-time feature; inputting the fused space-time feature into a semantic residual correction module to obtain a corrected space-time feature; and inputting the corrected space-time feature into a prediction convolution layer to obtain a human pose estimation result of the target image frame. The application can meet the stability and accuracy requirements of real scene human pose estimation.

Description

Technical Field

[0001] This application belongs to the field of image recognition technology, and in particular relates to a human pose estimation method, electronic device, storage medium and program product. Background Technology

[0002] Video pose estimation is a core task in computer vision, capable of outputting human joint localization results frame by frame, and is widely used in fields such as security monitoring, virtual-real interaction, and animation production. Real-world video scenes commonly suffer from problems such as rapid human movement, limb occlusion, and motion blur. Relying solely on single-frame spatial features easily leads to keypoint jitter and missed detections. Therefore, multi-frame temporal context modeling has become the core technical approach for high-precision video pose estimation.

[0003] Current mainstream solutions mainly improve the estimation accuracy of human pose by using multi-frame temporal modeling and optical flow technology, thereby mitigating detection errors caused by occlusion and motion.

[0004] However, existing technologies have many inherent flaws: the feature alignment stage is susceptible to motion blur interference, resulting in offset deviations; spatiotemporal modeling suffers from dimensional coupling problems, easily spreading noise and failing to dynamically adapt to changes in limb scale; and the decoding stage struggles to distinguish human symmetry semantics. Ultimately, this leads to pose localization drift and insufficient accuracy in complex scenarios, making it difficult to meet the stability and accuracy requirements of human pose estimation in real-world scenarios. Summary of the Invention

[0005] This application provides a human pose estimation method, electronic device, storage medium, and program product that can meet the stability and accuracy requirements of human pose estimation in real-world scenarios.

[0006] A first aspect of this application provides a human pose estimation method applied to a human pose estimation model. The human pose estimation model includes a backbone convolutional network, a motion alignment module, an adaptive spatiotemporal fusion module, a semantic residual correction module, and a predictive convolutional layer, comprising: The target image frame and the auxiliary image frame are input into the backbone convolutional network to obtain the target frame features and the first auxiliary frame features; the auxiliary image frame is the image frame before and after the target image frame. The target frame features and the first auxiliary frame features are input into the motion alignment module to obtain the second auxiliary frame features. The motion alignment module is used to eliminate the spatial misalignment between the auxiliary image frame and the target image frame. The second auxiliary frame features are auxiliary frame features that are spatially aligned with the target frame features. The target frame features and the second auxiliary frame features are input into the adaptive spatiotemporal fusion module to obtain fused spatiotemporal features; the adaptive spatiotemporal fusion module is used to perform multi-scale spatial feature fusion on the target frame features and the second auxiliary frame features. The fused spatiotemporal features are input into the semantic residual correction module to obtain the corrected spatiotemporal features; the semantic residual correction module is used to semantically enhance the fused spatiotemporal features. The corrected spatiotemporal features are input into the prediction convolutional layer to obtain the human pose estimation result of the target image frame.

[0007] A second aspect of this application provides an electronic device, the device comprising: a memory and a program or instructions stored in the memory and executable on a processor, wherein when the program or instructions are executed by the processor, they implement the human pose estimation method provided in any of the above-described embodiments of this application.

[0008] A third aspect of the embodiments of this application provides a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement the human pose estimation method provided by any aspect of the embodiments of this application described above.

[0009] A fourth aspect of the embodiments of this application provides a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the human pose estimation method provided in any of the embodiments of this application described above.

[0010] The human pose estimation method provided in this application first extracts the basic features of the target image frame and the preceding and following auxiliary image frames through a backbone convolutional network, providing a complete multi-frame feature foundation for temporal modeling. Then, the motion alignment module eliminates inter-frame spatial misalignment, solving the problems of motion blur interference and feature matching deviation in traditional alignment methods, and obtaining accurately aligned auxiliary frame features to ensure the effectiveness of temporal features. Subsequently, the adaptive spatiotemporal fusion module completes multi-scale fusion of multi-frame features, avoiding the defects of dimensional coupling and noise diffusion in traditional spatiotemporal modeling, dynamically adapting to changes in the scale of human limbs, fully mining effective spatiotemporal context information, and resisting occlusion interference. Furthermore, the semantic residual correction module performs semantic enhancement on the fused features, solving the pain point of symmetrical semantic confusion in existing technologies, optimizing feature expression, and correcting feature deviation. Finally, the pose estimation result is output by predicting the convolutional layer. Relying on the whole-process optimization system from preceding feature alignment, spatiotemporal fusion to semantic correction, various defects of traditional algorithms are avoided layer by layer, effectively suppressing key point jitter and positioning drift, and significantly improving the stability and accuracy of human pose estimation in complex real-world scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a human pose estimation method provided in one embodiment of this application; Figure 2 This is a schematic diagram of the overall architecture of a human pose estimation model provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0013] The features and exemplary embodiments of each aspect of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended only to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0014] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0015] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0016] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0017] In video human pose estimation applications, the feature alignment stage is susceptible to spatial offset deviations due to motion blur; the spatiotemporal modeling process suffers from dimensional coupling issues, leading to noise propagation and an inability to dynamically adapt to changes in limb scale; and the decoding stage struggles to distinguish human symmetry semantics. Consequently, pose localization is prone to drift in complex scenes, key point detection accuracy is insufficient, and the overall stability and accuracy of the system are constrained, thus affecting the practical application effects in fields such as security monitoring, virtual-real interaction, and animation production.

[0018] For example, in security monitoring scenarios, when pedestrians quickly cross densely populated areas, their limbs frequently occlude each other, and the high speed of movement causes motion blur in the images. In this scenario, existing technologies cannot accurately eliminate the spatial misalignment between auxiliary and target image frames during feature alignment, leading to increased alignment deviations between auxiliary and target frame features. Dimensional coupling issues during spatiotemporal modeling amplify noise and fail to adapt to limb movements at different scales, causing distortion in fused features. Furthermore, symmetrical joints, such as the left and right wrists, are difficult to distinguish during the decoding stage, resulting in keypoint localization errors. Consequently, pose estimation results exhibit keypoint jitter and missed detections, and the system cannot reliably output continuous and stable human motion sequences, thus affecting the accuracy of abnormal behavior recognition.

[0019] Based on this, this application provides a specific embodiment of a human pose estimation method, electronic device, storage medium, and program product, which can meet the stability and accuracy requirements of human pose estimation in real-world scenarios.

[0020] like Figure 1 The diagram illustrates a flowchart of a human pose estimation method. This method is applied to a human pose estimation model, which includes a backbone convolutional network, a motion alignment module, an adaptive spatiotemporal fusion module, a semantic residual correction module, and a predictive convolutional layer, comprising the following steps S110 to S150: S110, Input the target image frame and the auxiliary image frame into the backbone convolutional network to obtain the target frame features and the first auxiliary frame features; the auxiliary image frame is the image frame before and after the target image frame. S120, The target frame features and the first auxiliary frame features are input to the motion alignment module to obtain the second auxiliary frame features; the motion alignment module is used to eliminate the spatial misalignment between the auxiliary image frame and the target image frame, and the second auxiliary frame features are auxiliary frame features that are spatially aligned with the target frame features; S130, Input the target frame features and the second auxiliary frame features into the adaptive spatiotemporal fusion module to obtain fused spatiotemporal features; The adaptive spatiotemporal fusion module is used to perform multi-scale spatial feature fusion on the target frame features and the second auxiliary frame features; S140, The fused spatiotemporal features are input into the semantic residual correction module to obtain the corrected spatiotemporal features; the semantic residual correction module is used to semantically enhance the fused spatiotemporal features; S150, the corrected spatiotemporal features are input into the prediction convolutional layer to obtain the human pose estimation result of the target image frame.

[0021] For ease of understanding, the following explains some key terms in this embodiment: Human pose estimation methods refer to a series of processing steps using computer vision technology to identify and locate key points (such as joints and head) of the human body in images or videos, thereby inferring the human body's pose information. This method aims to achieve the understanding and analysis of human movement and behavior.

[0022] A human pose estimation model refers to a computational framework or algorithmic structure used to perform human pose estimation tasks. This model typically consists of multiple interconnected neural network modules, each responsible for specific feature extraction, information processing, or result prediction functions.

[0023] The backbone convolutional network is the fundamental feature extraction component in a deep learning model, typically composed of multiple convolutional layers, pooling layers, etc. Its main function is to extract visual features at different levels from the input image frames, providing rich semantic and spatial information for subsequent pose estimation tasks.

[0024] The motion alignment module is a component used to handle motion differences between different frames in a video sequence. This module calculates and compensates for spatial offsets between image frames, ensuring that features at different times are accurately aligned in space, thereby eliminating misalignment problems caused by human movement or camera movement.

[0025] The adaptive spatiotemporal fusion module is a component used to integrate feature information from different time points and spatial scales. This module can dynamically adjust the fusion strategy according to the feature content to effectively combine temporal context information and spatial details, generating more robust and comprehensive spatiotemporal fusion features.

[0026] The semantic residual correction module is a component used to semantically enhance the fused features. This module introduces a residual learning mechanism to fine-tune and correct the features, enhancing the expression of key semantic information and suppressing potential noise or irrelevant information, thereby improving the accuracy of pose estimation.

[0027] The predictive convolutional layer is the final output layer in a human pose estimation model. This layer typically consists of one or more convolutional operations, and its function is to transform the feature maps that have undergone multi-stage processing into the final human pose estimation result, such as generating heatmaps or directly predicting joint coordinates.

[0028] The target image frame refers to the current image frame in the video sequence from which human pose estimation needs to be performed. This frame is the core of the entire processing flow, and its pose information is the final output.

[0029] Auxiliary image frames refer to other image frames associated with the target image frame, typically adjacent frames on the timeline, such as the previous or next frame. These auxiliary image frames provide temporal context information, helping to compensate for the lack of information in a single frame.

[0030] Target frame features refer to the visual feature representation extracted from the target image frame by the backbone convolutional network. These features include information such as the texture, shape, and semantics of the target image frame.

[0031] The first auxiliary frame feature refers to the visual feature representation extracted from the auxiliary image frame by the backbone convolutional network. This feature is similar to the target frame feature, but corresponds to a different time point.

[0032] The second auxiliary frame feature refers to the auxiliary frame feature processed by the motion alignment module. This feature has been spatially aligned with the target frame feature, eliminating the misalignment caused by motion and enabling it to be fused with the target frame feature more effectively.

[0033] Fusion spatiotemporal features refer to the spatiotemporal information that combines the features of the target frame and the features of the second auxiliary frame after processing by the adaptive spatiotemporal fusion module. This feature contains richer and more robust spatiotemporal context information.

[0034] Corrected spatiotemporal features refer to features that have undergone semantic enhancement based on the fused spatiotemporal features after processing by the semantic residual correction module. These features have stronger semantic expressive power and help improve the accuracy of pose estimation.

[0035] Human pose estimation results refer to the final information about the human pose in a target image frame, which is predicted by the output of the convolutional layer. It is usually represented by two-dimensional or three-dimensional coordinates of human key points, or a heatmap representing the confidence of key points.

[0036] This embodiment provides a human pose estimation method, which is applied to a human pose estimation model, including a backbone convolutional network, a motion alignment module, an adaptive spatiotemporal fusion module, a semantic residual correction module, and a predictive convolutional layer.

[0037] First, the target image frame and auxiliary image frames are input into the backbone convolutional network to obtain the target frame features and the first auxiliary frame features. The auxiliary image frames can be the preceding and following frames of the target image frame. For example, when processing a video, the current frame can be selected as the target image frame, and its immediately preceding and following frames can be selected as auxiliary image frames. The backbone convolutional network can employ various existing structures, such as ResNet and HRNe, and its function is to extract deep semantic features from the image. In one implementation, the same backbone convolutional network can be shared, and images from different frames can be input sequentially for feature extraction.

[0038] Next, the target frame features and the first auxiliary frame features are input into the motion alignment module to obtain the second auxiliary frame features. This motion alignment module is used to eliminate the spatial misalignment between the auxiliary image frame and the target image frame, ensuring that the second auxiliary frame features are spatially aligned with the target frame features. For example, when a human body moves rapidly in a video, the position of the human body in the auxiliary image frame may deviate from the position of the human body in the target image frame. The motion alignment module can learn a transformation matrix to spatially transform the first auxiliary frame features, making them spatially correspond to the target frame features.

[0039] Next, the target frame features and the second auxiliary frame features are input into the adaptive spatiotemporal fusion module to obtain fused spatiotemporal features. This adaptive spatiotemporal fusion module is used to perform multi-scale spatial feature fusion of the target frame features and the second auxiliary frame features. For example, during the fusion process, feature information at different scales can be considered to capture the details and overall structure of different parts of the human body. In one implementation, the target frame features and the second auxiliary frame features can be directly concatenated or added element-wise, and then integrated through a convolutional layer to obtain multi-scale information.

[0040] Subsequently, the fused spatiotemporal features are input into the semantic residual correction module to obtain corrected spatiotemporal features. This semantic residual correction module is used to semantically enhance the fused spatiotemporal features. For example, the fused features may still contain some noise or incomplete semantic information, requiring further refinement. The semantic residual correction module can supplement and correct the fused spatiotemporal features by learning a residual mapping, thereby highlighting key semantic information related to human posture.

[0041] Finally, the corrected spatiotemporal features are input into the predictive convolutional layer to obtain the human pose estimation result for the target image frame. The predictive convolutional layer is responsible for converting the refined features into the final pose output. For example, this output can be a heatmap representing the positions of various joints in the human body, where the peak positions of the heatmap correspond to the coordinates of the joints.

[0042] This embodiment first extracts the basic features of the target image frame and the preceding and following auxiliary image frames through a backbone convolutional network, providing a complete multi-frame feature foundation for temporal modeling. Then, a motion alignment module eliminates inter-frame spatial misalignment, solving the problems of motion blur interference and feature matching deviation in traditional alignment methods, obtaining accurately aligned auxiliary frame features, and ensuring the effectiveness of temporal features. Subsequently, an adaptive spatiotemporal fusion module completes multi-scale fusion of multi-frame features, avoiding the defects of dimensional coupling and noise diffusion in traditional spatiotemporal modeling, dynamically adapting to changes in human limb scale, fully mining effective spatiotemporal context information, and resisting occlusion interference. Furthermore, a semantic residual correction module semantically enhances the fused features, solving the pain point of symmetrical semantic confusion in existing technologies, optimizing feature expression, and correcting feature deviations. Finally, by predicting the pose estimation results from the convolutional layer, relying on a full-process optimization system from preceding feature alignment, spatiotemporal fusion to semantic correction, various defects of traditional algorithms are avoided layer by layer, effectively suppressing keypoint jitter and positioning drift, and significantly improving the stability and accuracy requirements of human pose estimation in complex real-world scenarios.

[0043] In some embodiments, this application further proposes that S120 includes: Calculate the absolute difference between the features of the target frame and the features of the first auxiliary frame to obtain the difference feature map; The target frame features, the first auxiliary frame features, and the differential feature map are input into the motion estimation network to obtain the spatial offset and the modulation mask. The modulation mask is normalized to obtain the normalized modulation mask; Based on the spatial offset, deformable convolution is used to resample the regular sampling grid of the first auxiliary frame features to obtain the deformed auxiliary frame features. The features of the second auxiliary frame are obtained by processing the deformed auxiliary frame features based on the normalized modulation mask.

[0044] In this embodiment, the absolute difference between the target frame features and the first auxiliary frame features is calculated to obtain a difference feature map, which aims to capture the change information between the two frame features. This difference feature map can intuitively reflect the regions of motion or content change between image frames, providing a key input signal for subsequent motion estimation. This can be achieved by calculating the absolute difference between corresponding positions of the two feature maps element by element. For example, for feature maps F1 and F2, each element of the difference feature map D, D(i,j) = |F1(i,j) - F2(i,j)|.

[0045] The target frame features, the first auxiliary frame features, and the differential feature map are input into a motion estimation network to obtain a spatial offset and a modulation mask. The motion estimation network is a specially designed neural network whose function is to predict the motion information of the auxiliary frame relative to the target frame based on the input feature information. The output spatial offset is used to guide the spatial transformation of the features, while the modulation mask is used to adjust the strength or importance of the transformed features.

[0046] Normalizing the modulation mask yields a normalized modulation mask. The purpose of normalization is to adjust the numerical range of the modulation mask to a specific interval (e.g., [0, 1]) so that it can be stably applied to the weighted processing of auxiliary frame features after deformation. Normalization helps prevent instability caused by excessively large or small values ​​and ensures the reasonableness of the modulation effect. This normalization process can be achieved by using the Sigmoid activation function to process the modulation mask, mapping its output value to the (0, 1) interval.

[0047] Based on spatial offset, deformable convolution is used to deform and resample the regular sampling grid of the first auxiliary frame features, resulting in deformed auxiliary frame features. Deformable convolution is a convolution operation that can adaptively adjust the sampling position. By utilizing the spatial offset output by the motion estimation network, deformable convolution can deform the regular sampling grid of the first auxiliary frame features, allowing its sampling points to better align with the feature structure of the target image frame.

[0048] The deformed auxiliary frame features are processed using a normalized modulation mask to obtain second auxiliary frame features. The normalized modulation mask is used to weight or adjust the deformed resampled auxiliary frame features to further optimize their alignment with the target frame. The modulation mask can selectively enhance or suppress the deformed features based on the confidence or importance of motion estimation for different regions by the motion estimation network, thus obtaining more accurate and robust second auxiliary frame features.

[0049] The proposed solution generates a difference feature map by calculating the absolute difference between the target frame features and the first auxiliary frame features. This difference feature map visually reflects the change region between the two frame features, providing a direct basis for subsequent motion analysis. Subsequently, the target frame features, the first auxiliary frame features, and the difference feature map are input into a motion estimation network. This motion estimation network utilizes this multi-source information to learn and predict fine-grained spatial offsets and modulation masks. The spatial offset precisely describes the displacement of the auxiliary frame features relative to the target frame features, while the modulation mask indicates the motion confidence or importance of different regions. To ensure stable application of the modulation mask, it is normalized. Next, based on the predicted spatial offset, deformable convolution is used to deform and resample the regular sampling grid of the first auxiliary frame features. This adaptive sampling method allows the auxiliary frame features to be non-rigidly distorted, thereby achieving precise spatial alignment with the target frame features and generating deformed auxiliary frame features. Finally, the deformed auxiliary frame features are processed using a normalized modulation mask, such as by weighting, to further optimize the alignment effect, highlight areas with accurate motion estimation, and suppress areas with potential errors. This results in a second auxiliary frame feature spatially aligned with the target frame feature space. Through these steps, the motion alignment module effectively captures and compensates for complex motion patterns, significantly improving the spatial alignment accuracy between the auxiliary frame features and the target frame features. This provides high-quality input for the subsequent adaptive spatiotemporal fusion module, thereby enhancing the overall accuracy of human pose estimation.

[0050] Through the above technical solution, this application provides a refined motion alignment mechanism. This mechanism, by introducing differential feature maps as input to the motion estimation network, can more sensitively capture inter-frame changes, enabling the motion estimation network to generate more accurate spatial offsets and modulation masks. Utilizing deformable convolutions to perform non-rigid deformation resampling of auxiliary frame features based on these offsets effectively overcomes the limitations of traditional rigid transformations in handling complex motions, achieving pixel-level precise alignment. Furthermore, by weighting the deformed features using a normalized modulation mask, the importance of features can be dynamically adjusted according to the confidence level of motion estimation, further enhancing the robustness of the alignment effect. This refined motion alignment capability significantly improves the spatial consistency between auxiliary frame features and target frame features, providing high-quality, strongly aligned feature input for subsequent adaptive spatiotemporal fusion, thereby effectively avoiding fusion errors caused by feature misalignment and ultimately improving the accuracy and stability of human pose estimation.

[0051] In some embodiments, this application further proposes that, before inputting the target frame features, the first auxiliary frame features, and the differential feature map into the motion estimation network to obtain the spatial offset and modulation mask, the method further includes: The parameters of the terminal convolutional layers in the motion estimation network used for output spatial offset and modulation mask are initialized to zero so that deformation resampling is a standard regular convolution at the beginning of training.

[0052] In this embodiment, the terminal convolutional layer refers to the last one or more convolutional layers in the motion estimation network that are directly responsible for generating spatial offsets and modulation masks. These layers typically convert the feature maps extracted by the network into specific offset and mask values. Initializing the parameters (including weights and biases) of these layers to zero means that at the beginning of training, these layers will not perform any significant transformations or scaling on the input features, and their outputs will be close to zero.

[0053] Deformable resampling is the core operation of deformable convolution, adjusting the position of sampling points through spatial offsets. Standard regular convolution, on the other hand, samples on a fixed, regular grid. When the spatial offset and the output of the modulation mask are close to zero, the sampling points of deformable convolution will remain essentially on the regular grid, and the weighting effect of the modulation mask on the sampled values ​​will be minimized, making deformable resampling functionally equivalent to the standard regular convolution operation. This means that in the initial stages of training, the model will not introduce complex, potentially unstable deformations, but will instead begin learning from a simpler, more stable feature extraction pattern.

[0054] The proposed solution cleverly controls the behavior of deformable convolutions during the initial training phase by initializing the parameters of the terminal convolutional layers used to output spatial offsets and modulation masks in the motion estimation network to zero. Specifically, when the parameters of these terminal convolutional layers are set to zero, the spatial offsets output by the motion estimation network in the early stages of training will approach zero, and the modulation mask will also approach a constant value. In this initial state, the deformable resampling of the regular sampling grid of the first auxiliary frame features by the deformable convolution does not actually produce significant spatial deformation; its sampling points remain almost entirely at the original regular grid positions. Therefore, the deformable convolution functions as a standard regular convolution in the initial training phase, i.e., extracting features at fixed, regular sampling positions. This mechanism allows the model to start from a more stable and easier-to-learn state in the early stages of training, avoiding the drastic and meaningless deformations that might be caused by randomly initialized offsets and masks, thus laying a solid foundation for subsequent motion alignment learning.

[0055] Through the above technical solution, in the initial training phase of the human pose estimation model, the deformable convolutions in the motion alignment module can operate in a standard regular convolution manner. This effectively avoids unpredictable or unstable spatial offsets and modulation masks that may be caused by randomly initialized motion estimation network parameters, thus preventing the introduction of meaningless feature deformations in the early stages of training. This stable initial state helps the model converge faster, improves training stability, and provides a reliable baseline for subsequent refined motion alignment learning. Ultimately, this enables the motion alignment module to more effectively eliminate spatial misalignment between auxiliary image frames and target image frames, thereby improving the overall accuracy of human pose estimation.

[0056] In some embodiments, this application further proposes that S130 includes: Global average pooling and global max pooling are performed in parallel on the target frame features and the second auxiliary frame features to obtain merged pooled features. The merged pooled features are input into a multilayer perceptron to obtain the time-channel weight matrix; Based on the time-channel weight matrix, the target frame features and the second auxiliary frame features are aggregated element-wise to obtain the time-series denoising features. Perform three-dimensional convolution aggregation on the temporal denoising features to obtain two-dimensional spatial features; Based on two-dimensional spatial features, the spatiotemporal features of the fusion are determined.

[0057] In this embodiment, parallel execution of global average pooling and global max pooling is an effective feature extraction strategy. Global average pooling captures global contextual information by calculating the spatial average of each feature map, while global max pooling highlights the most salient feature responses by extracting the maximum value from each feature map. By applying these two pooling operations simultaneously, global information at different granularities can be comprehensively obtained from the target frame features and the second auxiliary frame features, including the overall distribution trend of features and key activation regions, thereby providing a richer and more comprehensive global description for the subsequent feature fusion process.

[0058] Merged pooling features are feature representations formed by combining the results of parallel global average pooling and global max pooling. This feature integrates global average information and global max information, and can more comprehensively reflect the overall distribution of input features and key activation points.

[0059] A multilayer perceptron is a feedforward neural network consisting of at least one input layer, one output layer, and one or more hidden layers, capable of learning nonlinear relationships in input data. Here, a multilayer perceptron is used to process pooled features to learn and generate a weight matrix that represents the importance of different channels.

[0060] The temporal-channel weight matrix is ​​a set of weights output by the multilayer perceptron, with dimensions corresponding to the number of channels in the feature map. Each element of this matrix represents the importance or saliency of the corresponding channel. By learning this weight matrix, the system can adaptively identify and emphasize feature channels that contribute more to human pose estimation during temporal fusion, while suppressing the influence of irrelevant or noisy channels.

[0061] Element-wise weighted aggregation refers to applying the learned temporal-channel weight matrix to each channel of the target frame feature and the second auxiliary frame feature, and then combining the weighted features. Specifically, each channel of each feature map is multiplied element-wise with its corresponding weight to adjust the contribution of that channel. Subsequently, these weighted features can be summed or concatenated element-wise to form a new feature that incorporates temporal information and has been adjusted for channel importance.

[0062] Temporal denoising features are feature representations obtained through element-wise weighted aggregation. These features fuse information from the target frame and auxiliary frames in the temporal dimension and undergo adaptive weight adjustment in the channel dimension, thereby effectively reducing noise and enhancing semantic information beneficial to pose estimation.

[0063] 3D convolutional aggregation is a convolutional operation capable of extracting features simultaneously in both spatial (height, width) and temporal (or channel-wise, when channels are considered depth) dimensions. Here, 3D convolutional aggregation is used to further process temporally denoised features to capture more complex spatiotemporal correlations. For example, a 3D convolutional kernel with a specific kernel size (e.g., 3x3x3) can be slid across the temporally denoised features to extract more discriminative spatiotemporal patterns from the fused features.

[0064] Two-dimensional spatial features are the output of three-dimensional convolutional aggregation operations. They further integrate the spatiotemporal information in temporal denoising features and ultimately transform them into a feature map with a two-dimensional spatial structure (height and width). This feature map contains spatiotemporal information that has been fused and refined through multiple stages, laying the foundation for the final determination of the fused spatiotemporal features.

[0065] Determining the spatiotemporal features of fusion refers to generating the final fusion features for subsequent semantic residual correction modules after the aforementioned series of processes. This step uses two-dimensional spatial features as a foundation, and through further refinement or direct output, forms the final spatiotemporal features of fusion.

[0066] This application's solution introduces a refined fusion strategy. First, it performs global average pooling and global max pooling in parallel on the target frame features and the second auxiliary frame features to comprehensively capture the global contextual and saliency information of both types of features, thus obtaining merged pooled features. This dual pooling mechanism ensures that the overall distribution of features and local key points can be considered simultaneously in subsequent processing, avoiding information loss that may result from a single pooling method. Subsequently, the merged pooled features are input into a multilayer perceptron, which learns to generate a temporal-channel weight matrix. This weight matrix can adaptively evaluate the importance of each channel in the target frame features and the second auxiliary frame features. In this way, the system can intelligently identify and emphasize feature channels that contribute more to human pose estimation, while suppressing channels that may contain noise or irrelevant information, thereby achieving effective noise reduction and information enhancement at the channel level. Then, based on this temporal-channel weight matrix, each channel of the target frame features and the second auxiliary frame features is weighted and aggregated element-wise to obtain temporal denoised features. This step is crucial for achieving adaptive fusion, enabling features from different time frames to be dynamically adjusted based on the importance of their channels during fusion, rather than simply being superimposed. This weighted aggregation effectively reduces noise that may be introduced by cross-frame fusion and enhances temporal consistency features, which are essential for pose estimation. To further capture the deep spatiotemporal correlations in the fused temporal denoising features, this application performs 3D convolutional aggregation on the temporal denoising features to obtain 2D spatial features. 3D convolution can extract features simultaneously in both spatial and (implicit) temporal dimensions, thus providing a more comprehensive understanding of the dynamic patterns and spatial structure in the feature maps. Finally, based on these 2D spatial features, the fused spatiotemporal features are determined. The entire process forms a complete chain from global information extraction, channel weight learning, adaptive weighted fusion to spatiotemporal depth aggregation, ensuring that the adaptive spatiotemporal fusion module can output high-quality, low-noise, and semantically rich fused spatiotemporal features, providing a solid foundation for subsequent semantic residual correction and final pose estimation.

[0067] Through the above technical solution, this application introduces a refined feature fusion mechanism in the adaptive spatiotemporal fusion module. By executing global average pooling and global max pooling in parallel, the global context and saliency information of features can be comprehensively captured, providing a rich basis for subsequent channel weight learning. The temporal-channel weight matrix learned by the multilayer perceptron enables the system to adaptively identify and emphasize feature channels that contribute more to human pose estimation, effectively suppressing noise and irrelevant information. The combination of element-wise weighted aggregation and 3D convolutional aggregation further ensures that, during the fusion process, not only is adaptive noise reduction achieved at the channel level, but also the spatiotemporal correlation in the fused features is deeply mined. Therefore, this solution can generate high-quality, low-noise, and semantically rich fused spatiotemporal features, significantly improving the robustness and accuracy of the human pose estimation model for complex scenes and dynamic poses.

[0068] In some embodiments, this application further proposes determining fused spatiotemporal features based on two-dimensional spatial features, including: Two-dimensional spatial features are synchronously input into multiple parallel depthwise separable convolutional branches to obtain multi-scale spatial features; each depthwise separable convolutional branch has a different convolutional kernel size. Based on the spatial gating weight matrix corresponding to each depth separable convolutional branch, the multi-scale spatial features are weighted and summed to obtain the fused spatiotemporal features.

[0069] In this embodiment, two-dimensional spatial features are synchronously input to multiple parallel depthwise separable convolutional branches, aiming to extract spatial information at different scales from the input features. The two-dimensional spatial features are feature representations after temporal denoising and 3D convolutional aggregation, containing spatial information fused from the target image frame and auxiliary image frames in both temporal and channel dimensions. Depthwise separable convolution is an efficient convolution operation that decomposes standard convolution into depthwise convolution (convolution is performed independently on each input channel) and pointwise convolution (1x1 convolution, used to combine the output channels of the depthwise convolution), thus significantly reducing the number of model parameters and computational cost. Multiple parallel depthwise separable convolutional branches mean that these convolutional branches simultaneously receive the same two-dimensional spatial features as input and perform feature extraction independently. For example, three parallel depthwise separable convolutional branches can be set up, each containing one depthwise convolutional layer and one pointwise convolutional layer.

[0070] Obtaining multi-scale spatial features refers to the set of features extracted from two-dimensional spatial features and captured at different spatial resolutions or receptive fields after the parallel branching process described above. These features can reflect information about objects or textures of different sizes in the image.

[0071] Having different kernel sizes for each depthwise separable convolutional branch is key to achieving multi-scale feature extraction. For example, kernels of different sizes such as 3x3, 5x5, and 7x7 can be used. Smaller kernels (such as 3x3) can capture local details, while larger kernels (such as 5x5 or 7x7) can capture broader contextual information.

[0072] Based on the spatial gating weight matrices corresponding to the separable convolutional branches at each depth, multi-scale spatial features are weighted and summed to dynamically adjust the importance of features at different scales and fuse them. The spatial gating weight matrix is ​​a spatially related weight map used to adaptively assign weights according to the feature content. The weighted summation involves multiplying the spatial features at different scales by their corresponding spatial gating weight matrices, and then adding these weighted feature maps element-wise to obtain the final fused spatiotemporal features.

[0073] The proposed scheme simultaneously inputs 2D spatial features into multiple parallel depthwise separable convolutional branches, configuring different kernel sizes for each branch. This allows the model to capture spatial information at different scales within the 2D features, resulting in rich multi-scale spatial features. This design enables the model to simultaneously focus on local details and global contextual information in the image. Furthermore, based on the spatial gating weight matrices corresponding to each depthwise separable convolutional branch, the multi-scale spatial features are weighted and summed, allowing the model to dynamically select and emphasize the most relevant spatial scale information according to the image content. This adaptive fusion mechanism enables the model to more flexibly handle pose estimation tasks in different scenarios, effectively addressing the limitations of single-scale feature extraction and enhancing the expressive power of fused spatiotemporal features and adaptability to complex scenes.

[0074] Through the above technical solution, this application can effectively extract and fuse multi-scale spatial information from two-dimensional spatial features. This adaptive multi-scale fusion mechanism enables the model to dynamically select and emphasize the most relevant spatial scale information according to the image content, thereby significantly enhancing the expressive power of fused spatiotemporal features and adaptability to complex scenes, providing more accurate and robust input for subsequent semantic enhancement and human pose estimation.

[0075] In some embodiments, this application further proposes that S140 includes: Global spatial pooling is performed on the fused spatiotemporal features to obtain the target pooling result; The target pooling result is input into the fully connected layer to obtain the channel recalibration weights; Enhanced semantic features are obtained by weighting the fused spatiotemporal features one channel at a time based on the channel recalibration weights. The enhanced semantic features are input into the convolutional layer to extract semantic residual features; The semantic residual features are superimposed on the fused spatiotemporal features to obtain the corrected spatiotemporal features.

[0076] In this embodiment, global spatial pooling is an operation that aggregates the spatial dimensional information of a feature map into a single numerical value or a smaller dimensional representation. It aims to extract global contextual information from the entire feature map, capture the overall semantic content of the feature map, and thus provide global awareness for subsequent channel recalibration.

[0077] The target pooling result is a vector or tensor that represents the global semantic information of the fused spatiotemporal features after global spatial pooling operation. It is used as the input of the fully connected layer to learn the dependencies between channels and then generate channel recalibration weights.

[0078] A fully connected layer is a type of layer in a neural network in which each input neuron is connected to each output neuron. It is used to learn the complex nonlinear mapping relationship between input features and output features, thereby generating weights for each channel based on global context information.

[0079] Channel recalibration weights are a set of weight values ​​corresponding to the number of channels in the fused spatiotemporal features. Each weight value is used to adjust the importance of the corresponding channel, aiming to weight each channel of the fused spatiotemporal features to enhance the semantic information of important channels and suppress noise or redundant information in unimportant channels.

[0080] Channel-wise weighting is an operation that multiplies the channel recalibration weights element-wise with the corresponding channels of the fused spatiotemporal features. Based on the learned channel importance, the activation intensity of each channel is dynamically adjusted, thereby highlighting semantic features that contribute more to pose estimation.

[0081] Enhanced semantic features are fused spatiotemporal features that enhance semantic information through channel-wise weighting. These enhanced features serve as input for subsequent extraction of semantic residual features, ensuring that the extracted residual features are based on more discriminative semantic information. Convolutional layers are layers in neural networks that scan and extract features from the input feature map using convolutional kernels, extracting more refined and discriminative semantic residual information from the enhanced semantic features.

[0082] Semantic residual features are features extracted from enhanced semantic features by convolutional layers, representing the increment or correction of semantic information. They are used to superimpose on the original fused spatiotemporal features to correct and enhance their semantic expression, thereby obtaining more accurate corrected spatiotemporal features.

[0083] The overlay operation adds two feature maps element by element. Borrowing the idea of ​​residual connection, semantic residual features are used as a correction signal and directly added to the fused spatiotemporal features to enhance the semantic expressive power of the features while avoiding the gradient vanishing problem.

[0084] Corrected spatiotemporal features are spatiotemporal features with richer semantic information and stronger expressive power after semantic residual correction. They are used as input to the predictive convolutional layer to generate the final human pose estimation result, and their quality directly affects the accuracy of the final estimation.

[0085] The proposed scheme extracts global contextual information from the fused spatiotemporal features by performing global spatial pooling, and then inputs this information into a fully connected layer to learn channel recalibration weights. These weights are used to weight the fused spatiotemporal features channel by channel, thereby obtaining enhanced semantic features. This process dynamically adjusts the importance of each channel, enhancing the semantic information that is more critical for pose estimation. Subsequently, the enhanced semantic features are input into a convolutional layer to extract semantic residual features, which represent further corrections and supplements to the semantic information. Finally, these semantic residual features are superimposed on the original fused spatiotemporal features to obtain corrected spatiotemporal features. This overall process effectively solves the problems of insufficient semantic information or insufficient robustness of feature representation that may exist in fused spatiotemporal features. By explicitly learning and applying semantic corrections, the feature representation becomes more discriminative, providing higher-quality input for subsequent pose estimation.

[0086] Through the above technical solutions, this application significantly enhances the semantic richness and discriminative ability of fused spatiotemporal features by explicitly performing semantic residual correction. The combination of global spatial pooling and fully connected layers enables dynamic recalibration of channel importance based on global contextual information, ensuring that semantic information more relevant to pose estimation is prioritized. The residual connection mechanism allows the model to refine features by adding learned semantic corrections, effectively mitigating the impact of semantic ambiguity or noise that may exist in the initial fused features. Therefore, the obtained corrected spatiotemporal features have stronger robustness and more accurate semantic alignment, thereby significantly improving the accuracy and reliability of human pose estimation results for target image frames.

[0087] In some embodiments, this application further proposes that S150 includes: The corrected spatiotemporal features are input into a single convolutional layer for channel dimensionality reduction to obtain a spatial heatmap; the number of channels after dimensionality reduction is equal to the number of human joints. The peak coordinates of each channel are extracted from the spatial heat map and used as the predicted coordinates of each key point. Based on the predicted coordinates of each joint, the human pose estimation result of the target image frame is determined.

[0088] In this embodiment, the corrected spatiotemporal features are the result of processing by the semantic residual correction module. They contain rich spatiotemporal context information and semantic enhancement information, providing high-quality input for subsequent pose estimation.

[0089] A single convolutional layer is a basic unit of neural network operation. Its function is to scan the input feature map and extract features through a series of learnable convolutional kernels.

[0090] Channel dimensionality reduction refers to reducing the number of channels in a feature map through convolution operations. This can be achieved by setting the number of output channels of the convolutional layer to match the number of joints in the target human body, thereby mapping high-dimensional abstract features to a specific dimension corresponding to each joint.

[0091] A spatial heatmap is a two-dimensional feature representation where each channel corresponds to a specific human joint. Within each channel, the pixel value represents the probability or confidence level of that joint occurring at different locations in the image space. Heatmaps can visually display the possible locations of each joint in an image and their confidence level distribution.

[0092] The setting that the number of channels after dimensionality reduction is equal to the number of human joints ensures that each independent human joint has a dedicated channel to represent its spatial probability distribution. For example, if the human pose model defines N joints, the number of output channels of the convolutional layer will be precisely set to N.

[0093] Extracting the peak coordinates of each channel aims to identify the pixel with the highest activation value (i.e., the highest probability or confidence) from the spatial heatmap channels corresponding to each key point. This can be achieved by directly finding the two-dimensional coordinates of the maximum value.

[0094] The predicted coordinates of each joint point refer to the specific location of each joint point in the two-dimensional image space obtained through the peak extraction process described above. These coordinates are usually represented in pixel coordinates, directly indicating the precise location of the joint point in the image.

[0095] The step of determining the human pose estimation result of the target image frame involves integrating all the predicted joint coordinates to form a complete and structured representation of the human pose. This can be represented as a list or array of joint coordinates, or further, by connecting these joints to draw a human skeleton diagram, thus visually presenting the pose of the person in the target image frame.

[0096] The proposed solution achieves accurate estimation of human pose by inputting semantically enhanced corrected spatiotemporal features into a prediction convolutional layer. Specifically, the corrected spatiotemporal features are first fed into a single convolutional layer. This convolutional layer, through a carefully designed convolutional kernel, performs channel dimensionality reduction on the input corrected spatiotemporal features, with the number of output channels precisely set to be equal to the total number of human joints. Through this operation, the abstract, high-dimensional corrected spatiotemporal features are effectively transformed into a series of spatial heatmaps, where each heatmap channel is specifically responsible for representing the probability distribution of a particular human joint in the image space. This transformation mechanism enables the model to map complex feature information onto an intuitive, localizable joint probability distribution. Subsequently, for each joint's corresponding spatial heatmap channel, the system accurately extracts its peak coordinates. The peak coordinates represent the most likely location of the joint in the image, i.e., the spatial point with the highest confidence. By performing this operation on all joint channels, the predicted coordinates of each joint in the target image frame can be obtained. Ultimately, these independent keypoint predicted coordinates are integrated to form the complete human pose estimation result for the target image frame. This scheme cleverly utilizes the corrected spatiotemporal features provided by preceding modules (including a backbone convolutional network, motion alignment module, adaptive spatiotemporal fusion module, and semantic residual correction module), which have undergone spatiotemporal fusion and semantic enhancement. These high-quality feature inputs enable the predictive convolutional layers to generate more accurate and robust spatial heatmaps. Because the corrected spatiotemporal features have effectively eliminated spatial misalignment, fused multi-scale information, and undergone semantic enhancement, the probability distribution of each keypoint is clearer and peak localization is more accurate during the heatmap generation stage, thus significantly improving the accuracy and stability of the final human pose estimation result.

[0097] Through the above technical solution, this application can efficiently and accurately convert complexly processed corrected spatiotemporal features into specific human joint coordinates. By inputting the corrected spatiotemporal features into a single convolutional layer for channel dimensionality reduction, and ensuring that the number of channels after dimensionality reduction is equal to the number of human joints, a spatial heatmap corresponding to each joint can be directly generated, thus laying the foundation for subsequent accurate joint localization. Extracting peak coordinates from these spatial heatmaps allows for the direct and effective acquisition of the predicted positions of each joint, avoiding complex post-processing steps. Finally, the human pose estimation result is determined based on these predicted coordinates, making the entire pose estimation process more direct and efficient. Given that the corrected spatiotemporal features have been fully optimized and enhanced through modules such as motion alignment, adaptive spatiotemporal fusion, and semantic residual correction, they contain highly accurate and robust spatiotemporal and semantic information. Therefore, using such high-quality features as input to the prediction convolutional layer ensures that the generated spatial heatmap has higher confidence and clearer peaks, thereby significantly improving the accuracy and stability of the predicted coordinates of each joint.

[0098] In some embodiments, this application further proposes a training phase for the human pose estimation model, including: During the first N training cycles, the adaptive spatiotemporal fusion module and the semantic residual correction module are frozen, and the network parameters of the backbone convolutional network and the motion alignment module are updated using the initial learning rate. Over the next M training cycles, the adaptive spatiotemporal fusion module is unfrozen, and the network parameters of the backbone convolutional network, motion alignment module, and adaptive spatiotemporal fusion module are updated using the first learning rate. In the final L training cycles, the semantic residual correction module is unfrozen, and the network parameters of the backbone convolutional network, motion alignment module, adaptive spatiotemporal fusion module, and semantic residual correction module are updated using a second learning rate; the second learning rate is less than the first learning rate, and the first learning rate is less than the initial learning rate.

[0099] In this embodiment, the training phase refers to the process by which a deep learning model adjusts its internal parameters by learning from training data so that it can perform human pose estimation tasks.

[0100] "Freezing" specific modules refers to keeping their network parameters unchanged during training and not updating gradients. This is typically achieved by setting the parameters of these modules to be untrainable, or by not calculating the gradients of these modules during backpropagation. Its purpose is to concentrate computational resources and gradient updates on the model's fundamental feature extraction and motion alignment in the early stages of training, preventing complex modules that are not yet fully trained from introducing noise or unstable gradients, thereby stabilizing the training process.

[0101] The learning rate is a hyperparameter in deep learning optimization algorithms that determines the step size at which the model updates the network parameters in each iteration. The initial learning rate is usually a relatively large value so that the model can quickly explore the parameter space in the early stages of training.

[0102] The backbone convolutional network is responsible for extracting basic visual features of the image, while the motion alignment module handles spatial misalignment between temporal frames. Prioritizing the updating of the parameters of these basic modules during the initial training phase helps the model establish stable feature extraction and temporal alignment capabilities. "Unfreezing" refers to resetting the parameters of previously frozen modules to a trainable state, enabling them to participate in gradient updates during subsequent training.

[0103] The adaptive spatiotemporal fusion module is responsible for fusing multi-scale spatial features from the target frame and auxiliary frame features. After the backbone network and motion alignment module have stabilized initially, this module is unfrozen, allowing it to learn more advanced spatiotemporal fusion strategies based on relatively stable features. The initial learning rate is typically lower than the initial learning rate. During the middle stages of training, as more modules are added and the model parameter space expands, reducing the learning rate helps the model finely adjust parameters, avoiding oscillations near the optimal solution and thus improving convergence accuracy.

[0104] The semantic residual correction module is used to semantically enhance the fused spatiotemporal features. After the model's basic feature extraction, motion alignment, and spatiotemporal fusion capabilities have been trained to a certain extent, unfreezing this module allows it to learn how to further enhance semantic information based on richer and more accurate spatiotemporal features, thereby improving the accuracy of pose estimation. The second learning rate is usually smaller than the first learning rate. In the final stage of training, all modules participate in training, and the learning rate is further reduced. This allows the model to fine-tune, searching for a more accurate optimal solution in the entire parameter space, thereby maximizing the model's performance.

[0105] The second learning rate is less than the first learning rate, and the first learning rate is less than the initial learning rate. This decreasing learning rate strategy allows the model to converge quickly in the early stages of training, while fine-tuning is performed in the later stages of training to avoid overfitting and improve the model's generalization ability.

[0106] This application employs a phased, progressive training strategy to effectively address the stability and convergence issues that may arise during the training of complex, multi-module human pose estimation models. In the initial training phase, the model prioritizes basic feature extraction and motion alignment capabilities. By freezing the adaptive spatiotemporal fusion module and the semantic residual correction module, and updating the network parameters of the backbone convolutional network and the motion alignment module using a relatively high initial learning rate, the model can quickly and stably learn effective target frame features and first auxiliary frame features when processing target and auxiliary image frames. Based on this, the motion alignment module eliminates spatial misalignment between the auxiliary and target image frames, obtaining second auxiliary frame features aligned with the target frame feature space. This strategy avoids gradient instability or noise interference that might result from premature training of complex modules before the model has established a solid foundation, thus laying a strong foundation for subsequent, more advanced feature processing. As training progresses, after the parameters of the backbone convolutional network and the motion alignment module have initially converged, the model enters the second training phase. At this point, the adaptive spatiotemporal fusion module is unfrozen and a relatively low first learning rate is adopted, allowing the network parameters of the backbone convolutional network, motion alignment module, and adaptive spatiotemporal fusion module to be updated collaboratively. The adaptive spatiotemporal fusion module begins learning how to fuse multi-scale spatial features of the target frame features and the second auxiliary frame features to obtain fused spatiotemporal features. This progressive unfreezing and training approach allows the model to gradually learn how to effectively integrate spatiotemporal information, given its already good basic features and alignment capabilities. This avoids the problem of the fusion module failing to learn effectively due to a lack of reliable input when the overall model structure is not yet stable. Finally, after the model's feature extraction, motion alignment, and spatiotemporal fusion capabilities have been sufficiently trained, the third training phase begins. At this point, the semantic residual correction module is unfrozen and a lower second learning rate is adopted to fine-tune the network parameters of all modules (backbone convolutional network, motion alignment module, adaptive spatiotemporal fusion module, and semantic residual correction module). The semantic residual correction module performs semantic enhancement on the fused spatiotemporal features at this stage to obtain corrected spatiotemporal features, which are ultimately used by the predictive convolutional layer to obtain the human pose estimation result of the target image frame. This training strategy, which progresses from coarse to fine and from simple to complex, enables the model to gradually master the complex mapping relationship from basic features to high-level semantics. Furthermore, by progressively reducing the learning rate, it ensures that the model can be finely adjusted in the later stages of training, thereby maximizing the model's performance and generalization ability while maintaining training stability.

[0107] The following is a specific example to illustrate this. As a concrete implementation, the training phase of the human pose estimation model can be performed as follows: First, in the first N training epochs, for example, N can be set to 50 training epochs, the parameters of the adaptive spatiotemporal fusion module and the semantic residual correction module are set to an untrainable state, i.e., these two modules are frozen. Simultaneously, an initial learning rate, for example 0.001, is used to update the network parameters of the backbone convolutional network and the motion alignment module. Second, in the next M training epochs, for example, M can be set to 100 training epochs, the adaptive spatiotemporal fusion module is unfrozen, making its parameters trainable. At this time, a first learning rate, for example 0.0001, is used to update the network parameters of the backbone convolutional network, the motion alignment module, and the adaptive spatiotemporal fusion module. In this stage, the model begins to learn how to effectively fuse spatiotemporal features from the target image frame and the auxiliary image frame. Finally, in the last L training epochs, for example, L can be set to 50 training epochs, the semantic residual correction module is unfrozen, making its parameters also trainable. At this point, a second learning rate, such as 0.00001, is used to update all network parameters of the backbone convolutional network, motion alignment module, adaptive spatiotemporal fusion module, and semantic residual correction module. It's important to note that the second learning rate of 0.00001 is less than the first learning rate of 0.0001, and the first learning rate of 0.0001 is less than the initial learning rate of 0.001. This decreasing learning rate strategy helps the model to converge stably and achieve better performance in the later stages of training.

[0108] By employing the aforementioned phased and progressive training strategy, this application effectively addresses the convergence difficulties, training instability, and poor performance issues that may arise during the training of complex multi-module human pose estimation models. In the initial training phase, the basic feature extraction and motion alignment modules are trained first, providing stable and high-quality input features for subsequent, more complex spatiotemporal fusion and semantic enhancement modules, thus avoiding training oscillations caused by inconsistent learning progress between modules. As training progresses, new modules are gradually unfrozen and introduced, coupled with a decreasing learning rate, enabling the model to progressively learn different levels of feature representation and processing capabilities, thereby ensuring that each module can be optimized in a relatively stable environment. This strategy significantly improves the model's training efficiency and stability, allowing the human pose estimation model to converge better to the global optimum, ultimately achieving higher accuracy and more robust human pose estimation results.

[0109] In some embodiments, this application further proposes that the method also includes: Obtain the initial training sample set. Each training sample in the initial training sample set contains video frame images and their true annotations of key points. Calculate the image quality evaluation index for each training sample. The image quality evaluation index includes at least one of image blur, occlusion ratio, and human joint visibility. The training sample set after filtering is obtained by selecting training samples whose image quality evaluation indicators meet the preset image quality conditions from the initial training sample set; The human pose estimation model is trained based on the selected training sample set.

[0110] In this embodiment, the initial training sample set is the basic data source for training the human pose estimation model. Each sample typically consists of the original video frame image and the precise coordinate annotations of the corresponding human key points (such as head, shoulders, elbows, knees, etc.). These annotations can be manually marked frame by frame by professionals, or completed with the assistance of semi-automatic annotation tools.

[0111] Image quality assessment metrics are used to quantify the quality level of training samples. Image blur measures the sharpness of an image. For example, it can be quantified by calculating the gradient magnitude, Laplacian operator variance, or Fourier transform high-frequency energy; alternatively, a pre-trained deep learning model can be used to predict the blur level. Occlusion ratio measures the degree to which a human body or its keypoints are occluded by other objects. For example, it can be calculated by analyzing the overlap between the human body's bounding box and the image boundary or other objects in the scene, or by using semantic segmentation techniques to identify occluders and assess their coverage of the human body; it can also be achieved by manually labeling or using semi-automatic tools to mark occluded keypoints or regions and calculating their proportion of the total area. Human joint visibility measures whether human keypoints are clearly visible in an image. For example, it can be evaluated by analyzing the confidence scores of keypoint annotations, or by using a specialized visibility prediction model; it can also be determined by manual inspection or based on local image features (such as contrast and edge information).

[0112] The initial training sample set selection process aims to remove low-quality training samples and retain high-quality samples to optimize the model's training data. For example, image blur thresholds (e.g., blur value must be higher than a certain preset value), occlusion ratio thresholds (e.g., occlusion ratio must be lower than a certain preset value), and / or human joint visibility thresholds (e.g., visibility must be higher than a certain preset value) can be set. Only samples that meet all preset conditions will be retained.

[0113] Finally, the human pose estimation model is trained based on the selected training sample set. Using a selected high-quality training sample set to train the human pose estimation model can significantly improve the model's learning efficiency and final performance.

[0114] This application's solution introduces a training sample quality screening mechanism to ensure that the human pose estimation model receives high-quality, high-reliability data input during the training phase. Specifically, firstly, an initial training sample set containing video frame images and their ground truth annotations is acquired, providing foundational data for model training. Then, for each training sample, image quality evaluation metrics such as image blur, occlusion ratio, and human joint visibility are calculated. These metrics objectively quantify the visual quality and annotation reliability of each sample. Based on these evaluation metrics, the system rigorously screens the initial training sample set according to preset image quality conditions, removing blurry, severely occluded, or low-visibility keypoint samples, thus obtaining a high-quality, screened training sample set. Finally, the human pose estimation model will be trained solely on this screened, high-quality dataset. Through this mechanism, the model can avoid learning erroneous or unrobust features from low-quality, noisy samples, enabling the backbone convolutional network to extract more accurate features, the motion alignment module to more effectively eliminate spatial misalignment, the adaptive spatiotemporal fusion module to perform more precise multi-scale spatial feature fusion, and the semantic residual correction module to perform more effective semantic enhancement. This high-quality data-driven training method enables the model to focus more on learning the essential features and spatiotemporal correlations of human posture, significantly improving the accuracy and generalization ability of the model in human posture estimation in complex scenarios.

[0115] The above technical solution effectively avoids the negative impact of low-quality training samples on model performance during the training phase of the human pose estimation model. By rigorously evaluating and screening the training samples, the model learns only from clear samples with minimal occlusion and high keypoint visibility, thereby significantly improving the training efficiency and final pose estimation accuracy. This method enables the model to learn more robust and generalizable feature representations, especially when dealing with complex and varied human poses and environments, producing more accurate and stable estimation results, effectively solving the technical problem of performance degradation caused by training data quality issues.

[0116] In some embodiments, this application further proposes training a human pose estimation model based on a filtered training sample set, including: Calculate the occlusion ratio and motion blur degree of each training sample in the filtered training sample set; Based on the occlusion ratio and motion blur degree of each training sample, sample weights are assigned to each training sample; among them, training samples with higher occlusion ratios and higher motion blur degrees are assigned higher sample weights. The weighted loss value is determined based on the sample weights of each training sample; The network parameters of the human pose estimation model are updated based on the weighted loss value.

[0117] In this embodiment, calculating the occlusion ratio and motion blur degree of each training sample in the filtered training sample set is a crucial step. The occlusion ratio refers to the degree to which a human target is occluded by other objects in an image, and its calculation can be performed in various ways. For example, a pre-trained semantic segmentation model can be used to identify the human body region and potential occluding object regions, and then the overlap ratio between the occluding object region and the human body region can be calculated. Motion blur degree refers to the degree of blurring in an image caused by the movement of objects or the camera, and its calculation can be based on frequency domain analysis of the image, such as quantifying the blur degree by analyzing the attenuation of high-frequency components using Fourier transform.

[0118] After calculating the occlusion ratio and motion blur level, sample weights are assigned to each training sample based on these ratios. Sample weights indicate the level of attention the model pays to specific samples during training. The purpose of assigning sample weights is to guide the model to focus more on challenging samples, thereby improving model performance in complex scenes. This can be achieved by normalizing the occlusion ratio and motion blur level, and then mapping them to a weight value using a linear combination or a non-linear function (such as the sigmoid function or an exponential function).

[0119] Subsequently, a weighted loss value is determined based on the sample weights of each training sample. The weighted loss value is obtained by multiplying the loss of each sample by its corresponding sample weight when calculating the difference between the model's prediction and the ground truth label. Its purpose is to ensure that samples with higher weights contribute more to the total loss, thereby generating larger gradients during backpropagation and prompting the model to prioritize learning the features of these difficult samples.

[0120] Finally, the network parameters of the human pose estimation model are updated based on the weighted loss value. Network parameter updating refers to using optimization algorithms to adjust the internal parameters of the human pose estimation model (such as convolutional kernel weights, biases, etc.) according to the calculated weighted loss value, in order to reduce the loss and improve the model's prediction accuracy. Its purpose is to enable the model to learn effective feature representations and mapping relationships from the training data.

[0121] The proposed solution, when training the human pose estimation model, first calculates the occlusion ratio and motion blur level for each training sample in the selected training sample set. These metrics objectively reflect the inherent complexity of the samples and the challenge they pose to the model's learning. Subsequently, based on these calculated occlusion ratios and motion blur levels, a sample weight is dynamically assigned to each training sample. This weighting mechanism is designed to give higher weights to samples with higher occlusion ratios and motion blur levels, thus granting these more challenging samples greater learning priority during training. During the loss calculation phase, the model determines a weighted loss value based on the weight of each sample, meaning that high-weighted, difficult samples will have a greater impact on the total loss function. Finally, the optimizer updates the network parameters of the human pose estimation model based on this weighted loss value. Through this mechanism, the model is guided to focus more on and learn from scenarios that are more common and challenging in the real world (such as blur caused by severe occlusion or vigorous motion), effectively overcoming the difficulty differences that still exist in the selected sample set. This weighted training strategy, combined with the previous method of selecting training samples through image quality assessment, forms a more complete and robust training process. It not only ensures the overall quality of the training data but also optimizes the model's learning efficiency and depth of complex information in the data.

[0122] Through the above technical solution, even in a pre-selected training sample set, the model can adaptively learn based on the actual difficulty of the samples (such as the degree of occlusion and motion blur). This weighted training strategy allows the model to more effectively focus on and learn from challenging samples during training, thereby significantly improving the robustness and generalization ability of the human pose estimation model in complex, real-world scenarios (such as those with severe occlusion or intense motion blur). The model no longer treats all samples equally but can selectively extract deeper features from difficult samples, thus improving the overall pose estimation accuracy and stability.

[0123] It should be noted that the various embodiments described in this application can be combined with each other or implemented individually without conflict, and this application does not limit this.

[0124] For ease of understanding, a specific embodiment will be used as an example: like Figure 2The diagram illustrates the overall architecture of a human pose estimation model. To overcome the limitations of traditional strong spatiotemporal coupling, a progressive modeling paradigm from low-level visual alignment to high-level structural correction is constructed: the backbone convolutional network HRNet maintains multi-resolution branches in parallel to preserve fine-grained spatial semantics; subsequently, the Motion Alignment Module (MAM) uses explicit feature differences to drive deformable convolutions, aiming to establish anti-blurring temporal correlations and eliminate inter-frame spatial misalignment. The Adaptive Spatio-temporal Fusion Module (ASF) adopts a spatiotemporal decoupling strategy, using cross-frame attention to suppress temporal noise in damaged frames, and dynamically allocating multi-scale receptive fields in space using a pixel-level gating mechanism to reconstruct damaged features. The Semantic Residual Correction Module (SRC) uses a channel attention mechanism to suppress semantic confusion in symmetrical parts, and achieves refined correction of the anatomical consistency of keypoint heatmaps through a spatial residual smoothing mechanism.

[0125] Based on the human pose estimation method provided above in this application, this application also provides a specific embodiment of an electronic device.

[0126] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown. The electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0127] Specifically, the processor 301 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0128] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 302 may include removable or non-removable (or fixed) media. Where appropriate, memory 302 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 302 is non-volatile solid-state memory.

[0129] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0130] The processor 301 reads and executes computer program instructions stored in the memory 302 to implement any of the human pose estimation methods in the above embodiments.

[0131] In one example, the electronic device may also include a communication interface 303 and a bus 310. For example, Figure 3 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 310 and complete communication with each other.

[0132] The communication interface 303 is mainly used to realize communication between each module, device, unit and / or equipment in the embodiments of this application.

[0133] Bus 310 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 310 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0134] Furthermore, in conjunction with the human pose estimation methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the human pose estimation methods in the above embodiments.

[0135] Furthermore, in conjunction with the human pose estimation methods in the above embodiments, this application embodiment can provide a computer program product for implementation. When the instructions in this computer program product are executed by the processor of an electronic device, the electronic device performs any of the human pose estimation methods in the above embodiments.

[0136] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0137] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0138] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0139] Each aspect of this disclosure has been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0140] The above description is merely a specific embodiment of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of each equivalent modification or substitution within the technical scope disclosed in this application, and such modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for estimating human pose, characterized in that, The method is applied to a human pose estimation model, which includes a backbone convolutional network, a motion alignment module, an adaptive spatiotemporal fusion module, a semantic residual correction module, and a predictive convolutional layer. The method includes: The target image frame and the auxiliary image frame are input into the backbone convolutional network to obtain the target frame features and the first auxiliary frame features; the auxiliary image frame is the image frame before and the image frame after the target image frame. The target frame feature and the first auxiliary frame feature are input to the motion alignment module to obtain the second auxiliary frame feature; the motion alignment module is used to eliminate the spatial misalignment between the auxiliary image frame and the target image frame, and the second auxiliary frame feature is an auxiliary frame feature that is spatially aligned with the target frame feature; The target frame features and the second auxiliary frame features are input into the adaptive spatiotemporal fusion module to obtain fused spatiotemporal features; the adaptive spatiotemporal fusion module is used to perform multi-scale spatial feature fusion on the target frame features and the second auxiliary frame features. The fused spatiotemporal features are input into the semantic residual correction module to obtain corrected spatiotemporal features; the semantic residual correction module is used to semantically enhance the fused spatiotemporal features. The corrected spatiotemporal features are input into the prediction convolutional layer to obtain the human pose estimation result of the target image frame.

2. The method according to claim 1, characterized in that, The step of inputting the target frame features and the first auxiliary frame features into the motion alignment module to obtain the second auxiliary frame features includes: Calculate the absolute difference between the target frame features and the first auxiliary frame features to obtain a difference feature map; The target frame features, the first auxiliary frame features, and the differential feature map are input into the motion estimation network to obtain the spatial offset and modulation mask; The modulation mask is normalized to obtain a normalized modulation mask; Based on the spatial offset, deformable convolution is used to resample the regular sampling grid of the first auxiliary frame features to obtain deformed auxiliary frame features. The deformed auxiliary frame features are processed based on the normalized modulation mask to obtain the second auxiliary frame features.

3. The method according to claim 2, characterized in that, Before inputting the target frame features, the first auxiliary frame features, and the differential feature map into the motion estimation network to obtain the spatial offset and modulation mask, the method further includes: The parameters of the terminal convolutional layer in the motion estimation network used to output the spatial offset and the modulation mask are initialized to zero so that the deformation resampling belongs to the standard regular convolution at the beginning of training.

4. The method according to claim 1, characterized in that, The step of inputting the target frame features and the second auxiliary frame features into the adaptive spatiotemporal fusion module to obtain fused spatiotemporal features includes: Global average pooling and global max pooling are performed in parallel on the target frame features and the second auxiliary frame features to obtain merged pooled features; The merged pooling features are input into a multilayer perceptron to obtain a time-channel weight matrix; Based on the time-channel weight matrix, the target frame features and the second auxiliary frame features are aggregated element-wise to obtain time-series noise reduction features. Perform three-dimensional convolution aggregation on the temporal denoising features to obtain two-dimensional spatial features; Based on the two-dimensional spatial features, the fused spatiotemporal features are determined.

5. The method according to claim 4, characterized in that, The determination of the fused spatiotemporal features based on the two-dimensional spatial features includes: The two-dimensional spatial features are synchronously input into multiple parallel depthwise separable convolutional branches to obtain multi-scale spatial features; each depthwise separable convolutional branch has a different convolutional kernel size. Based on the spatial gating weight matrix corresponding to each of the depthwise separable convolutional branches, the multi-scale spatial features are weighted and summed to obtain the fused spatiotemporal features.

6. The method according to claim 1, characterized in that, The step of inputting the fused spatiotemporal features into the semantic residual correction module to obtain corrected spatiotemporal features includes: Global spatial pooling is performed on the fused spatiotemporal features to obtain the target pooling result; The target pooling result is input into the fully connected layer to obtain the channel recalibration weights; Based on the channel recalibration weights, the fused spatiotemporal features are weighted channel by channel to obtain enhanced semantic features; The enhanced semantic features are input into a convolutional layer to extract semantic residual features; The semantic residual features are superimposed on the fused spatiotemporal features to obtain the corrected spatiotemporal features.

7. The method according to claim 1, characterized in that, The step of inputting the corrected spatiotemporal features into the prediction convolutional layer to obtain the human pose estimation result of the target image frame includes: The corrected spatiotemporal features are input into a single-layer convolutional layer for channel dimensionality reduction to obtain a spatial heatmap; wherein, the number of channels after dimensionality reduction is equal to the number of human joints; The peak coordinates of each channel are extracted from the spatial heat map and used as the predicted coordinates of each joint point. Based on the predicted coordinates of each of the joints, the human pose estimation result of the target image frame is determined.

8. The method according to claim 1, characterized in that, The training phase of the human pose estimation model includes: During the first N training cycles, the adaptive spatiotemporal fusion module and the semantic residual correction module are frozen, and the network parameters of the backbone convolutional network and the motion alignment module are updated using the initial learning rate. In the next M training cycles, the adaptive spatiotemporal fusion module is unfrozen, and the network parameters of the backbone convolutional network, the motion alignment module, and the adaptive spatiotemporal fusion module are updated using the first learning rate; During the final L training cycles, the semantic residual correction module is unfrozen, and the network parameters of the backbone convolutional network, the motion alignment module, the adaptive spatiotemporal fusion module, and the semantic residual correction module are updated using a second learning rate; the second learning rate is less than the first learning rate, and the first learning rate is less than the initial learning rate.

9. The method according to claim 1, characterized in that, The method further includes: Obtain an initial training sample set, wherein each training sample in the initial training sample set contains video frame images and their real-world annotations of key points; Calculate the image quality evaluation index for each training sample, wherein the image quality evaluation index includes at least one of image blur, occlusion ratio, and human joint visibility; Training samples that meet the preset image quality conditions for the image quality evaluation index are selected from the initial training sample set to obtain the selected training sample set; The human pose estimation model is trained based on the selected training sample set.

10. The method according to claim 9, characterized in that, The step of training the human pose estimation model based on the selected training sample set includes: Calculate the occlusion ratio and motion blur degree of each training sample in the filtered training sample set; Based on the occlusion ratio and motion blur degree of each training sample, a sample weight is assigned to each training sample; wherein, the training sample with a higher occlusion ratio and a higher motion blur degree is assigned a higher sample weight. The weighted loss value is determined based on the sample weights of each training sample; Based on the weighted loss value, the network parameters of the human pose estimation model are updated.

Images