Real-time fitness posture recognition method based on local and global feature fusion

Abstract

The invention discloses a real-time fitness posture recognition method based on local and global feature fusion. The method comprises the following steps: firstly, extracting preliminary features of an input image through a CSPNeXt backbone network; afterwards, by utilizing an improved AFT-RPB reasoning module, by introducing relative position bias and fusing a Sigmoid weighting mechanism, the modeling capability of the model for a complex dependency relationship between key points is enhanced, and local details and global context information are effectively balanced; further, probability graph model optimization is carried out on the spatial relation between the key points through an NCRFs reasoning module, and the consistency of attitude estimation is improved; meanwhile, a lightweight depth block formed by depth separable convolution is adopted to replace traditional large kernel convolution, so that the complexity and parameter quantity of the model are reduced; finally, key point coordinate prediction is converted into a classification task based on a SimCC algorithm, quantization errors are reduced, and high-precision and real-time fitness posture recognition is achieved. The reasoning speed and the lightweight level of the model are remarkably improved, and the method is suitable for real-time application scenes with limited resources.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology. Specifically, it relates to a real-time fitness posture recognition method based on the fusion of local and global features, which is particularly suitable for real-time and accurate human key point detection and posture evaluation of fitness movements on resource-constrained devices. Background Technology

[0002] Human pose estimation is one of the core tasks in computer vision, aiming to accurately locate the spatial positions of key joints in the human body from images or videos. In the fitness and health field, this technology can monitor and evaluate the correctness of a user's movements in real time, which is of great significance for improving exercise effectiveness and preventing sports injuries.

[0003] Traditional pose estimation methods primarily rely on hand-designed features or color-image-based convolutional neural networks. These methods often struggle to balance accuracy and speed in complex scenes, under occlusion, or with specific fitness movements. On one hand, some methods focus on capturing local details such as joints through high-resolution networks or complex modules, but neglect the global structural information of human pose, leading to predictions that violate anatomical constraints under limb occlusion or atypical poses. On the other hand, while some methods can integrate global context, they lack sufficient perception of subtle features such as joints and fingertips, and are insensitive to local changes. Furthermore, many high-precision models have high computational complexity and a large number of parameters, making real-time inference difficult on mobile or embedded devices, limiting their application in scenarios such as online fitness guidance.

[0004] In recent years, the Transformer architecture and its variants have been introduced into the field of pose estimation. Their powerful global modeling capabilities have brought performance improvements, but the computational complexity of traditional self-attention mechanisms increases quadratically with sequence length, making it difficult to meet real-time requirements. Meanwhile, effectively balancing local feature details and global structural information within a lightweight network framework while maintaining low computational resource consumption remains a key challenge for current pose estimation techniques in practical fitness applications.

[0005] Existing technologies mainly suffer from the following three problems: First, at the feature extraction level, traditional methods struggle to effectively balance local details and global structural information. They either overemphasize local features such as joint edges while neglecting the overall posture's coherence, leading to ergonomically incorrect predictions in complex fitness movements; or they overemphasize global context while losing crucial subtle features for judging the correctness of fitness movements, such as finger orientation and joint angles. Second, at the model architecture level, while existing Transformer-based solutions improve accuracy, their secondary complexity attention mechanism incurs significant computational overhead. Furthermore, traditional convolutional networks struggle to establish long-range dependencies, making it impossible for models to achieve an ideal balance between accuracy and speed, thus failing to meet the real-time requirements of fitness guidance scenarios. Finally, at the output optimization level, conventional methods lack explicit modeling of the human skeletal topology, resulting in insufficient robustness to complex situations such as occlusion and truncation, and failing to guarantee the spatial rationality between predicted keypoints. Summary of the Invention

[0006] Technical Problem: To address the shortcomings of existing technologies, this invention provides a real-time fitness posture recognition method based on the fusion of local and global features. This invention aims to solve the technical challenges of imbalance between local and global features, the contradiction between computational efficiency and recognition accuracy, and insufficient rationality of posture structure through innovative network structure design.

[0007] Technical Solution: To achieve the above objectives, a real-time fitness posture recognition method based on the fusion of local and global features includes the following steps:

[0008] Step 1: Input the fitness exercise image and perform data preprocessing. The preprocessing involves scaling the image pixel values ​​to a preset range [0, 1], and then uniformly adjusting the image resolution. ;

[0009] Step 2: Input the preprocessed image into the CSPNEXt backbone network. Its core feature is the use of a specific channel segmentation ratio to divide the input feature map X into high-frequency sub-images. In the low-frequency subgraph This structure, based on open-source architectures such as RTMDet, is optimized and adapted to the feature distribution of joint edges (high frequency) and the whole torso (low frequency) in fitness movements. First, the input feature map is divided into two parts along the channel dimension. Then, features are extracted from the first part using a depthwise separable convolutional layer with large kernels, and features are extracted from the second part using dilated convolutional layers and global average pooling layers in sequence. Finally, the two feature parts are concatenated along the channel dimension and processed through a... Convolutional layers fuse the data, outputting a preliminary feature map;

[0010] Step 3: Input the preliminary feature map obtained in Step 2 into the AFT-RPB (Attention Free Transformer with Relative Position Bias) inference module. This is an improvement on the original attention-free transformer (AFT) by introducing a trainable relative position bias matrix. Its core lies in using element-wise multiplication instead of traditional Softmax attention calculation, thereby enhancing spatial awareness while maintaining linear complexity. First, the input features are mapped to a query vector Q, a key vector K, and a value vector V through a linear transformation layer. Then, a trainable relative position bias matrix is ​​added to K, and the result is input into the Softmax function (the Softmax function is a well-known normalized exponential function used to map the components of the input vector to the (0, 1) interval with a sum of 1, thus generating a probability distribution or attention weights) to calculate the attention weights. These weights are then weighted and summed with V to obtain the attention features. Finally, the Sigmoid function (the Sigmoid function is a well-known sigmoid activation function, whose mathematical expression is...) is used. The query vector Q is transformed to generate the gate weights (which are used to map the query vector to a gate weight with a value range between [0, 1]). The gate weights are then multiplied by the aforementioned attention features for calibration. Finally, an enhanced feature map is output through a linear projection layer.

[0011] Step 4: Input the enhanced features into the NCRFs inference module, which is a spatial relationship optimization component built on conditional random field theory. It performs consistency correction on the initial coordinates output by explicitly modeling the anatomical constraints of keypoints. First, it calculates the single-element potential to evaluate the positional confidence of each keypoint; simultaneously, it calculates the bi-element potential to characterize the spatial constraint relationships between pairs of keypoints; then, it adds the single-element and bi-element potentials to construct an energy function; finally, by minimizing the value of this energy function, it outputs the optimized spatial distribution of keypoints.

[0012] Step 5: Input the optimized keypoint spatial distribution features from Step 4 into a depth block; perform two identical sets of operations on this depth block in sequence, each set of operations consisting of: a 3×3 depthwise convolution followed by a 1×1 pointwise convolution; finally output the transformed features.

[0013] Step 6: Input the transformed features output from Step 5 into the prediction head; First, divide the image plane evenly into multiple intervals along the horizontal and vertical axes respectively; Then, use two independent classifiers to predict the horizontal and vertical intervals to which the key points belong based on the input features respectively; Finally, combine the intervals predicted by the two classifiers to output the two-dimensional coordinates of the key points.

[0014] Step 7: Based on the two-dimensional coordinates of all key points output in Step 6, draw line segments between adjacent key points according to the preset human skeleton topology to generate a skeleton diagram; overlay the skeleton diagram onto the original input image to complete the visualization.

[0015] in,

[0016] The backbone network is a feature extraction architecture that uses a channel segmentation strategy to divide the input feature map into two independent branches in the channel dimension; the inference module is an improved attention mechanism for enhancing the spatial dependencies between key points in the feature map; the inference module is a spatial relationship optimization module for improving the rationality of human key point detection construction.

[0017] Step 2 specifically includes the following steps:

[0018] Step 2-1: Input the preprocessed image into the CSPNEXt backbone network; firstly, segment the input feature map X along the channel dimension, dividing it into two complementary sub-feature maps: sub-feature maps AND feature map Among them, sub-feature maps This is used to carry and subsequently process high-frequency components in the image, including local detail information such as human joint edges and finger contours; sub-feature map It is used to carry and subsequently process low-frequency components in the image, including macroscopic structural information such as the overall posture of the human body and the direction of the torso;

[0019] Step 2-1 specifically involves:

[0020] Step 2-1-1: High-frequency path processing: Sub-feature map The features are reorganized sequentially through fully connected layers, and then local detail features are extracted through large-kernel depthwise separable convolutions. The output is represented as follows. To capture subtle features such as joint edges and finger postures; among them, It is the output feature map of the high-frequency path branch, which carries local detail information of the human joint edge; It is a depthwise convolution, the first part of a depthwise separable convolution, used to extract local texture features in the spatial dimension. It is a fully connected layer, used here to process the input feature map. Perform linear feature recombination;

[0021] Step 2-1-2: Low-frequency path processing: Sub-feature map The process involves sequentially capturing contextual information through dilated convolutional layers, extracting global features through global average pooling layers, and finally restoring the spatial dimension through upsampling. The output is represented as follows: To obtain macroscopic structural information such as the overall posture and torso orientation of the human body; among which It is the output feature map of the low-frequency path branch, which carries macroscopic structural information such as the overall posture of the human body and the direction of the torso. It is dilated convolution, which expands the receptive field by introducing a dilation rate to capture long-range dependencies between keypoints; It is a global average pooling layer that extracts global statistical features by calculating the mean of the spatial dimension of the feature map; It is an upsampling operation used to restore the pooled low-resolution features to the same spatial size as the high-frequency branches, ensuring the alignment of subsequent features;

[0022] Step 2-1-3: Concatenate the outputs of the high-frequency and low-frequency paths along the channel dimension: ,in It is a channel splicing operation, which will and The two tensors are superimposed in the channel dimension to form a unified representation that contains both local details and global semantics.

[0023] Step 3 specifically includes the following steps:

[0024] Step 3-1: Represent the unified features As input, the query vector Q, key vector K, and value vector V are generated through three independent linear transformation layers, respectively, and their calculation methods are as follows: , , ,in , , These are the trainable weight matrices; , , These are the trainable bias terms;

[0025] Step 3-2: Construct the relative position bias matrix: For a sequence length T, from a trainable relative position bias parameter vector of length 2T-1 ( Starting from this point, perform the following operations in sequence:

[0026] Extract the valid interval: ;

[0027] Symmetrical filling: ;

[0028] Repeated expansion: ;

[0029] Matrix Reshaping: ;

[0030] Each element of the matrix The relative distance relationship between encoding positions i and j is finally generated as a symmetric matrix called the relative position bias matrix bias, where i represents the index of the current element in the query vector sequence and j represents the index of the reference element in the corresponding key vector sequence;

[0031] Step 3-3: Add the key vector K obtained in Step 3-1 to the relative position bias matrix bias obtained in Step 3-2 to obtain the key vector with fused position information; normalize the addition result along the sequence dimension using the Softmax function to generate the attention weight distribution; sum this weight distribution with the value vector V using the following formula: , where K is the key vector obtained by mapping the input features through a linear transformation layer; ⊙ represents element-wise multiplication; T is the sequence length;

[0032] Step 3-4: Input the query vector Q generated in Step 3-1 into the Sigmoid activation function to generate a gated weight matrix with a value range of [0,1]. Multiply this weight matrix element-wise with the attention result obtained in Step 3-3 to achieve adaptive feature recalibration. The calculation formula is as follows: ,in σ represents the gated feature vector obtained after adaptive recalibration via Sigmoid gate; Q represents the Sigmoid function; ⊙ represents the query vector obtained by mapping the input features through a linear transformation layer; ⊙ represents element-wise multiplication.

[0033] Step 3-5: Gated feature vectors obtained in Step 3-4 at each position The complete feature tensor By using a linear projection layer, the feature dimensions are mapped back from the hidden dimensions to the standard output dimensions, resulting in the enhanced feature representation. The calculation formula is: ,in ; ; The hidden layer feature dimensions are preset within the model; The shape of a three-dimensional feature tensor is represented by B; the batch size is B; the length of the feature sequence is T; and the dimension of the feature vector is dim. It is a trainable projection matrix; This is a trainable bias term; this operation completes the dimensional transformation and channel information fusion, and the output is used as the input feature of the subsequent pose recognition head network.

[0034] Step 4 specifically includes the following steps:

[0035] Step 4-1: Enhanced features The input is fed into the NCRFs inference module based on conditional random fields to establish the comprehensive energy function. This function is derived from the single-element potential energy. and dual-element potential energy The weighted composition is calculated using the following formula: ,in The value of represents the total energy function; the lower the value, the more the keypoint configuration conforms to the human anatomy. h and w are the height and width of the feature map, respectively. N is the size of the local window used to limit the calculation range of the bi-element potential energy. The single-element potential energy is used to assess the confidence that a single keypoint can independently appear at a certain location; The bi-element potential energy is used to model the spatial structural constraint relationships between pairs of key points;

[0036] Step 4-2: For each key point in image I Calculate its single-element potential energy This potential energy is determined by the network's prediction of the keypoint confidence score. Calculated using negative logarithms: This potential energy value represents the key point. The probability confidence level of independent occurrence; the lower the potential energy, the greater the probability that the key point is located at the current position.

[0037] Step 4-3: For each pair of key points Through learnable weight functions A comprehensive assessment of its appearance features With spatial context The degree of matching, and the coordinate distance. Multiplying them together gives the potential energy value: This potential function effectively penalizes key point location combinations that do not conform to anatomical structures through a distance constraint mechanism.

[0038] Step 4-4: Minimize the total energy function consisting of the single-element potential energy and the two-element potential energy. The mathematical expression for finding the globally optimal keypoint configuration X is as follows: ,in This means changing the variable The value of is determined such that the subsequent function expression achieves its minimum value, and the corresponding value is returned. The optimization process integrates the network's local prediction confidence with the spatial structure constraints between key points to achieve global consistency optimization of the enhanced feature map output from step 3, ultimately generating a reasonable key point sequence that conforms to human anatomy.

[0039] Step 5 specifically includes the following steps:

[0040] Step 5-1: Decode the optimized keypoint features using a lightweight deep block structure: First, use two sets of 3×3 depthwise separable convolutions to extract features channel by channel in the spatial dimension. The computational complexity is O(n log n). ,in Input the number of channels; The kernel size; and These are the feature map width and height, respectively; this operation focuses on learning more refined spatial patterns and local contextual information in the keypoint feature map while keeping the number of channels constant.

[0041] Step 5-2: After spatial feature extraction is completed using depthwise convolution, a 1×1 pointwise convolution is used to process the channel dimension, with a computational complexity of O(n). , where D is the number of output channels; this operation achieves cross-channel information fusion through linear combination, completes channel compression or expansion, and optimizes feature representation capabilities;

[0042] Step 5-3: Concatenate depthwise convolution and pointwise convolution to form a depthwise separable convolution module. Its total computational complexity is... In module instantiation, the first 1×1 pointwise convolution reduces the number of feature channels to 384, and the second 1×1 pointwise convolution further compresses it to the number of human keypoints. Matching dimensions enable efficient feature dimension reduction.

[0043] Step 6 specifically includes the following steps:

[0044] Step 6-1: The SimCC (Simple Coordinate Classification) algorithm is used to transform the keypoint coordinate prediction into a discrete classification task. The SimCC algorithm is a well-known coordinate classification algorithm in the art that achieves sub-pixel localization accuracy by discretizing continuous coordinate values ​​into horizontal and vertical dual-axis classification labels: the feature map is divided horizontally into... Vertically divided into There are several intervals, among which and Feature map size; segmentation factor Control the discretization granularity; by increasing This achieves sub-pixel-level positioning accuracy while balancing quantization error and computational complexity.

[0045] Step 6-2: Use two completely independent lightweight linear classifiers to handle the horizontal coordinate classification task and the vertical coordinate classification task respectively. The horizontal coordinate classifier receives the features output by the depth block and generates the horizontal coordinates of each keypoint. probability distribution over intervals The vertical coordinate classifier generates keypoints in the vertical direction in the same way. probability distribution over intervals ;

[0046] Step 6-3: Optimize the coordinate classifier using the KL divergence loss function, and generate softened labels based on a Gaussian distribution; use a metric function... Calculate the distance similarity between the true coordinates and the center of each category, and obtain a smooth label distribution by Softmax normalization. ,in This represents the metric function used to calculate the true coordinates. With Category Center Distance similarity between them; This represents the standard deviation parameter used to control the width of the Gaussian distribution; This represents the generated smoothed and softened label distribution; This is the index of the current interval; The total number of preset discrete intervals on the horizontal or vertical axis; This represents an exponential function with base e as the natural constant; temperature parameter. By controlling the smoothness of the distribution, this softened label provides a continuous monitoring signal, effectively reducing discretization quantization errors;

[0047] Step 6-4: Analyze the probability distributions of the two classifiers. and Perform maximum value index extraction operations separately to obtain the discrete interval indices of each keypoint in the horizontal and vertical directions. Then divide these index values ​​by the segmentation factor. Transform back to the original continuous coordinate space: , ,in This represents the maximum index extraction function, used to obtain the position index corresponding to the maximum array in the probability distribution vector; and They represent the first Interval probability distribution vectors of key points in the horizontal and vertical directions; The dividing factor; and These represent the calculated horizontal and vertical two-dimensional coordinates of each key point on the human body in the original continuous coordinate space, respectively.

[0048] Beneficial Effects: This invention achieves channel-level segmentation and parallel processing of image features through the CSPNEXt backbone network, effectively extracting and fusing local features containing joint details and global features representing the overall pose. Furthermore, it enhances the modeling capability of relative positional relationships between keypoints through the AFT-RPB module and performs structured optimization of the prediction results in conjunction with the NCRFs module. Finally, it significantly reduces computational complexity while maintaining accuracy through lightweight depth blocks and the SimCC coordinate classification strategy. This series of designs achieves end-to-end optimization from feature extraction and context enhancement to output decoding, thereby enabling high-precision and high-efficiency human pose recognition in complex fitness scenarios.

[0049] Specifically:

[0050] (1) A feature fusion mechanism based on channel segmentation is proposed. By processing high-frequency and low-frequency dual paths in parallel, it can capture the fine features of joint points and the overall posture structure of the human body at the same time, thus solving the problem that it is difficult to take into account both local details and global information.

[0051] (2) An AFT-RPB attention module that integrates relative position bias was designed, which enhances the model’s ability to perceive the spatial relationship of key points while maintaining linear computational complexity, effectively balancing computational efficiency and model performance.

[0052] (3) A lightweight depth block structure and SimCC coordinate classification method were introduced. Through depth-separable convolution and coordinate discretization strategy, the number of model parameters and computational overhead were greatly reduced, ensuring the real-time performance of the pose recognition process.

[0053] (4) An NCRFs inference module based on conditional random fields was constructed. By explicitly modeling the spatial constraint relationship between key points through energy functions, the structural rationality of attitude output and robustness under complex conditions such as occlusion were significantly improved. Attached Figure Description

[0054] Figure 1 This is a comprehensive flowchart of the fitness posture recognition method of the present invention.

[0055] Figure 2 This is a diagram of the lightweight depthwise separable convolutional group architecture in the feature extraction module of this invention.

[0056] Figure 3 This is a schematic diagram of the dual-axis coordinate classifier structure of the present invention.

[0057] Figure 4 This is a schematic diagram illustrating the recognition and visualization of key points in fitness postures according to the present invention. Detailed Implementation

[0058] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0059] like Figure 1-4 As shown, this invention is a real-time fitness posture recognition method based on the fusion of local and global features, comprising the following steps:

[0060] Step 1: Input images of fitness movements and perform data preprocessing;

[0061] Step 2: Input the preprocessed image into the CSPNEXt backbone network. The network divides the input feature map into two parts in the channel dimension through a channel segmentation mechanism. One part extracts high-frequency detail features through a depthwise separable convolutional layer containing large convolutional kernels, and the other part extracts low-frequency global features through dilated convolution and global average pooling layers. Finally, the two parts of features are concatenated and fused in the channel dimension to output a preliminary feature map rich in multi-scale semantic information.

[0062] Step 21: The preprocessed image is input into the CSPNEXt backbone network. First, precise feature map segmentation is performed in the channel dimension, dividing the input feature map X into two complementary subsets: ,in Designed to carry high-frequency components in images, such as local details like the edges of human joints and the contours of fingers; and It is used to carry low-frequency components in the image, such as macroscopic structural information like the overall posture of the human body and the orientation of the torso;

[0063] Step 211: In the high-frequency path First, features are reorganized using a fully connected layer, and then local details are extracted using a large-kernel depthwise separable convolution. This pathway is specifically designed to capture subtle features such as joint edges and finger postures.

[0064] Step 212: In the low-frequency path First, a dilated convolutional layer expands the receptive field of the convolutional kernel to capture broader contextual information and long-range dependencies between different parts of the human body. Next, a global average pooling layer compresses the feature map to extract image-level statistical features and overall semantic information. Finally, an upsampling operation restores the pooled feature map to its original spatial dimension, maintaining spatial alignment with high-frequency feature maps. This pathway focuses on macroscopic information such as the overall posture of the human body and the direction of the torso;

[0065] Step 213: The outputs of the two paths are concatenated along the channel dimension: This forms a unified representation that includes both local details and global semantics.

[0066] Step 3: Input the preliminary feature map obtained in Step 2 into the AFT-RPB inference module. This module first generates query Q, key K, and value V vectors through linear transformation. The core innovation lies in introducing a trainable relative position bias matrix. This matrix is ​​added to K and then used to calculate attention weights through Softmax. Finally, it is weighted and summed with V to effectively capture the spatial dependencies between key points with linear complexity. Subsequently, the sigmoid function is used to transform Q to generate gating weights, which refine the attention results and adaptively enhance important features. Finally, an enhanced feature map that deeply fuses local details and global context is output through linear projection.

[0067] Step 31: Represent the unified features obtained above As input, it is projected into different semantic subspaces through three independent linear transformation layers to generate query vector Q, key vector K, and value vector V, respectively: , , The weight matrix , , These are trainable parameters that map input features to different representation spaces suitable for calculating similarity, providing contextual information, and preserving the original features, respectively; bias terms. , and This enhances the model's expressive power;

[0068] Step 32: To overcome the limitations of traditional absolute position encoding in handling variable-length sequences and capturing relative positional relationships, a trainable relative position bias matrix is ​​constructed. For a given sequence length T, starting from a trainable parameter vector rel_bias of length 2T-1, a T×T symmetric matrix bias is finally obtained through truncation, symmetric padding, and repetition operations:

[0069] Each element in the matrix The relative distance between the i-th and j-th positions in the sequence is encoded, enabling the model to perceive the spatial structure between feature points;

[0070] Step 33: Add the key vector K obtained in Step 31 to the relative position bias matrix bias obtained in Step 32, so that the key vector incorporates precise positional relationships while retaining the original semantic information; then normalize the summation result along the sequence dimension using the Softmax function to generate the attention weight distribution, and multiply and sum the weights element-wise with the value vector V. This operation is implemented using element-wise multiplication, and the computational complexity remains linear.

[0071] Step 34: To further refine the output of the attention mechanism, a Sigmoid gating mechanism is introduced. The query vector Q generated in Step 31 is passed through the Sigmoid activation function to produce a gating weight matrix with values ​​in the range [0,1]. Then, this gating weight is multiplied element-wise with the attention result calculated in Step 33: This operation enables adaptive recalibration of attention output features, which can highlight important features and suppress redundant or noisy information;

[0072] Step 35: Gated feature vectors obtained in Step 3-4 at each position The complete feature tensor By using a linear projection layer, the feature dimensions are mapped back from the hidden dimensions to the standard output dimensions, resulting in the enhanced feature representation. : ,in , This projection operation uses a trainable weight matrix. and bias This not only completes the dimensional transformation of features but also further integrates information between channels. Its output will serve as the input to the subsequent pose recognition head network, providing a high-level feature representation rich in spatial context information for the final keypoint coordinate regression.

[0073] Step 4: Input the enhanced features into the NCRFs inference module, which models and optimizes the spatial relationships of keypoints based on conditional random field theory. The module evaluates the probability of each keypoint occurring independently by calculating single-element potential energy, and simultaneously characterizes the spatial constraints between keypoints using two-element potential energy, penalizing abnormal position combinations that do not conform to human anatomy. Finally, a unified energy function is constructed, and global optimization of the spatial distribution of keypoints is achieved by minimizing the energy value, effectively improving the structural rationality and prediction consistency of pose estimation, especially when handling complex actions and occluded scenes.

[0074] Step 41: Enhanced features The input is fed into the NCRFs inference module based on conditional random fields. First, an energy function that comprehensively considers local confidence and spatial consistency is established. This function consists of two parts: a single-element potential energy and a two-element potential energy. Where h and w represent the height and width of the input feature map, respectively, N is the defined local window size, and the energy function is... By weighted integration of these two potential energy terms;

[0075] Step 42: Calculate the potential energy of a single element This item is used to evaluate each key point in image I. The confidence score for the probability of an independent occurrence. Its value is directly derived from the keypoint confidence score predicted by the network. The negative logarithm: This potential energy term reflects the model's local prediction confidence in the location of a single key point. The lower the value, the greater the probability that the key point is located at the current position, thus constituting a univariate constraint term in the energy function.

[0076] Step 43: Calculate the two-element potential energy This is specifically used to model the spatial constraints between key points. It utilizes a learnable weight function. Taking into account the feature similarity of the two key points , and its spatial location , and the coordinate distance between them Multiplication: This potential term can effectively penalize positional combinations that do not conform to human anatomy, such as excessively large or small distances that are impossible between the elbow and wrist joints, thereby introducing paired constraints between key points.

[0077] Step 44: Minimize the total energy function composed of the single-element potential energy and the two-element potential energy. Find the globally optimal key point configuration X: It effectively integrates the local prediction confidence of the network with the spatial structure prior between key points, and can correct the local prediction errors or spatial inconsistencies that may exist in step 3. Finally, it outputs a human pose key point sequence that is more reasonable and accurate in terms of structure and local confidence.

[0078] Step 5: Input the optimized features from Step 4 into the depth block. The depth block consists of two sets of cascaded 3×3 depth convolutions and 1×1 point-directed convolutions. First, spatial features are extracted by depth convolution while keeping the number of channels constant. Then, information fusion and dimensionality transformation between channels are performed by point-directed convolution. This achieves efficient feature transformation while significantly reducing the number of parameters and computation.

[0079] Step 51: To efficiently decode the keypoint features after structural optimization and generate the final coordinates, the model adopts a lightweight deep block structure. This structure first uses two sets of 3×3 depthwise convolutions to extract features from the input features channel by channel in the spatial dimension. Its computational complexity is: This operation focuses on learning more refined spatial patterns and local contextual information in the keypoint feature map without changing the number of channels;

[0080] Step 52: After spatial feature extraction is completed using depthwise convolution, a 1×1 pointwise convolution is then used to process the channel dimension. The computational complexity of this operation is: Its core function is to realize cross-channel information interaction and fusion. By linearly combining the feature maps of different channels, it can compress or expand the channels, optimize the representation capability of features, and effectively integrate spatial features with channel relationships.

[0081] Step 53: By concatenating depthwise convolution and pointwise convolution, a complete depthwise separable convolution module is formed, with a total computational complexity of: Compared to standard convolution... This design significantly reduces the number of model parameters and computational burden. During the module instantiation process, the first 1×1 pointwise convolution significantly reduces the number of feature channels to 384, while the second 1×1 pointwise convolution further compresses it to a level comparable to the number of human keypoints. Matching dimensions enables efficient feature dimensionality reduction and outputs a compact feature representation suitable for final coordinate prediction at extremely low computational cost.

[0082] Step 6: Input the processed features into the prediction head and use the SimCC algorithm for keypoint localization. This algorithm innovatively transforms continuous coordinate prediction into a classification task, uniformly dividing the image into multiple fine intervals in the horizontal and vertical directions. Two lightweight classifiers predict the horizontal and vertical intervals where the keypoints are located, respectively, achieving sub-pixel level localization accuracy. This method avoids the upsampling and post-processing steps of traditional heatmap methods, significantly improving inference speed. Simultaneously, a soft label training strategy ensures localization accuracy, ultimately outputting precise two-dimensional coordinates of the keypoints.

[0083] Step 61: Transform continuous coordinate prediction into a discrete classification problem. This method uses the SimCC algorithm to transform the continuous coordinate prediction problem into a discrete classification task. Specifically, the spatial dimension of the feature map is divided horizontally into... The intervals are divided vertically into [number] sections. There are several intervals, among which , , splitting factor By controlling the discretization granularity, the quantization error and computational complexity can be balanced. Subpixel-level positioning accuracy can be achieved in a short time;

[0084] Step 62: Use two completely independent lightweight linear classifiers to handle the horizontal coordinate classification task and the vertical coordinate classification task respectively. The horizontal coordinate classifier receives the features output by the depth block and generates the horizontal coordinates of each keypoint. probability distribution over intervals The vertical coordinate classifier generates keypoints in the vertical direction in the same way. probability distribution over intervals ;

[0085] Step 63: The coordinate classifier is optimized using the KL discrete loss function, which generates soft labels based on a Gaussian distribution instead of traditional hard labels. This is achieved through a metric function. Calculate the distance similarity between the true coordinates and the center of each category, and then use Softmax normalization to obtain a smooth label distribution. Among them, temperature parameters Used to adjust the distribution shape. This softened label provides richer supervision signals for model training and effectively reduces quantization errors caused by discretization;

[0086] Step 64: Obtain the probability distributions from the outputs of the two classifiers. and The argmax is used to obtain the discrete interval indices of each keypoint in the horizontal and vertical directions, and then these index values ​​are divided by the segmentation factor. Transform back to the original continuous coordinate space: , ,in This represents the maximum index extraction function, used to obtain the position index corresponding to the maximum array in the probability distribution vector; and They represent the first Interval probability distribution vectors of key points in the horizontal and vertical directions; The dividing factor; and These represent the horizontal and vertical two-dimensional coordinates of each human keypoint in the original continuous coordinate space, respectively; the final output of human keypoint coordinates maintains sub-pixel level positioning accuracy.

[0087] Step 7: Based on the two-dimensional coordinates of all human body key points output in Step 6, connect each key point according to the preset human skeleton topology, and overlay them on the original input image to complete the final fitness posture recognition and visualization.

[0088] The goal of this invention is to fully utilize the local details and global pose information of the human body in images. Through an innovative feature fusion mechanism and lightweight network design, it achieves accurate recognition of fitness postures while ensuring real-time performance. Finally, the features, after multi-level optimization, are input into the keypoint coordinate decoding module, thereby completing the accurate estimation of human posture. This method effectively solves the problem of traditional methods struggling to balance accuracy, speed, and structural rationality, and has significant academic value and practical application significance for fields such as intelligent fitness and motion assessment.

[0089] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A real-time fitness posture recognition method based on local and global feature fusion, characterized in that, The method includes the following steps: Step 1, input the bodybuilding action image and perform data preprocessing, the preprocessing is to scale the image pixel value to a preset range [0, 1], and then uniformly adjust the image resolution to ; Step 2, input the preprocessed image into the CSPNeXt backbone network, first, divide the input feature map into two parts in the channel dimension; then, use a depth separable convolution layer containing a large convolution kernel to extract features from the first part, and use an expansion convolution layer and a global average pooling layer in turn to extract features from the second part; finally, the two obtained feature parts are spliced in the channel dimension, and a convolution layer fusion is performed to output a preliminary feature map; Step 3: Input the preliminary feature map obtained in Step 2 into the AFT-RPB inference module. First, the input features are mapped to query vector Q, key vector K and value vector V through a linear transformation layer. Then, a trainable relative position bias matrix is ​​added to K, the result is input into the Softmax function to calculate the attention weights, and then the weights are weighted and summed with V to obtain the attention features. Then, the query vector Q is transformed using the Sigmoid function to generate gating weights, which are then calibrated by multiplying with the aforementioned attention features; finally, an enhanced feature map is output through a linear projection layer. Step 4: Input the enhanced features into the NCRFs inference module. First, calculate the single-element potential to evaluate the location confidence of each key point; at the same time, calculate the two-element potential to characterize the spatial constraint relationship between each pair of key points. Then, the single-element potential energy and the two-element potential energy are added together to construct an energy function; finally, by minimizing the value of this energy function, the optimized spatial distribution of key points is output. Step 5: Input the optimized keypoint spatial distribution features from Step 4 into a depth block; perform two identical sets of operations on this depth block in sequence, each set of operations consisting of: a 3×3 depthwise convolution followed by a 1×1 pointwise convolution; finally output the transformed features. Step 6: Input the transformed features output from Step 5 into the prediction head; First, divide the image plane evenly into multiple intervals along the horizontal and vertical axes respectively; Then, use two independent classifiers to predict the horizontal and vertical intervals to which the key points belong based on the input features respectively; Finally, combine the intervals predicted by the two classifiers to output the two-dimensional coordinates of the key points. Step 7: Based on the two-dimensional coordinates of all key points output in Step 6, draw line segments between adjacent key points according to the preset human skeleton topology to generate a skeleton diagram; overlay the skeleton diagram onto the original input image to complete the visualization.

2. The real-time fitness posture recognition method based on the fusion of local and global features according to claim 1, characterized in that, The backbone network is a feature extraction architecture that uses a channel segmentation strategy to divide the input feature map into two independent branches in the channel dimension; the inference module is an improved attention mechanism for enhancing the spatial dependencies between key points in the feature map; the inference module is a spatial relationship optimization module for improving the rationality of human key point detection construction.

3. The real-time fitness posture recognition method based on the fusion of local and global features according to claim 2, characterized in that, Step 2 specifically includes the following steps: Step 2-1: Input the preprocessed image into the CSPNEXt backbone network; firstly, segment the input feature map X along the channel dimension, dividing it into two complementary sub-feature maps: sub-feature maps AND feature map Among them, sub-feature maps This is used to carry and subsequently process high-frequency components in the image, including local detail information such as human joint edges and finger contours; sub-feature map It is used to carry and subsequently process low-frequency components in the image, including macroscopic structural information such as the overall posture of the human body and the direction of the torso.

4. The real-time fitness posture recognition method based on the fusion of local and global features according to claim 3, characterized in that, Step 2-1 specifically involves: Step 2-1-1: High-frequency path processing: Sub-feature map The features are reorganized sequentially through fully connected layers, and then local detail features are extracted through large-kernel depthwise separable convolutions. The output is represented as follows. To capture subtle features such as joint edges and finger postures; among them, It is the output feature map of the high-frequency path branch, which carries local detail information of the human joint edge; It is a depthwise convolution, the first part of a depthwise separable convolution, used to extract local texture features in the spatial dimension. It is a fully connected layer, used here to process the input feature map. Perform linear feature recombination; Step 2-1-2: Low-frequency path processing: Sub-feature map The process involves sequentially capturing contextual information through dilated convolutional layers, extracting global features through global average pooling layers, and finally restoring the spatial dimension through upsampling. The output is represented as follows: To obtain macroscopic structural information such as the overall posture and torso orientation of the human body; among which It is the output feature map of the low-frequency path branch, which carries macroscopic structural information such as the overall posture of the human body and the direction of the torso. It is dilated convolution, which expands the receptive field by introducing a dilation rate to capture long-range dependencies between keypoints; It is a global average pooling layer that extracts global statistical features by calculating the mean of the spatial dimension of the feature map; It is an upsampling operation used to restore the pooled low-resolution features to the same spatial size as the high-frequency branches, ensuring the alignment of subsequent features; Step 2-1-3: Concatenate the outputs of the high-frequency and low-frequency paths along the channel dimension: ,in It is a channel splicing operation, which will and The two tensors are superimposed in the channel dimension to form a unified representation that contains both local details and global semantics.

5. The real-time fitness posture recognition method based on the fusion of local and global features according to claim 4, characterized in that, Step 3 specifically includes the following steps: Step 3-1: Represent the unified features As input, the query vector Q, key vector K, and value vector V are generated through three independent linear transformation layers, respectively, and their calculation methods are as follows: , , ,in , , These are the trainable weight matrices; , , These are the trainable bias terms; Step 3-2: Construct the relative position bias matrix: For a sequence length T, from a trainable relative position bias parameter vector of length 2T-1 ( Starting from this point, perform the following operations in sequence: Extract the valid interval: ; Symmetrical filling: ; Repeated expansion: ; Matrix Reshaping: ; Each element of the matrix The relative distance relationship between encoding positions i and j is finally generated as a symmetric matrix called the relative position bias matrix bias, where i represents the index of the current element in the query vector sequence and j represents the index of the reference element in the corresponding key vector sequence; Step 3-3: Add the key vector K obtained in Step 3-1 to the relative position bias matrix bias obtained in Step 3-2 to obtain the key vector with fused position information; normalize the addition result along the sequence dimension using the Softmax function to generate the attention weight distribution; sum this weight distribution with the value vector V using the following formula: , where K is the key vector obtained by mapping the input features through a linear transformation layer; ⊙ represents element-wise multiplication; T is the sequence length; Step 3-4: Input the query vector Q generated in Step 3-1 into the Sigmoid activation function to generate a gated weight matrix with a value range of [0,1]. Multiply this weight matrix element-wise with the attention result obtained in Step 3-3 to achieve adaptive feature recalibration. The calculation formula is as follows: ,in σ represents the gated feature vector obtained after adaptive recalibration via Sigmoid gate; Q represents the Sigmoid function; ⊙ represents the query vector obtained by mapping the input features through a linear transformation layer; ⊙ represents element-wise multiplication. Step 3-5: Gated feature vectors obtained in Step 3-4 at each position The complete feature tensor By using a linear projection layer, the feature dimensions are mapped back from the hidden dimensions to the standard output dimensions, resulting in the enhanced feature representation. The calculation formula is: ,in ; ; The hidden layer feature dimensions are preset within the model; The shape of a three-dimensional feature tensor is represented by B; the batch size is B; the length of the feature sequence is T; and the dimension of the feature vector is dim. It is a trainable projection matrix; This is a trainable bias term; this operation completes the dimensional transformation and channel information fusion, and the output is used as the input feature of the subsequent pose recognition head network.

6. The real-time fitness posture recognition method based on the fusion of local and global features according to claim 5, characterized in that, Step 4 specifically includes the following steps: Step 4-1: Enhanced features The input is fed into the NCRFs inference module based on conditional random fields to establish the comprehensive energy function. This function is derived from the single-element potential energy. and dual-element potential energy The weighted composition is calculated using the following formula: ,in The value of represents the total energy function; the lower the value, the more the keypoint configuration conforms to the human anatomy. h and w are the height and width of the feature map, respectively. N is the size of the local window used to limit the calculation range of the bi-element potential energy. The single-element potential energy is used to assess the confidence that a single keypoint can independently appear at a certain location; The bi-element potential energy is used to model the spatial structural constraint relationships between pairs of key points; Step 4-2: For each key point in image I Calculate its single-element potential energy This potential energy is determined by the network's prediction of the keypoint confidence score. Calculated using negative logarithms: This potential energy value represents the key point. The probability confidence level of independent occurrence; the lower the potential energy, the greater the probability that the key point is located at the current position. Step 4-3: For each pair of key points Through learnable weight functions A comprehensive assessment of its appearance features With spatial context The degree of matching, and the coordinate distance. Multiplying them together gives the potential energy value: This potential function effectively penalizes key point location combinations that do not conform to anatomical structures through a distance constraint mechanism. Step 4-4: Minimize the total energy function consisting of the single-element potential energy and the two-element potential energy. The mathematical expression for finding the globally optimal keypoint configuration X is as follows: ,in This means changing the variable The value of is determined such that the subsequent function expression achieves its minimum value, and the corresponding value is returned. The optimization process integrates the network's local prediction confidence with the spatial structure constraints between key points to achieve global consistency optimization of the enhanced feature map output from step 3, ultimately generating a reasonable key point sequence that conforms to human anatomy.

7. The real-time fitness posture recognition method based on the fusion of local and global features according to claim 6, characterized in that, Step 5 specifically includes the following steps: Step 5-1: Decode the optimized keypoint features using a lightweight deep block structure: First, use two sets of 3×3 depthwise separable convolutions to extract features channel by channel in the spatial dimension. The computational complexity is O(n log n). ,in For input channels; The kernel size; and These are the feature map width and height, respectively; this operation focuses on learning more refined spatial patterns and local contextual information in the keypoint feature map while keeping the number of channels constant. Step 5-2: After spatial feature extraction is completed using depthwise convolution, a 1×1 pointwise convolution is used to process the channel dimension, with a computational complexity of O(n). , where D is the number of output channels; this operation achieves cross-channel information fusion through linear combination, completes channel compression or expansion, and optimizes feature representation capabilities; Step 5-3: Concatenate depthwise convolution and pointwise convolution to form a depthwise separable convolution module. Its total computational complexity is... In module instantiation, the first 1×1 pointwise convolution reduces the number of feature channels to 384, and the second 1×1 pointwise convolution further compresses it to the number of human keypoints. Matching dimensions enable efficient feature dimension reduction.

8. The real-time fitness posture recognition method based on the fusion of local and global features according to claim 7, characterized in that, Step 6 specifically includes the following steps: Step 6-1: Using the SimCC algorithm, divide the feature map horizontally into... Vertically divided into There are several intervals, among which and Feature map size; segmentation factor Control the discretization granularity; by increasing This achieves sub-pixel-level positioning accuracy while balancing quantization error and computational complexity. Step 6-2: Use two completely independent lightweight linear classifiers to handle the horizontal coordinate classification task and the vertical coordinate classification task respectively. The horizontal coordinate classifier receives the features output by the depth block and generates the horizontal coordinates of each keypoint. probability distribution over intervals The vertical coordinate classifier generates keypoints in the vertical direction in the same way. probability distribution over intervals ; Step 6-3: Optimize the coordinate classifier using the KL divergence loss function, and generate softened labels based on a Gaussian distribution; use a metric function... Calculate the distance similarity between the true coordinates and the center of each category, and obtain a smooth label distribution by Softmax normalization. ,in This represents the metric function used to calculate the true coordinates. With Category Center Distance similarity between them; This represents the standard deviation parameter used to control the width of the Gaussian distribution; This represents the generated smoothed and softened label distribution; This is the index of the current interval; The total number of preset discrete intervals on the horizontal or vertical axis; This represents an exponential function with base e as the natural constant; temperature parameter. By controlling the smoothness of the distribution, this softened label provides a continuous monitoring signal, effectively reducing discretization quantization errors; Step 6-4: Analyze the probability distributions of the two classifiers. and Perform maximum value index extraction operations separately to obtain the discrete interval indices of each keypoint in the horizontal and vertical directions. Then divide these index values ​​by the segmentation factor. Transform back to the original continuous coordinate space: , ,in This represents the maximum index extraction function, used to obtain the position index corresponding to the maximum array in the probability distribution vector; and They represent the first Interval probability distribution vectors of key points in the horizontal and vertical directions; The dividing factor; and These represent the calculated horizontal and vertical two-dimensional coordinates of each key point on the human body in the original continuous coordinate space, respectively.

Images