Video analysis and fitness tracking method based on artificial intelligence

Abstract

The invention discloses a video analysis and fitness tracking method based on artificial intelligence, and relates to the technical field of artificial intelligence, and the method comprises the steps: S1, generating human body target positioning data in combination with a key point detection model and obtained video data; s2, generating human body target tracking data in combination with the human body target positioning data and a multi-target tracking model; s3, generating a posture recognition category based on the human body target tracking data and the human body motion posture recognition model; and S4, performing comparison with a standard posture template based on the posture recognition category to generate abnormal point information, and performing broadcasting based on the abnormal point information. According to the method, the kinematics velocity field is calculated through multi-step difference, and the visual angle robustness and generalization ability of the model are remarkably improved through the learnable visual angle compensation matrix.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based video analysis and fitness tracking method. Background Technology

[0002] With the popularization of home fitness and online sports courses, users' demand for intelligent and personalized fitness guidance tools is growing. Traditional methods that rely on wearable device sensors have limitations such as high usage threshold, limited coverage area, and inability to perceive overall posture. In contrast, video-based fitness movement recognition methods have become a hot research and industrial direction due to their non-contact nature, low cost, and high convenience.

[0003] 1. Conventional methods cannot effectively model temporal continuity when detecting missing key points, and are prone to introducing noise and disrupting the skeletal topology.

[0004] 2. Traditional methods directly use absolute coordinates for training, ignoring feature shifts caused by differences in viewpoints, resulting in a significant drop in model performance from non-training viewpoints.

[0005] 3. Existing graph convolution and other methods use a fixed limb graph structure, which cannot flexibly model the constantly changing joint relationships in the action, thus limiting the accuracy of action recognition.

[0006] 4. Most existing methods rely on static posture recognition and lack trajectory modeling of the complete motion process, making it impossible to distinguish fine-grained differences in execution quality, such as "standard squat" and "knee-valgus squat". Summary of the Invention

[0007] The purpose of this application is to provide an AI-based video analysis and fitness tracking method. This method calculates the velocity field through multi-step difference analysis, predicts missing keypoints in discontinuous frames, corrects occlusion and blurring issues, and stabilizes the skeleton. It integrates pelvic coordinate transformation and viewpoint compensation matrices, using attention to adapt to different viewpoints, thus improving robustness and generalization. Frame-level joint attention, combined with limb constraints and dynamic relationships, allows for fine-grained differentiation of similar movements (e.g., standard and inward squats). It combines trajectory contrast loss to enhance quality sensitivity and adds silhouette alignment regularization to improve generalization and suppress background interference.

[0008] To achieve the above objectives, this application provides the following solution: Firstly, this application provides an artificial intelligence-based video analysis and fitness tracking method, the system comprising: S1. Generate human target localization data by combining the key point detection model and the acquired video data; S2. Generate human target tracking data by combining human target localization data and multi-target tracking model; S3. Generate posture recognition categories based on human target tracking data and human motion posture recognition models; S4. Generate anomaly information by comparing the pose recognition category with the standard pose template, and broadcast the anomaly information.

[0009] Optionally, the pose recognition categories generated in S3 also include: S301. Construct a training dataset for human motion posture recognition; S302. A spatiotemporal consistency completion mechanism based on motion trajectory prediction is used to dynamically repair missing key points in the training dataset and generate a second training dataset by utilizing the kinematic characteristics of adjacent frames. S303. By transforming the pelvic center coordinate system and using a learnable perspective compensation mechanism, the influence of shooting perspective differences in the second training dataset is eliminated, and perspective-invariant skeletal features are extracted to generate the third training dataset. S304. Train a human motion posture recognition model based on the third training dataset; S305. Generate posture recognition categories based on human motion posture recognition models.

[0010] Optionally, generating a second training dataset also includes: S3021. Complete keypoint labeling for each frame in the training dataset. If the key point is complete, mark it. =1, then execute S3022; if the key point is complete. =0, then execute S3023; S3022. Obtain the original detection joint coordinates of multiple frames with different difference step sizes before and after the current frame, calculate the average value of the joint coordinate difference, and obtain the kinematic velocity estimation matrix. S3023. Based on the obtained neighboring valid reference frames of the current frame, predict the position of each missing joint point and generate the completed joint point coordinates. S3024. Generate a second training dataset based on the completed joint coordinates.

[0011] Optionally, generate a third training dataset: S3031. Transform the completed joint coordinates in the second training dataset to a relative coordinate system with the pelvic center as the origin, to obtain the relative skeletal coordinates, represented as: ; In the formula, For the first Relative bone coordinates after frame conversion Indicates the first The coordinates of the joints after frame completion; For the first Average coordinates of the pelvic points in the frame; S3032. Construct standard view templates for four orthogonal directions: front, back, left, and right. S3033. Generate a view compensation matrix by combining the relative skeletal features of the current frame and the standard view template; S3034. Multiply the relative bone coordinates by the view compensation matrix to obtain the projected bone features, and generate the third training dataset based on the projected bone features.

[0012] Optionally, training the human motion pose recognition model based on the third training dataset in S304 also includes: S3041, Definition of Human Movement Posture Recognition Model Structure; S3042. Initialization of the human motion posture recognition model structure; S3043. Set up the attention mechanism of the human motion posture recognition model, and generate local features of predefined limb partitions based on the attention mechanism, and generate spatiotemporal graph convolution features of the current frame by combining all local features of the current frame. S3044. Obtain the long-term time-dependent features and short-term micro-motion features of the spatiotemporal graph convolution features, and generate fused macro-micro features based on the long-term time-dependent features and short-term micro-motion features. Dynamically integrate macro-motion patterns and micro-instantaneous change information through a gating mechanism. S3045. Calculate the hierarchical joint loss based on the fusion of macro and micro features, and construct a semantic pyramid classification layer based on the hierarchical joint loss; S3046. Calculate the trajectory contrast loss of a continuous frame sequence and decouple visual similarity features based on the trajectory contrast loss; S3047. Use a differentiable bone renderer as the regularization term; S3048. The Adam optimizer is used to minimize the loss function during training to obtain the final human motion posture recognition model.

[0013] Optionally, the spatiotemporal graph convolutional features used to generate the current frame in S3043 also include: 1) Generate the original joint correlation matrix based on the projected skeleton features; 2) Apply the Sigmoid activation function to the original joint correlation matrix to perform non-linear activation mapping and generate the joint attention matrix; 3) Based on the predefined limb partitions, obtain the local features of each limb partition in the current frame; 4) Generate spatiotemporal graph convolution features for the current frame based on the features of all limb partitions and the joint attention matrix respectively.

[0014] Optionally, generating fused macro- and micro-features in S3044 also includes: 1) Use a bidirectional long short-term memory network to process the sequence composed of spatiotemporal graph convolutional features of all frames to extract long temporal dependency features; 2) First, calculate the inter-frame difference of the spatiotemporal graph convolutional feature sequence to obtain the difference sequence that highlights instantaneous changes. Then, use a temporal convolutional network to process the difference sequence and extract short-term micro-motion features. 3) Combine long-term time-dependent features and short-term micro-motion features to generate fused macro- and micro-features, and dynamically integrate macro-motion patterns and micro-instantaneous change information through a gating mechanism.

[0015] Optionally, constructing the semantic pyramid classification layer in S3045 also includes: 1) Calculate the average vector along the time dimension for the fused macro and micro features, and generate a coarse-grained classification probability vector based on the average vector; 2) The time-averaged vector that integrates macro and micro features is concatenated with the learnable semantic embedding vector of the current coarse category to generate concatenated features, and a fine-grained classification probability vector is generated based on the concatenated features. 3) Based on the logarithm of the predicted probability of the sample on the true coarse category of the current sample and the logarithm of the predicted probability on the true fine category, the hierarchical joint loss is obtained, and the semantic pyramid classification layer is constructed based on the hierarchical joint loss.

[0016] Optionally, calculating the trajectory contrast loss for consecutive frame sequences in S3046 also includes: 1) Obtain the projected skeletal features of continuous frame sequences to form motion trajectory segments, process the motion trajectory segment sequence through a bidirectional long short-term memory network, and generate the trajectory embedding vector of each motion trajectory segment. 2) Obtain the corresponding positive and negative samples based on the current action trajectory segment, and calculate the trajectory comparison loss of the current action trajectory segment based on the InfoNCE loss framework.

[0017] Optionally, using a differentiable bone renderer as a regularization term in S3047 also includes: 1) Obtain the differentiable rendering regularization loss by combining the projected bone features of the current frame with the differentiable bone renderer; 2) Combine differentiable rendering regularization loss to obtain a mask image of simulated human silhouette.

[0018] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides an AI-based video analysis and fitness tracking method that calculates the kinematic velocity field through multi-step difference calculation and predicts the location of missing key points using discontinuous reference frames, effectively repairing detection gaps caused by occlusion and blurring, and maintaining the stability of the skeleton structure.

[0019] By combining the relative coordinate transformation of the pelvic center with a learnable viewpoint compensation matrix, and utilizing an attention mechanism, the model can be dynamically adapted to different viewpoints, significantly improving its viewpoint robustness and generalization ability.

[0020] Employing a frame-level joint collaborative attention mechanism that integrates limb structural anatomical constraints and dynamic synergistic relationships, it enables fine-grained differentiation of similar movements, such as standard squats and knee valgus squats.

[0021] By combining InfoNCE trajectory contrast loss to enhance the model's sensitivity to differences in execution quality, and with pixel-level silhouette alignment rendering regularization, the model's generalization is enhanced and background interference is suppressed. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of an artificial intelligence-based video analysis and fitness tracking method is provided in this application; Figure 2 The figure shows a comparison of spatiotemporal consistency completion effects provided in an embodiment of this application. In the figure, 2a represents the coordinate error performance of spatiotemporal consistency completion, and 2b represents the coordinate error performance of zero-filling in conventional techniques. Figure 3 An experimental comparison diagram of viewpoint-invariant feature projection under different shooting angles, provided for another embodiment of this application; Figure 4 Experimental diagram illustrating the effect of joint attention mechanism on fine-grained motion classification, provided in another embodiment of this application; Figure 5 This is a comparison experiment diagram of the training convergence of dual-stream temporal fusion provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] An artificial intelligence-based video analysis and fitness tracking method, the system comprising: S1. Generate human target localization data by combining the key point detection model and the acquired video data; S2. Generate human target tracking data by combining human target localization data and multi-target tracking model; S3. Generate posture recognition categories based on human target tracking data and human motion posture recognition models; S4. Generate anomaly information by comparing the pose recognition category with the standard pose template, and broadcast the anomaly information. Furthermore, the human target recognition module uses an open-source key point detection model to achieve accurate localization of human targets. For example, based on the AlphaPose model architecture, it can extract human feature maps from input video frames through a pre-trained AlphaPose model, process candidate joint heatmaps using non-maximum suppression technology, and finally output the two-dimensional coordinate information of human skeletal key points. This enables real-time detection of human contours of single or multiple people in a single frame image, identification of 17 or 25 standard joints (such as head, shoulder, elbow, wrist, hip, knee, ankle, etc.), and generation of human skeletal topology.

[0027] It should be noted that the AlphaPose model is a deep learning-based object detection model that is widely used in the field of human pose recognition and tracking. It can achieve basic multi-person pose estimation. It first locates the human bounding boxes in the image through the object detector, then performs single-person pose estimation on each bounding box, and finally connects the detected key points to form the human skeleton.

[0028] Furthermore, the human target tracking module adopts an open-source multi-target tracking model to track human targets. For example, based on the DeepSORT model architecture, a pre-trained DeepSORT model can be used in conjunction with the key point information output by the human target recognition module to achieve cross-frame target association. Specifically, the pre-trained DeepSORT model uses Kalman filtering to predict the position and state of the target in the next frame, uses the Hungarian algorithm to match the similarity of detection boxes between adjacent frames, fuses the cosine distance of appearance features with the consistency of key point motion trajectory, and assigns a unique ID to each target, thereby maintaining the consistency of the fitness participant's identity in the continuous video sequence, eliminating the target loss problem caused by brief occlusion or camera switching, and recording the motion trajectory of the joints.

[0029] It should be noted that the DeepSORT model is a multi-object tracking algorithm based on deep learning. By combining target detection, feature extraction, Kalman filtering, and Hungarian matching, it can achieve basic human target tracking. First, it uses a target detector to obtain the target bounding box in each frame. Then, it uses a deep learning model to extract the appearance features of the target and uses a Kalman filter to predict the target's motion trajectory. At the same time, it uses the Hungarian algorithm to perform optimal matching between the detected target and the tracked trajectory based on the similarity between appearance and motion features.

[0030] Furthermore, the pose recognition categories generated in S3 also include: S301. Construct a training dataset for human motion posture recognition; S302. A spatiotemporal consistency completion mechanism based on motion trajectory prediction is used to dynamically repair missing key points in the training dataset and generate a second training dataset by utilizing the kinematic characteristics of adjacent frames. S303. By transforming the pelvic center coordinate system and using a learnable perspective compensation mechanism, the influence of shooting perspective differences in the second training dataset is eliminated, and perspective-invariant skeletal features are extracted to generate the third training dataset. S304. Train a human motion posture recognition model based on the third training dataset; S305. Generate posture recognition categories based on human motion posture recognition models.

[0031] Furthermore, the training dataset for human motion pose recognition is constructed as follows: A dataset for training the human motion posture recognition model is constructed. Optionally, the training dataset is constructed by combining publicly available fitness datasets such as the Fitness-AIA dataset and the UCF101 dataset with self-collected videos. In the annotation process, the initial joint coordinates are automatically generated using a pre-trained HRNet keypoint detector, and then the detection errors are corrected through the Label-Studio manual verification platform.

[0032] Optionally, the annotation specifications follow the COCO standard, and the annotation content includes the coordinates and visibility markers of 17 joints. At the same time, action category labels such as "squat" and "push-up" are added to each frame, as well as execution quality labels such as "standard" and "knee valgus".

[0033] Optionally, data augmentation strategies can be employed to improve data quality, encompassing spatiotemporal transformations. Spatially, random rotation, scaling, and cropping are used, while temporally, segmented sampling and frame interpolation are employed. The dataset must cover multiple viewpoints (front / back / side), different lighting conditions, and body size differences to ensure the effectiveness of subsequent spatiotemporal consistency completion and viewpoint-invariant feature projection.

[0034] It should be noted that the Fitness-AIA dataset is an open-source dataset designed specifically for fitness movement analysis, containing multi-view videos and fine-grained movement labels, such as standard squats and knee-valgus squats.

[0035] It should also be noted that the UCF101 dataset is a classic benchmark dataset for video classification, covering 13,000 real-world video clips with 101 action categories.

[0036] It should also be noted that the pre-trained HRNet is a joint detection model based on a high-resolution feature-preserving architecture. It is pre-trained on datasets such as COCO and can directly output the coordinates of key points of the human skeleton.

[0037] It should also be noted that Label-Studio is an open-source data annotation platform that supports video key point annotation, classification label verification, and multi-person collaborative annotation workflow management.

[0038] It should also be noted that the COCO standard is a general human pose estimation annotation specification that defines 17 key points (nose, eye, shoulder, elbow, wrist, hip, knee, and ankle) and their topological connections.

[0039] Furthermore, S302 also includes the following: In fitness videos, human targets are often occluded, and motion blur leads to missing key point detection, and joint movements between consecutive frames have spatiotemporal correlations. Existing technologies directly use detection tools to output coordinates and zero-fill missing points, which will destroy the skeletal topology and introduce noise interference.

[0040] This invention employs a spatiotemporal consistency completion mechanism based on motion trajectory prediction, which dynamically repairs missing key points by utilizing the kinematic characteristics of adjacent frames.

[0041] Optionally, generating a second training dataset also includes: S3021. Complete keypoint labeling for each frame in the training dataset. If the key point is complete, mark it. =1, then execute S3022; if the key point is complete. =0, then execute S3023; Specifically, 1) Key point integrity determination and neighboring frame selection The process checks the completeness of keypoints in each frame. For the current frame, if the keypoint completeness flag indicates a missing keypoint, frames with specific offsets before and after it are selected as valid reference frames based on a pre-defined set of neighboring frame offsets. This identifies the frames requiring keypoint completion and locates suitable neighboring reference frames for the missing frames, avoiding unnecessary operations on frames with complete keypoints. Specifically, the set of neighboring frame offsets is defined as follows: , No. Frame key point complete flag The total number of key points is ,and , .

[0042] In the above, Indicates the first The frame key point integrity flag is used to control whether to perform completion, avoiding unnecessary operations on the complete frame; It is a set of offsets of neighboring frames, containing two non-contiguous offsets before and after the current frame. , Avoid noise interference between adjacent frames; This refers to the total number of joints; for example, a human skeletal model typically contains 17 or 25 joints. This is the index of the current frame, i.e., the sequence number of the frame in the video, ranging from 1 to the total number of frames in the video. .

[0043] It should be noted that, It is a frame integrity flag, when When, it indicates the first All frames Each key point was completely detected, when When this occurs, it indicates that there are missing key points in the frame, and the completion mechanism needs to be triggered.

[0044] S3022. Obtain the original detection joint coordinates of multiple frames with different difference step sizes before and after the current frame, calculate the average value of the joint coordinate difference, and obtain the kinematic velocity estimation matrix. Specifically, based on the complete frame sequence of keypoints, the average of the original detected joint coordinates from multiple frames with different difference step sizes before and after the current frame is calculated to obtain the kinematic velocity estimation matrix. This matrix represents the average motion velocity and direction of the joints over a period of time, providing motion trend information for missing point prediction. Furthermore, the stability of the velocity field estimation is enhanced by averaging across multiple difference step sizes, as shown below: ; In the formula, for Kinematic velocity estimation matrix of each joint; Indicates the first The coordinates of the key points detected in the original frame; Indicates the first The coordinates of the key points detected in the original frame; The difference step size can be 1, 2, 3 or 4. The four step sizes cover the short-term motion modes. The stability of the velocity field is enhanced by averaging the four sets of differences.

[0045] It should be noted that the score It is a fixed hyperparameter setting. When the difference step size is 1, 2, 3 or 4, it is necessary to calculate the average of these 4 difference step sizes to enhance the stability of the velocity field estimation and reduce noise interference.

[0046] S3023. Based on the obtained neighboring valid reference frames of the current frame, predict the position of each missing joint point and generate the completed joint point coordinates. Specifically, for frames with missing keypoints, the position of each missing keypoint is predicted using the original detected keypoint coordinates and kinematic velocity estimation matrix of nearby valid reference frames. Specifically, each offset in the set of offsets of nearby frames is traversed, and the product of the reference frame coordinates corresponding to that offset and the motion velocity vector with the offset is calculated. The calculated results for all offsets are then averaged to generate the completed keypoint coordinates for the current missing frame. The completed keypoint coordinates can integrate the positional information of nearby frames and the overall motion trend, repairing the missing points caused by occlusion or blurring, as shown below: ; In the formula, Indicates the first The coordinates of the joints after frame completion; Indicates the first The coordinates of the key points detected in the original frame; Indicates the first The coordinates of the key points detected in the original frame; The cardinality of the set of offsets of neighboring frames, such as, ; To obtain specific values ​​for the offset, iterate through... medium elements .

[0047] It should be noted that if the set of adjacent frame offsets If it contains 4 elements, then The value must be 4. Choosing non-continuous offset values ​​such as ±2 and ±3, rather than adjacent frames, is to avoid noise interference from adjacent frames while maintaining spatiotemporal coverage.

[0048] It should also be noted that during the process of completing the key point coordinates, The project uses a kinematic velocity field estimation formula to dynamically calculate the average motion trend across multiple frames, combined with a set of non-continuous offsets from neighboring frames. It achieves kinematically driven repair of occluded missing points through predictive compensation, effectively utilizes the prior knowledge of human motion continuity, enhances the stability of the velocity field through multi-step differential averaging, and can reduce the temporal jitter noise of joint coordinates.

[0049] In one embodiment, the effectiveness of the spatiotemporal consistency completion mechanism in video occlusion scenarios is verified by visually comparing the spatiotemporal consistency completion strategy of this technology with that of conventional technology (zero-filling) in processing the coordinate error of 17 key joints in 100 frames of video using heatmaps. Figure 2 As shown, 2a represents the coordinate error performance of spatiotemporal consistency completion, and 2b represents the coordinate error performance of the conventional zero-fill technique. The vertical axis of the figure shows the human joints from head to toe, and the horizontal axis represents the video frame sequence. The occlusion area marks the occlusion interval of 40-60 frames. The thermal color depth characterizes the error magnitude, with yellow / red indicating high error and dark green indicating low error. The experimental results show that in the occlusion area, the conventional technique produces large areas of high error patches (yellow area), especially in the lower limb joints such as the knee and ankle, while this technique only has slight error fluctuations in a few joints (light green). This proves that it effectively repairs the joint point loss caused by occlusion through kinematic velocity field estimation and discontinuous frame compensation, maintaining the integrity of the skeletal topology. The experimental results verify that the dynamic modeling of the kinematic characteristics of adjacent frames by this technique can significantly reduce the completion error compared with simple zero-fill.

[0050] S3024. Generate a second training dataset based on the completed joint coordinates.

[0051] Optionally, generate a third training dataset: Furthermore, the varied shooting angles in fitness videos lead to significant differences in the absolute coordinates of bones in the same pose. Existing techniques that directly use absolute coordinates to train models require additional learning of viewpoint invariance, increasing model complexity and reducing generalization ability.

[0052] This invention eliminates the influence of shooting angle differences and extracts perspective-invariant skeletal features by transforming the pelvic center coordinate system and using a learnable perspective compensation mechanism.

[0053] S3031. Transform the joint coordinates of the completed second training dataset into a relative coordinate system with the pelvic center as the origin to obtain the relative skeletal coordinates. Specifically, the average pelvic point coordinates of frame f are subtracted from the completed joint point coordinates, and then the joint point coordinates are transformed into a relative coordinate system with the pelvic center as the origin to obtain relative skeletal coordinates. This eliminates the influence of global position changes of the human body in the video frame, and is represented as: ; In the formula, For the first Relative bone coordinates after frame conversion; is the average coordinate of the pelvic point in frame f.

[0054] It should be noted that the average coordinates of the pelvic point in frame f are... It is obtained by calculating the average coordinates of the joints in the pelvic region. For example, assuming the pelvic region includes the left hip, right hip, and central pelvic joints, the indices are respectively... , and ,in This is the first joint point in the pelvic region. This is a joint point in the second pelvic region. If it is a joint point in the third pelvic region, then , For the first Frame number The coordinates of each joint point Let be the i-th pelvic region joint. In practical cases, if the COCO keypoint model is used, then... , , .

[0055] It should also be noted that the COCO keypoint model also includes: (Common-Objects-in-Context-Keypoints-Model) is an open-source human pose estimation model that defines a standardized set of 17 human keypoints and their connections.

[0056] S3032. Construct standard view templates for four orthogonal directions: front, back, left, and right. Specifically, standard view templates for four orthogonal directions—front, back, left, and right—are pre-constructed. Each standard viewpoint template contains a key vector that characterizes the features of that viewpoint template. and the value vector representing the viewpoint compensation vector The key matrix and value matrix store template features and compensation vectors from four standard perspectives.

[0057] In the above, The key matrix stores template features from four standard viewpoints, and ; It is a value matrix that stores the compensation vectors for the four standard viewpoints, and ; Key matrix The Row vector, representing the first row vector. Template features for standard perspectives, such as template features for front, back, left, and right perspectives; Value matrix The Row vector, representing the first row vector. A compensation vector from a standard perspective; For viewpoint indexing, such as, These correspond to the four orthogonal directions: front, back, left, and right. Represents the set of real numbers; Represent a OK A real matrix of columns; This represents the view template generation function, which uses the bone vectors of four orthogonal viewpoints (front, back, left, and right) as the reference. The key-value feature dimension is a trainable parameter that characterizes the expressive power of the template features and determines the representational power of viewpoint compensation. It is adjusted through training to capture viewpoint invariance, such as... .

[0058] It should be noted that the viewpoint template generation function is implemented using predefined standard viewpoint videos. By collecting fitness movement videos from four orthogonal viewpoints (front, back, left, and right), a keypoint detector is used to extract the relative skeletal coordinates for each viewpoint. Then, the average features of each viewpoint are calculated as a template. For example, the front view template is the average value of the relative bone coordinates of all frames in the front view.

[0059] S3033. Generate a view compensation matrix by combining the relative skeletal features of the current frame and the standard view template; Specifically, using an attention mechanism, the relative skeletal features of the current frame are mapped to the query space through a learnable viewpoint compensation query matrix. Then, a similarity weight is calculated between this matrix and the key matrix storing standard viewpoint template features. Finally, this weight is used to perform a weighted summation of the value matrix storing compensation vectors to generate a viewpoint compensation matrix. This matrix integrates compensation information from multiple standard viewpoints and can dynamically adjust the compensation based on the actual viewpoint of the current frame, as shown below: ; In the formula, For viewpoint compensation matrix; The view compensation query matrix is ​​a learnable parameter that maps relative skeletal features to the query space. For the Softmax function; for The transpose of .

[0060] S3034. Multiply the relative bone coordinates by the view compensation matrix to obtain the projected bone features, and generate the third training dataset based on the projected bone features.

[0061] Specifically, the relative skeleton coordinates are multiplied by the viewpoint compensation matrix to project them into a new feature space, resulting in projected skeleton features. These projected skeleton features are invariant to changes in the shooting viewpoint and focus more on describing the pose itself, represented as: ; In the formula, The projected skeletal features are for the f-th frame; The coordinates are relative to the center of the pelvis.

[0062] It should also be noted that, based on the relative coordinate transformation of the pelvic center, the viewpoint-invariant feature projection combined with a learnable compensation mechanism constructs... This is used to characterize orthogonal viewpoint template key-value pairs, and dynamically generates projected skeleton features by calculating an attention-weighted viewpoint compensation matrix. This allows for the mapping of absolute coordinates to a viewpoint-invariant subspace, eliminating differences in shooting angles.

[0063] In one embodiment, the performance of the evaluation viewpoint-invariant feature projection module under extreme shooting angles is as follows: Figure 3 As shown, by constructing five typical fitness shooting angles (including a 90° extreme side angle), the motion recognition accuracy of conventional techniques and our proposed technique is compared. Conventional techniques use the current mainstream deep learning recognition framework based on absolute coordinates. Conventional techniques directly input the pixel coordinates of the detected joints in the image into the recognition model without any viewpoint normalization processing. Furthermore, conventional techniques use standard image-level enhancement methods, applying random rotation (±30° range) and random scaling (0.8-1.2 times) transformations to video frames during training to attempt to simulate different shooting angles. The baseline neural network architecture of conventional techniques uses an unmodified spatiotemporal graph convolutional network as the recognition backbone. This network uses a fixed human skeleton topology map, and the classification layer adopts a flat, fully connected network structure. Experimental results show that the accuracy of conventional techniques (red bars) drops sharply when the viewing angle shifts, especially at a 90° side view where the recognition ability is almost ineffective. In contrast, this technique (blue bars) forms a smooth performance degradation curve through pelvic center coordinate system transformation and a learnable viewing angle compensation mechanism. The lower green gain bars quantify this, showing that this technique brings a performance leap of nearly 40 percentage points at extreme viewing angles. The composite chart structure in the figure, through the correspondence between the upper accuracy bars and the lower gain distribution chart, proves the ability of viewing angle invariant feature projection to eliminate interference from shooting angle.

[0064] Optionally, training the human motion pose recognition model based on the third training dataset in S304 also includes: S3041, Definition of Human Movement Posture Recognition Model Structure; The human motion posture recognition model employs a multi-stage processing architecture, as detailed below: The input layer receives the completed sequence of joint coordinates. The feature extraction layer includes a view-invariant skeleton feature projection module; The core processing layer is a joint attention-driven adaptive spatiotemporal graph convolutional network, whose output features are input to a dual-stream temporal fusion module; The classification layer adopts a semantic pyramid structure, which includes a coarse-grained classifier and a fine-grained sub-classifier.

[0065] In addition, the model has two auxiliary branch modules: a trajectory comparison learning branch and a differentiable skeleton rendering branch.

[0066] S3042. Initialization of the human motion posture recognition model structure; The trainable parameters of the model are initialized using the Xavier initialization strategy to provide a starting point for iterative training of the model.

[0067] S3043. Set up the attention mechanism of the human motion posture recognition model, and generate local features of predefined limb partitions based on the attention mechanism, and generate spatiotemporal graph convolution features of the current frame by combining all local features of the current frame. Conventional spatiotemporal graph convolutional networks employ a fixed limb partitioning strategy, which cannot adapt to the dynamically changing joint collaboration patterns in fitness movements, such as the strong knee-hip joint correlation during squats, resulting in insufficient extraction of movement features.

[0068] This invention employs an attention-driven adaptive graph convolution mechanism, which enhances dynamic collaborative relationships through frame-level joint attention while preserving basic anatomical constraints.

[0069] S3044. Obtain the long-term time-dependent features and short-term micro-motion features of the spatiotemporal graph convolution features, and generate fused macro-micro features based on the long-term time-dependent features and short-term micro-motion features. Dynamically integrate macro-motion patterns and micro-instantaneous change information through a gating mechanism. Classifying fitness movements requires simultaneously modeling joint motion trajectories and local micro-motion features. Conventional single-stream neural networks struggle to handle both aspects. A single architecture either over-smooths short-term dynamics (e.g., LSTM, which ignores high-frequency micro-motions) or fails to capture long-term dependencies (e.g., TCN, which under-models long sequences), resulting in limited classification performance, especially for movements sensitive to differences in execution quality, such as standard squats versus knee-valgus squats.

[0070] This invention constructs a dual-path temporal processing mechanism to extract long-term temporal dependency features and short-term micro-motion features respectively, and uses a gating fusion strategy to dynamically integrate macro and micro information in order to comprehensively capture the dynamic characteristics of actions.

[0071] S3045. Calculate the hierarchical joint loss based on the fusion of macro and micro features, and construct a semantic pyramid classification layer based on the hierarchical joint loss;

[0072] Fitness movement categories have a hierarchical semantic structure, such as "squat" - "weighted squat" - "lunge squat". Conventional flat classification layers ignore the subordinate relationship between categories, leading to fine-grained movements such as standard squat and knee-in-valgus squat being easily misclassified due to visual similarity. Furthermore, subclasses like weighted squat with few training samples are difficult to obtain sufficient supervision, and the model cannot use coarse-class semantics to constrain the decision boundaries of fine-class movements.

[0073] This invention constructs a hierarchical classifier that guides fine-grained decision-making through coarse-grained classification and designs a joint loss to explicitly model the category tree structure.

[0074] S3046. Calculate the trajectory contrast loss of a continuous frame sequence and decouple visual similarity features based on the trajectory contrast loss; In fitness videos, it is difficult to distinguish visually similar but semantically different movements, such as a standard squat and a knee-valgus squat. Current technologies rely on single-frame posture classification, ignoring the dynamic characteristics of the movement execution process and are not sensitive to differences in execution quality, such as joint trajectory deviation. They cannot decouple visually similar but different movement features.

[0075] This invention employs trajectory contrast loss, which constructs positive and negative samples to force the model to focus on the subtle differences in the action execution trajectory, thereby decoupling visual similarity features.

[0076] S3047. Use a differentiable bone renderer as the regularization term; S3048. The Adam optimizer is used to minimize the loss function during training to obtain the final human motion posture recognition model.

[0077] Optionally, the spatiotemporal graph convolutional features used to generate the current frame in S3043 also include: 1) Generate the original joint correlation matrix based on the projected skeleton features; Specifically, the projected skeleton features are multiplied by the learnable joint association query weight matrix and the joint association key weight matrix respectively to obtain query features and key features. Then, the dot product of the query features and the transpose of the key features is calculated and scaled to generate the original joint association degree matrix, which represents the strength of the dynamic cooperative relationship between each joint in the current frame, as shown below: ; In the formula, Indicates the first Original joint correlation matrix of the frame; The joint association query weight matrix is ​​a learnable parameter; The joint association key weight matrix is ​​a learnable parameter. As the input feature dimension, based on the projected skeleton features Dimension settings, such as, ; express Transpose of the item.

[0078] 2) Apply the Sigmoid activation function to the original joint correlation matrix to perform non-linear activation mapping and generate the joint attention matrix; Applying the Sigmoid activation function to the original joint correlation matrix performs a non-linear activation mapping, mapping each element in the original joint correlation matrix to the interval [0,1], generating a joint attention matrix. Specifically, for the... The original joint correlation matrix of the frame is used to perform non-linear activation mapping using the Sigmoid activation function to obtain the first frame. Frame joint attention matrix, defined Indicates the first Frame joint attention matrix.

[0079] It should be noted that, The term maps the original joint correlation matrix to In the interval, each value in the joint attention matrix represents the dynamic collaboration strength weight between two corresponding joints in the current frame; that is, for the original joint correlation matrix... Each element is nonlinearly mapped to obtain the joint attention matrix. The i-th row of the... Column elements are , representing the relationship between the i-th node and the i-th node The dynamic cooperation strength of each node in the current frame.

[0080] 3) Obtaining the local features of each limb partition in the current frame based on predefined limb partitions also includes; Based on predefined limb partitions, corresponding limb adjacency masks are applied to the projected skeletal features. Then, the masked features are linearly transformed with the learnable convolutional kernel weights of the corresponding partitions to extract the local features of each limb partition in the current frame, preserving the basic anatomical constraints of the human body, represented as: ; In the formula, This indicates that the v-th limb partition is in the... Frame characteristics; Let v be the adjacency mask for the v-th limb, which is a diagonal matrix, with 1 corresponding to the joint position inside the limb; This represents the Hadamard product, which is an element-wise multiplication operation. The convolutional kernel weights, which are specific to the v-th partition, are learnable parameters.

[0081] It should be noted that the limb adjacency mask Based on predefined limb zones, the joints are grouped, for example... Divisions: left upper limb, right upper limb, trunk, left lower limb, right lower limb, for the first The partition, when the first The first joint belongs to the first When there are multiple partitions, diagonal elements Otherwise, it is 0. for The element in the i-th row and i-th column.

[0082] 4) Generate spatiotemporal graph convolution features for the current frame based on the features of all limb partitions and the joint attention matrix respectively.

[0083] The features of each limb segment are multiplied by the joint attention matrix, and then the results of the multiplications of all limb segments are summed to generate the spatiotemporal graph convolutional features of the current frame. The spatiotemporal graph convolutional features can both fuse local limb features based on anatomical structure and enhance the dynamic inter-joint cooperation relationships in the current frame through the joint attention matrix, as shown below: ; In the formula, Indicates the first Spatiotemporal graph convolutional features of frames; To predefine the total number of limb divisions, such as left upper limb, right lower limb, etc., for example, .

[0084] It should be noted that in joint attention-driven adaptive spatiotemporal graph convolution, the calculation method of spatiotemporal graph convolution features combines a predefined limb adjacency mask and a dynamic joint attention matrix. Instead of conventional fixed graph convolution kernels, it can utilize joint attention matrices. Strengthen the collaborative weights between key points in the current frame while preserving features. Based on fundamental anatomical constraints, such as partitioning the upper and lower limbs, adaptively capturing action-specific correlations, and enhancing spatiotemporal graph convolutional features. The model's discriminative power effectively distinguishes subtle differences in joint movement patterns between standard squats and knee-valgus squats, enhancing its generalization performance for diverse fitness movements.

[0085] In one embodiment, the performance improvement of joint attention mechanisms on fine-grained motion classification is analyzed. The performance of this technique (joint attention map convolution) and the conventional technique (fixed map convolution) across eight fitness motion categories is compared using parallel coordinate graphs. Figure 4 The four vertical axes represent precision, recall, F1 score, and area under the AUC curve, respectively. Each movement is connected by a separate colored line. The experimental results show that the performance of this technology is higher than that of conventional technology for all movements, especially for easily confused movements such as squats and lunges. This proves that this technology strengthens the dynamic collaboration relationship through the joint attention matrix while retaining basic anatomical constraints. It can keenly capture subtle differences between similar movements, and its adaptive spatiotemporal modeling capability enables the model to accurately distinguish between visually similar movements such as standard squats and knee-valgus squats.

[0086] Optionally, generating fused macro- and micro-features in S3044 also includes: 1) Use a bidirectional long short-term memory network to process the sequence composed of spatiotemporal graph convolutional features of all frames to extract long temporal dependency features; Specifically, a bidirectional long short-term memory network is used to process the sequence composed of spatiotemporal graph convolutional features of all frames, extracting long-term temporal dependency features to capture the macroscopic motion trajectory and long-term temporal dependencies of fitness movements, represented as: ; In the formula, This represents the long temporal dependency features extracted by the bidirectional long short-term memory network; This represents the total number of frames in the video. This refers to the hidden feature dimension of a bidirectional long short-term memory network, i.e., the feature size of the hidden state, such as... ; This represents a bidirectional long short-term memory network that captures temporal context dependencies through forward and backward LSTM units; This represents the input feature sequence of a bidirectional long short-term memory network. Indicates the first The spatiotemporal graph convolutional features of the frame, and, This represents the spatiotemporal graph convolutional features of the first frame. This represents the spatiotemporal graph convolutional features of the F-th frame.

[0087] 2) First, calculate the inter-frame difference of the spatiotemporal graph convolutional feature sequence to obtain the difference sequence that highlights instantaneous changes. Then, use a temporal convolutional network to process the difference sequence and extract short-term micro-motion features. Specifically, firstly, the inter-frame differences of the spatiotemporal graph convolutional feature sequence are calculated to obtain a difference sequence that highlights instantaneous changes. Then, a temporal convolutional network is used to process this difference sequence to extract short-term micro-motion features, thereby enhancing the sensitivity to subtle joint jitters or instantaneous changes, as shown below: ; In the formula, This represents the short-term micro-motion features extracted by a temporal convolutional network; This represents a temporal convolutional network that captures local temporal patterns through convolutional kernels. This represents the difference operation, used to calculate the inter-frame difference of the input sequence. Let the input sequence for the difference operation be... The calculation method is expressed as ; The input sequence for the difference operation; Indicates from The subsequence from frame 2 to frame F; Indicates from The subsequence from frame 1 to frame F-1.

[0088] 3) Combine long-term time-dependent features and short-term micro-motion features to generate fused macro- and micro-features, and dynamically integrate macro-motion patterns and micro-instantaneous change information through a gating mechanism.

[0089] Specifically, long-term time-dependent features and short-term micro-motion features are multiplied by their corresponding learnable fusion weight matrices and then summed, followed by a nonlinear transformation using the hyperbolic tangent function. Simultaneously, the two feature sets are concatenated and multiplied by a learnable gating weight matrix, and gating weights are generated using the sigmoid function. Finally, the result of the nonlinear transformation is multiplied element-wise with the gating weights to generate fused macro- and micro-features. This gating mechanism dynamically integrates macro-motion patterns and micro-instantaneous change information, as shown below: ; In the formula, This indicates the integration of macroscopic and microscopic features; Represents the hyperbolic tangent activation function; This represents the Sigmoid activation function; is the fusion weight matrix of macro features, and is a learnable parameter; is the fusion weight matrix for the micro-features, and is a learnable parameter; is the gated weight matrix, which consists of learnable parameters; Represents the pair along the feature dimension and Then, the parts are assembled.

[0090] It should be noted that, in order to effectively realize the long-term dependence and short-term micro-movement of fitness movements, collaborative modeling is required. In the fusion of dual-stream temporal features, macro and micro features are integrated. The computational method employs a gating mechanism to integrate the macroscopic trajectory features of a bidirectional long short-term memory network and the microscopic difference features of a temporal convolutional network, rather than a conventional single-stream network. It dynamically weights and fuses macro- and microscopic information, thereby avoiding the limitations of a single architecture and comprehensively capturing the dynamic characteristics of actions, such as the long-term trajectory during the squatting phase and the instantaneous knee micro-movement during the rising phase. This enhances the sensitivity to differences in execution quality and optimizes feature representation through the gating mechanism, thereby improving the accuracy of the classification model on fine-grained actions.

[0091] In one embodiment, the training efficiency and final performance of the dual-stream temporal fusion mechanism are verified. The performance of the proposed dual-stream temporal fusion strategy versus single-stream methods (Long Short-Term Memory network, Temporal Convolutional Network) over 100 training epochs is compared using convergence curves with confidence intervals. The curves represent the average accuracy, and the shaded areas reflect the fluctuation range across five experiments. The experimental results are as follows: Figure 5As shown, this technique leads from the early stages of training, achieving the performance of conventional methods after about 50 cycles in about 20 cycles, and ultimately achieving an accuracy that is about 5 percentage points higher. This proves that this technique integrates the macroscopic motion trajectory extracted by the bidirectional long short-term memory network and the microscopic instantaneous changes extracted by the temporal convolutional network through a gating fusion strategy, overcoming the inherent defects of the single-stream architecture. The dual-path design can simultaneously model the long-term trajectory of the squatting phase and the knee micro-movements of the rising phase, giving the model higher discriminative power in motion quality assessment.

[0092] Optionally, constructing the semantic pyramid classification layer in S3045 also includes: 1) Calculate the average vector along the time dimension for the fused macro and micro features, and generate a coarse-grained classification probability vector based on the average vector; Specifically, the average vector of the fused macro and micro features is calculated along the time dimension. This average vector is then linearly transformed using a learnable coarse-grained weight matrix. Finally, the coarse-grained classification probability vector is calculated using the Softmax function, representing the probability that the input video belongs to each coarse-grained action category, expressed as:

[0093] In the formula, This represents a coarse-grained classification probability vector with a dimension of 1×C; This represents the total number of coarse-grained movement categories, used for hierarchical classification, such as the number of parent categories for fitness movements, including "squat", "push-up", "lunge", etc. This is a coarse-grained weight matrix, which is a trainable parameter that enables a linear mapping from the feature space to coarse categories. To integrate macro and micro features The average vector along the time dimension, i.e. ; For the first The fusion of macro and micro features of the frame, i.e. the first The fusion of macroscopic and microscopic features of time steps.

[0094] 2) The time-averaged vector that integrates macro and micro features is concatenated with the learnable semantic embedding vector of the current coarse category to generate concatenated features, and a fine-grained classification probability vector is generated based on the concatenated features. The time-averaged vector fusing macro and micro features is concatenated with the learnable semantic embedding vector of the coarse category retrieved from the sample's true coarse category index. Then, the concatenated features are linearly transformed with the corresponding coarse category's dedicated learnable weight matrix. Finally, the fine-grained classification probability vector is calculated using the Softmax function. Guided by the coarse category semantics, a fine-grained classification probability vector is generated, representing the probability that the input video belongs to each fine-grained action category under the coarse category, expressed as: ; In the formula, This represents a fine-grained classification probability vector; For belonging to the first The number of subcategories within each coarse category; Coarse category index; For the first The weight matrix specific to each coarse class is a trainable parameter; Indicates to and Concatenate along the feature dimension; For the first The semantic embedding vectors of each coarse category are used to learn the category semantic representation through a pre-defined trainable embedding layer, mapping the discrete coarse category IDs to continuous vectors. For example, for the pre-defined trainable embedding layer, the input is the coarse category ID, and the output is the semantic embedding vector. The dimension can be set to 32.

[0095] Furthermore, fine-grained classification probability vectors This represents the probability distribution of a video belonging to various fine-grained action categories under the predicted coarse category, such as different posture categories like "standard squat" and "knee-valgus squat"; In one embodiment, the classification hierarchy is assumed to be: a) Broad category: Squat ( ),push-up( ); b) Subcategories (Squats): Standard Squat ( ), knee-in squat ( ).

[0096] The model output is: Coarse-grained probability: [0.92, 0.08] (selection) (Squat); Fine-grained probability (in) Below): [0.35, 0.65] Select (Knee-valgus squat).

[0097] 3) Based on the logarithm of the predicted probability of the sample on the true coarse category of the current sample and the logarithm of the predicted probability on the true fine category, the hierarchical joint loss is obtained, and the semantic pyramid classification layer is constructed based on the hierarchical joint loss.

[0098] Calculate the logarithm of the predicted probability of a sample in its true coarse category and the logarithm of the predicted probability in its true fine category. Then, use the hierarchical supervision strength coefficient to perform a weighted sum of these two negative logarithms. Finally, sum the weighted sums of all samples to obtain the joint loss. This process supervises the model to correctly predict both coarse and fine categories and forces the model to satisfy the hierarchical semantic constraints of the action categories, as expressed below: ; In the formula, Indicates joint loss; Indicates the number of samples entered in the batch; Indicates the first Each sample in its true coarse category The predicted probability; For the first The true coarse category label of each sample; Indicates the first The predicted probability of a sample in its true subclass; For the first The true subcategories of each sample; The hierarchical supervision strength coefficient balances the coarse / fine granular loss weights, such as... ; It is a logarithmic function, with the default base being the natural constant.

[0099] It should be noted that in the semantic pyramid classification layer, the joint loss... The model constructs a hierarchical structure of action categories, such as the coarse class "squat" and the fine class "knee-valgus squat," instead of using a conventional flat classification loss, and employs a hierarchical supervision strength coefficient. By balancing the supervision intensity of coarse-grained and fine-grained probabilities, the model is forced to use coarse-class semantics to constrain the decision boundaries of fine-classes, thereby improving the discriminative power of fine-grained actions. Furthermore, by enhancing the model's generalization ability through hierarchical guidance, more accurate action quality assessment can be achieved.

[0100] Optionally, calculating the trajectory contrast loss for consecutive frame sequences in S3046 also includes: 1) Obtain the projected skeletal features of continuous frame sequences to form motion trajectory segments, process the motion trajectory segment sequence through a bidirectional long short-term memory network, and generate the trajectory embedding vector of each motion trajectory segment. Specifically, projected skeletal features of consecutive frame sequences are extracted from the video to form motion trajectory segments. These segments are then processed using a bidirectional long short-term memory network to generate a trajectory embedding vector for each motion trajectory segment, representing the overall dynamic characteristics of the motion execution process within that segment, as shown below: ; In the formula, Indicates the first Trajectory embedding vectors of each sample; This is for bidirectional long short-term memory network processing operations; A motion trajectory segment, composed of projected skeletal features from consecutive frames, is defined as follows: ; For the projected skeletal features of the f-th frame, then For the first Frame projection skeletal features For the first Frame projection skeletal features; This is the starting frame index for the trajectory segment. This is the index of the end frame of the trajectory segment, and .

[0101] It should be noted that, and The length of a segment can be determined by sliding window or random sampling, taking a fixed-length segment. The segment length is set based on the video length and motion cycle, such as 16 or 32 frames.

[0102] 2) Within each training batch, for the motion trajectory segment of the anchor sample, select trajectory segments of the same action category but different execution quality as positive samples, and select several visually similar trajectory segments of different action categories as negative samples. Then, calculate the cosine similarity between the anchor sample trajectory embedding vector and the positive sample trajectory embedding vector, as well as the cosine similarity between the anchor sample trajectory embedding vector and each negative sample trajectory embedding vector. Finally, based on the InfoNCE loss framework, maximize the similarity between the anchor sample and the positive sample, while minimizing the similarity between the anchor sample and each negative sample, and calculate the trajectory contrast loss. This forces the model to focus on subtle differences in the action execution trajectory, decouples visually similar but different action features, and improves the model's ability to distinguish execution quality. This is expressed as: ; In the formula, Indicates trajectory comparison loss; The cosine similarity function is used to calculate the directional similarity between two vectors. Embed the trajectory of the anchor sample; Embedded for positive samples; Embedding for negative samples; To control the coefficient of difficulty in distinguishing samples, such as, ; This represents the number of negative samples in the batch. Index for negative samples .

[0103] It should be noted that when calculating the trajectory comparison loss, it is assumed that the first... The nth sample is the anchor sample, and the nth sample is the anchor sample. Each sample is a positive sample. Positive samples belong to the same category as anchor samples but have different execution quality. Trajectory comparison loss. It is a batch-level loss, which needs to be calculated and averaged for all samples within the batch.

[0104] It should also be noted that InfoNCE (Info-Noise-Contrastive-Estimation) loss is a commonly used loss function in contrastive learning. Its core idea is to bring similar samples closer together and widen the distance between dissimilar samples. The design of trajectory contrastive loss is inspired by the InfoNCE loss framework. It treats the embedding vector of a motion trajectory segment as an anchor sample, the embedding vector of another trajectory segment of the same motion category but with different execution quality as a positive sample, such as a standard squat versus a knee-valgus squat, and the embedding vector of a trajectory segment of different motion categories but visually similar as a negative sample, such as a squat versus a lunge. Through this contrastive learning, the model is forced to pay attention to subtle but crucial differences in the motion execution trajectory, such as whether the knee is valgus or not, thereby better distinguishing visually similar but different types of motions, decoupling visual similarity features, and improving sensitivity to execution quality.

[0105] It should also be noted that when calculating the trajectory comparison loss, the action trajectory segment is defined. For anchor samples, The sample was positive. Negative samples are those that are in the same action category as the anchor sample but have different performance quality, used to enhance intra-class invariance, such as a standard squat versus a squat with slightly valgus knees. Positive samples are those that are in different action categories but visually similar to the anchor sample, used to enhance inter-class differentiation, such as a squat versus a lunge. In addition, during batch sampling, for each anchor sample, another segment of the same category is selected as positive, and similar segments of different categories are selected as negative.

[0106] Optionally, using a differentiable bone renderer as a regularization term in S3047 also includes: 1) Obtain the differentiable rendering regularization loss by combining the projected bone features of the current frame with the differentiable bone renderer; Specifically, pose classification tasks are susceptible to video background interference, leading to model overfitting. Conventional data augmentation methods cannot effectively maintain the topological constraints of the human skeleton and are difficult to utilize pixel-level supervision signals.

[0107] This invention employs a differentiable bone renderer as a regularization term, aligning intermediate bone features to pixel space and combining kinematic constraints to improve the model's generalization ability. The specific steps are as follows: 1) Define the differentiable rendering loss: Input the projected bone features of the current frame into the learnable inverse projection module to obtain 3D joint angle parameters, then input them into the differentiable bone renderer to generate a simulated human silhouette mask image. Then calculate the L2 norm squared difference between this rendered mask and the real human silhouette binary mask extracted from the video. Finally, sum the squared differences over all frames to obtain the differentiable rendering regularization loss. This constrains the intermediate bone features so that after inverse projection and rendering, they can approximate the real video silhouette, thereby achieving pixel-level supervision and improving the model's generalization ability. This is expressed as: ; It is a differentiable rendering regularization loss used to constrain skeletal features; A differentiable bone renderer that generates a mask image that simulates a human silhouette; It is a learnable inverse projection module. The input is the projected bone features, and the output is 3D joint angle parameters; For the first The real human silhouette binary mask of the frame is extracted from the video and obtained using a pre-trained segmentation model or manual annotation. Pre-trained segmentation models include U-Net or MaskR-CNN. It is an L2 norm.

[0108] It should be noted that the learnable inverse projection module can employ a multilayer perceptron, such as a 3-layer perceptron, with 2D projected skeletal features as input. It outputs 3D joint angle parameters, mapping 2D features to 3D kinematic space.

[0109] It should also be noted that, in order to specifically address the issues of video background interference and skeletal topology preservation, differentiable rendering regularization loss is used. Combining a differentiable renderer and an inverse projection module, instead of conventional data augmentation, to project skeletal features Inverse projection is applied to the 3D joint corners and rendered as a silhouette mask. A pixel-level supervised method is used to compare the mask with a real human silhouette binary mask. Alignment enhances the model's generalization ability, reduces overfitting to specific backgrounds, and strengthens the physical rationality of features through kinematic constraints.

[0110] 2) Combine differentiable rendering regularization loss to obtain a mask image of simulated human silhouette.

[0111] Specifically, the differentiable bone renderer, based on a predefined set of human bone edges, uses a differentiable line segment rendering function to generate binary masks of bone segments between the 3D joint coordinates output by the learnable inverse projection module. Then, all the bone segment masks are superimposed, and finally, a Gaussian blur operation is applied to the superimposed mask. The differentiable bone renderer can generate a mask image simulating a human silhouette, preserving the topological connectivity of the human skeleton, and simulating the motion blur effect in real video through the blur operation, as shown below: ; In the formula, Define the set of joint coordinates input to the renderer. ; A differentiable bone renderer that generates a mask image that simulates a human silhouette; For a predefined set of human skeleton edges, describe the topological connections between joints, such as shoulder-elbow, hip-knee, etc. For rendering differentiable line segments, at the joints... and Generate a line segment mask between intervals; The coordinates of the starting joint; The coordinates of the final joint; Indicates the first The first key point and the first The skeletal connections at each joint, such as... It represents the bones from the shoulder to the elbow; For Gaussian blur operation, a Gaussian convolution kernel is used to mask the line segments to simulate motion blur effect to match the characteristics of real video silhouettes.

[0112] It should be noted that the predefined human skeleton edge set Predefined features are implemented based on human anatomy, such as using the COCO keypoint model, which includes 17 keypoints. This includes connecting pairs such as (shoulder, elbow), (elbow, wrist), (hip, knee), (knee, ankle), etc., represented as The numbers are the key indexes.

[0113] It should also be noted that the differentiable line segment rendering function generates the coordinates of the starting joint in image space. Coordinates to the end joint The line segment, such as the differentiable version of the Bresenham algorithm, outputs a binary mask. The Bresenham algorithm is a classic raster graphics algorithm used to efficiently draw a line segment on a discrete pixel grid that is as close as possible to an ideal straight line. It uses integer arithmetic to avoid floating-point arithmetic and rounding. By calculating decision variables based on the slope of the line, it determines whether a pixel needs to be moved in another axis when moving one pixel along the principal axis, in order to select the pixel point closest to the ideal straight line. The differentiable Bresenham algorithm allows for computation... Regarding the set of key point coordinates The gradient, thus making the differentiable rendering regularization loss. The gradient can be backpropagated to the projected bone features through the renderer and the inverse projection module. Above, regularization is implemented to achieve pixel-level supervision.

[0114] Furthermore, parameter updates and iterative training also include: The model uses the Adam optimizer with a learning rate of 0.001 and is trained to minimize the joint loss function. Trajectory comparison loss With differentiable rendering regularization loss The summation of the loss values.

[0115] Optionally, 32 video clips are input in each batch, and gradient clipping is used to prevent gradient explosion, with the clipping threshold set to 1.0.

[0116] During the iterative training of the model, the loss function value is calculated in the forward propagation process, and all trainable parameters are updated in the backpropagation process. The training lasts for 1000 epochs. After each epoch, the classification accuracy is calculated on the validation set. If the accuracy does not improve for 5 consecutive epochs, or when the maximum number of epochs is reached, the training is stopped, and the model parameters with the best performance on the validation set are saved.

[0117] It should be noted that 10% of the samples in the training set can be reserved for validation of model performance during training.

[0118] The human motion posture recognition in S305 also includes: In the inference stage, the system performs full-process processing on the input fitness video. First, the human target recognition and tracking module extracts the coordinates of joint points in consecutive frames. Then, the spatiotemporal consistency completion mechanism is used to repair the missing key points, and the viewpoint invariant feature projection module is used to generate projected skeleton features. Furthermore, the projected skeletal features are input into the trained pose recognition model, and the recognition result is determined by the coarse-grained highest probability category index and its corresponding fine-grained maximum probability. For example, the final output pose recognition category is "squat (knee valgus)," which means that the coarse category is "squat" and the fine category is "knee valgus."

[0119] Furthermore, S4 also includes generating anomaly information based on the comparison between the pose recognition category and the standard pose template, and broadcasting the anomaly information. The feedback output and human-computer interaction module realizes real-time visualization feedback of fitness movements and historical data analysis. Through the OpenGL rendering engine, the identified skeletal key points are superimposed on the original video stream to generate dynamic skeletal animation. At the same time, the module compares the standard movement template with the user's current posture, calculates the joint angle deviation (such as the angle between the knee and ankle relative to the hip joint), and marks abnormal points in real time (such as knee valgus highlighted in red). Optionally, a speech synthesis engine can be used to read out correction instructions, such as using Google-TTS to read "Please keep your back straight".

[0120] In terms of historical data analysis, the system aggregates multiple training data to generate trend charts, including action completion score curves, joint range of motion radar charts, and calorie consumption estimates, and pushes weekly reports through mobile API.

[0121] It should be noted that OpenGL is a cross-platform graphics rendering interface library used to generate and overlay skeletal animations onto the user's video stream in real time within a fitness system.

[0122] It should also be noted that Google-TTS is a cloud-based text-to-speech service that converts fitness posture correction instructions into real-time voice prompts.

[0123] It should also be noted that API stands for Application Programming Interface, which enables the fitness system to make function calls and exchange data with external services.

[0124] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0125] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video analysis and fitness tracking method based on artificial intelligence, characterized in that, The methods include: S1. Generate human target localization data by combining the key point detection model and the acquired video data; S2. Generate human target tracking data by combining human target localization data and multi-target tracking model; S3. Generate posture recognition categories based on human target tracking data and human motion posture recognition models; S4. Generate anomaly information by comparing the pose recognition category with the standard pose template, and broadcast the anomaly information.

2. The video analysis and fitness tracking method based on artificial intelligence according to claim 1, characterized in that, The pose recognition categories generated in S3 also include: S301. Construct a training dataset for human motion posture recognition; S302. A spatiotemporal consistency completion mechanism based on motion trajectory prediction is used to dynamically repair missing key points in the training dataset and generate a second training dataset by utilizing the kinematic characteristics of adjacent frames. S303. By transforming the pelvic center coordinate system and using a learnable perspective compensation mechanism, the influence of shooting perspective differences in the second training dataset is eliminated, and perspective-invariant skeletal features are extracted to generate the third training dataset. S304. Train a human motion posture recognition model based on the third training dataset; S305. Generate posture recognition categories based on human motion posture recognition models.

3. The video analysis and fitness tracking method based on artificial intelligence according to claim 2, characterized in that, Generating the second training dataset also includes: S3021. Complete keypoint labeling for each frame in the training dataset. If the key point is complete, mark it. =1, then execute S3022; if the key point is complete. =0, then execute S3023; S3022. Obtain the original detection joint coordinates of multiple frames with different difference step sizes before and after the current frame, calculate the average value of the joint coordinate difference, and obtain the kinematic velocity estimation matrix. S3023. Based on the obtained neighboring valid reference frames of the current frame, predict the position of each missing joint point and generate the completed joint point coordinates. S3024. Generate a second training dataset based on the completed joint coordinates.

4. The video analysis and fitness tracking method based on artificial intelligence according to claim 2, characterized in that, Generate a third training dataset: S3031. Transform the completed joint coordinates in the second training dataset to a relative coordinate system with the pelvic center as the origin, to obtain the relative skeletal coordinates, represented as: ; In the formula, For the first Relative bone coordinates after frame conversion Indicates the first The coordinates of the joints after frame completion; For the first Average coordinates of the pelvic points in the frame; S3032. Construct standard view templates for four orthogonal directions: front, back, left, and right. S3033. Generate a view compensation matrix by combining the relative skeletal features of the current frame and the standard view template; S3034. Multiply the relative bone coordinates by the view compensation matrix to obtain the projected bone features, and generate the third training dataset based on the projected bone features.

5. The video analysis and fitness tracking method based on artificial intelligence according to claim 2, characterized in that, The human motion posture recognition model trained on the third training dataset in S304 also includes: S3041, Definition of Human Movement Posture Recognition Model Structure; S3042. Initialization of the human motion posture recognition model structure; S3043. Set up the attention mechanism of the human motion posture recognition model, and generate local features of predefined limb partitions based on the attention mechanism, and combine all local features of the current frame to generate spatiotemporal graph convolution features of the current frame. S3044. Obtain the long-term time-dependent features and short-term micro-motion features of the spatiotemporal graph convolution features, and generate fused macro-micro features based on the long-term time-dependent features and short-term micro-motion features. Dynamically integrate macro-motion patterns and micro-instantaneous change information through a gating mechanism. S3045. Calculate the hierarchical joint loss based on the fusion of macro and micro features, and construct a semantic pyramid classification layer based on the hierarchical joint loss; S3046. Calculate the trajectory contrast loss of a continuous frame sequence and decouple visual similarity features based on the trajectory contrast loss; S3047. Use a differentiable bone renderer as the regularization term; S3048. The Adam optimizer is used to minimize the loss function during training to obtain the final human motion posture recognition model.

6. The video analysis and fitness tracking method based on artificial intelligence according to claim 5, characterized in that, The spatiotemporal graph convolutional features used to generate the current frame in S3043 also include: 1) Generate the original joint correlation matrix based on the projected skeleton features; 2) Apply the Sigmoid activation function to the original joint correlation matrix to perform non-linear activation mapping and generate the joint attention matrix; 3) Based on the predefined limb partitions, obtain the local features of each limb partition in the current frame; 4) Generate spatiotemporal graph convolution features for the current frame based on the features of all limb partitions and the joint attention matrix respectively.

7. The video analysis and fitness tracking method based on artificial intelligence according to claim 5, characterized in that, The generation of fused macro- and micro-features in S3044 also includes: 1) Use a bidirectional long short-term memory network to process the sequence composed of spatiotemporal graph convolutional features of all frames to extract long temporal dependency features; 2) First, calculate the inter-frame difference of the spatiotemporal graph convolutional feature sequence to obtain the difference sequence that highlights instantaneous changes. Then, use a temporal convolutional network to process the difference sequence and extract short-term micro-motion features. 3) Combine long-term time-dependent features and short-term micro-motion features to generate fused macro- and micro-features, and dynamically integrate macro-motion patterns and micro-instantaneous change information through a gating mechanism.

8. The video analysis and fitness tracking method based on artificial intelligence according to claim 5, characterized in that, The construction of the semantic pyramid classification layer in S3045 also includes: 1) Calculate the average vector along the time dimension for the fused macro and micro features, and generate a coarse-grained classification probability vector based on the average vector; 2) The time-averaged vector that integrates macro and micro features is concatenated with the learnable semantic embedding vector of the current coarse category to generate concatenated features, and a fine-grained classification probability vector is generated based on the concatenated features. 3) Based on the logarithm of the predicted probability of the sample on the true coarse category of the current sample and the logarithm of the predicted probability on the true fine category, the hierarchical joint loss is obtained, and the semantic pyramid classification layer is constructed based on the hierarchical joint loss.

9. The video analysis and fitness tracking method based on artificial intelligence according to claim 5, characterized in that, The trajectory contrast loss for calculating consecutive frame sequences in S3046 also includes: 1) Obtain the projected skeletal features of continuous frame sequences to form motion trajectory segments, process the motion trajectory segment sequence through a bidirectional long short-term memory network, and generate the trajectory embedding vector of each motion trajectory segment. 2) Obtain the corresponding positive and negative samples based on the current action trajectory segment, and calculate the trajectory comparison loss of the current action trajectory segment based on the InfoNCE loss framework.

10. The video analysis and fitness tracking method based on artificial intelligence according to claim 5, characterized in that, The use of a differentiable bone renderer as a regularization term in S3047 also includes: 1) Obtain the differentiable rendering regularization loss by combining the projected bone features of the current frame with the differentiable bone renderer; 2) Combine differentiable rendering regularization loss to obtain a mask image of simulated human silhouette.

Images