A Vision-Based and Deep Learning-Based Method and System for Predicting Fitness Exercise Energy Consumption

Abstract

This invention belongs to the field of computer vision and pattern recognition technology, specifically relating to a method and system for predicting energy consumption during fitness activities based on vision and deep learning. It aims to address the problem that existing vision-based general energy consumption prediction models suffer from severe feature interference and limited prediction accuracy due to neglecting biomechanical differences between different movement patterns. The invention includes: acquiring video stream data and extracting key human body points to construct a skeleton sequence tensor; identifying movement pattern categories using a spatiotemporal graph convolutional network; extracting associated kinematic feature time series based on movement categories; inputting the features into an energy consumption prediction regression model with an independent parameter space, encoding temporal dependencies based on a self-attention mechanism, and outputting predicted values ​​of body activity intensity. This invention, through a hierarchical architecture of identification followed by prediction, effectively improves the accuracy and robustness of energy consumption prediction across movement patterns.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition, and specifically relates to a method and system for predicting the energy consumption of fitness movements based on vision and deep learning. Background Technology

[0002] Accurate estimation of human energy expenditure is of significant value for health management, exercise prescription development, and chronic disease prevention. Physical activity intensity is typically characterized by metabolic equivalents. In the field of energy expenditure measurement, the double-labeled water method and indirect calorimetry are widely recognized as the gold standard methods for caloric measurement due to their high reliability and accuracy. However, these methods rely on expensive and complex specialized equipment, have cumbersome operating procedures, and are usually limited to laboratory environments, making it difficult to meet the convenient and universal energy consumption monitoring needs of everyday fitness scenarios.

[0003] Energy consumption prediction methods based on wearable sensors have expanded the flexibility of monitoring scenarios to some extent. These methods collect acceleration signals during movement by wearing inertial measurement units such as accelerometers on specific parts of the body and establishing regression models to estimate energy consumption. However, wearable methods face several limitations in practical applications: their monitoring accuracy is highly sensitive to the sensor's wearing position, and positional deviations will significantly increase prediction errors; furthermore, these methods are difficult to deploy in large-scale, group-based monitoring scenarios, and long-term wear affects user comfort and compliance.

[0004] Compared to the methods mentioned above, computer vision-based energy consumption prediction methods have significant advantages such as being non-invasive, non-contact, and supporting multi-person parallel monitoring. Currently, vision-based energy consumption prediction research typically employs a single, general-purpose deep learning model, directly using video stream data as input and outputting end-to-end metabolic equivalent predictions. However, such general-purpose models ignore the inherent biomechanical differences between different movement patterns. Because the work mechanisms and involved muscle groups differ across different fitness movements, the key kinematic features for predicting energy consumption also differ fundamentally. When a general-purpose model attempts to fit mixed data containing multiple movement patterns within a unified parameter space, it not only struggles to capture subtle intensity variations within the same movement type but also suffers from severe feature interference: features that have significant predictive power for one type of movement often appear as noise signals in the prediction task for another type of movement. This feature interference leads to a significant decrease in the prediction accuracy of general-purpose models across different movement patterns and individual subjects, resulting in insufficient robustness.

[0005] In summary, existing computer vision-based energy consumption prediction methods face limitations in prediction accuracy within the framework of general models due to their inability to distinguish between the biomechanical and metabolic differences among different action modes. Therefore, there is an urgent need for an energy consumption prediction method capable of extracting differentiated kinematic features for different action categories and establishing independent prediction mechanisms. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, namely the severe feature interference and limited prediction accuracy caused by neglecting the biomechanical differences between different movement patterns in existing general vision-based energy consumption prediction models, this invention provides a method and system for predicting fitness movement energy consumption based on vision and deep learning.

[0007] In a first aspect, this invention proposes a method for predicting the energy consumption of fitness movements based on vision and deep learning, comprising:

[0008] The video stream data of the subject's actions is acquired, the pose is estimated from the video stream data, the two-dimensional coordinate sequence of human key points is extracted, and a skeleton sequence tensor representing the dynamic changes of human pose is constructed based on the two-dimensional coordinate sequence. The skeleton sequence tensor is input into a pre-trained action pattern recognition classification model. The action pattern recognition classification model extracts the spatial structure features and temporal dynamic features in the skeleton sequence through a spatiotemporal graph convolutional network, and outputs the action pattern category to which the current action belongs based on the extracted features. Based on the action pattern category, extract the kinematic feature time series associated with the current action pattern category from the skeleton sequence tensor. The kinematic feature time series contains multiple kinematic parameters with physiological significance. According to the action pattern category, the extracted kinematic feature time series is input into an independent energy consumption prediction regression model corresponding to the action pattern category. The independent energy consumption prediction regression model has a parameter space that is completely independent of the independent energy consumption prediction regression models corresponding to other action pattern categories. The independent energy consumption prediction regression model encodes the time-series dependence of the input kinematic features based on its self-attention mechanism and outputs the predicted value of the physical activity intensity corresponding to the current action.

[0009] Furthermore, the construction of the skeleton sequence tensor representing the dynamic changes in human posture further includes: Perform frame-by-frame pose estimation on the video stream data to obtain the coordinates of two-dimensional key points of multiple human targets in each frame; The system uses a pre-defined main target selection mechanism to calculate the area of ​​each detected human bounding box and its distance from the center of the image, which is then used to select the main subject targets that meet the pre-defined area and position conditions. Two-dimensional key point coordinate sequences of the main subject target are extracted, and the two-dimensional key point coordinate sequences are sliced ​​by sliding window based on a preset time window length to construct a four-dimensional skeleton sequence tensor with preset batch dimension, feature channel dimension, time dimension and key point number dimension.

[0010] Furthermore, the action pattern recognition classification model extracts spatial structural features and temporal dynamic features from the skeleton sequence through a spatiotemporal graph convolutional network, further including: The input skeleton sequence tensor is constructed into a topological graph structure of the human body, where the nodes of the topological graph structure correspond to the extracted human body key points, and the edges of the topological graph structure correspond to the natural connection relationships of the human skeleton. The topological graph structure is processed sequentially by stacking multiple spatiotemporal graph convolutional blocks, with each spatiotemporal graph convolutional block connected in series with a spatial graph convolutional module and a temporal convolutional module. The spatial graph convolution module is used to aggregate the spatial structural features between different human key points within the same video frame, and the temporal convolution module is used to extract the dynamic change features of human key point motion along the time dimension. Residual connections are introduced in each spatiotemporal graph convolution block to prevent network gradient vanishing.

[0011] Furthermore, the extraction of the kinematic feature time series associated with the current action pattern category further includes: Based on the coordinate changes of key human body points in the skeleton sequence tensor, multiple original kinematic features, including center of gravity displacement, center of gravity velocity, center of gravity acceleration, angular velocity of specified joints, and gait-related parameters, are calculated. The calculated raw kinematic features are standardized to eliminate dimensional differences between different raw kinematic features. The standardization process is performed based on the mean and standard deviation of the feature set. Based on the determined action pattern category, a subset of features that are correlated with the biomechanical properties of the action pattern category and meet preset conditions are selected from multiple raw kinematic features after standardization, and these subsets are used to construct the kinematic feature time series.

[0012] Furthermore, the independent energy consumption prediction regression model is a deep learning regression model based on a self-attention mechanism, and the method further includes: Independent energy consumption prediction regression models are pre-built and trained for various different action mode categories. The network weight parameters of each independent energy consumption prediction regression model are not shared with other independent energy consumption prediction regression models for other action mode categories during the training and inference phases.

[0013] Furthermore, the independent energy consumption prediction regression model encodes the time-dimensional dependencies of the input kinematic feature time series based on its self-attention mechanism, further including: The kinematic parameters constituting the kinematic feature time series are embedded as input and fed into a network structure composed of stacked multi-layer encoders; In each encoder layer, the correlation weights between different time frames in the input sequence are calculated through a multi-head self-attention mechanism. The calculation of the correlation weights involves the mapping query matrix, key matrix, and value matrix of the input sequence, and the product of the query matrix and the key matrix is ​​scaled based on the dimension of the key matrix. The features aggregated by the multi-head self-attention mechanism are input into the feedforward neural network for nonlinear transformation, and residual connections are introduced at the output of each sub-layer. The encoded features are linearly mapped through the output layer to obtain the final predicted value of physical activity intensity.

[0014] Furthermore, when processing the kinematic feature time series, the independent energy consumption prediction regression model captures the metabolic lag effect in the human energy consumption process through the self-attention mechanism, which is used to model the dependency between the predicted value of physical activity intensity at the current moment and the exercise state at the previous moment.

[0015] Furthermore, the training process of the action pattern recognition classification model and the independent energy consumption prediction regression model further includes: Construct a raw dataset containing video streams of various motion modes and their synchronously acquired real-world physical activity intensity values; The original dataset is divided into a training set and a test set. The action pattern recognition classification model is trained using the training set until its action classification accuracy reaches the preset condition. Based on action classification, the training set data is divided into multiple subsets according to the identified action pattern categories. Each subset is then used to independently train and adjust the hyperparameters of the corresponding independent energy consumption prediction regression model until the prediction error index of each independent energy consumption prediction regression model converges.

[0016] Furthermore, the predicted value of the physical activity intensity is a metabolic equivalent value.

[0017] In a second aspect, the present invention proposes a fitness movement energy consumption prediction system based on vision and deep learning, for executing a fitness movement energy consumption prediction method based on vision and deep learning, the system comprising: The skeleton sequence construction module is used to acquire video stream data of the subject's actions, perform pose estimation on the video stream data, extract two-dimensional coordinate sequences of human key points, and construct a skeleton sequence tensor representing the dynamic changes of human pose based on the two-dimensional coordinate sequences. An action pattern recognition module is used to receive the skeleton sequence tensor and input the skeleton sequence tensor into a pre-trained action pattern recognition classification model. The action pattern recognition classification model extracts spatial structure features and temporal dynamic features from the skeleton sequence through a spatiotemporal graph convolutional network and outputs the action pattern category to which the current action belongs based on the extracted features. The feature extraction module is used to extract a kinematic feature time series associated with the current action pattern category from the skeleton sequence tensor according to the action pattern category. The kinematic feature time series contains multiple kinematic parameters with physiological significance. The model selection and input module is used to input the extracted kinematic feature time series into an independent energy consumption prediction regression model corresponding to the action pattern category, based on the action pattern category. The independent energy consumption prediction regression model has a parameter space that is completely independent of the independent energy consumption prediction regression models corresponding to other action pattern categories. The energy consumption prediction output module is used to encode the time-dimensional dependency of the input kinematic feature time series by the independent energy consumption prediction regression model based on its self-attention mechanism, and output the predicted value of the physical activity intensity corresponding to the current action.

[0018] The beneficial effects of this invention are: This invention effectively overcomes the limitation of prediction accuracy caused by feature interference in existing general-purpose models when handling multiple fitness movements by constructing a hierarchical deep learning framework that first identifies and then predicts. Specifically, this invention first uses a spatiotemporal graph convolutional network to classify the movement patterns of human skeletal sequences, clarifying the specific movement category performed by the current subject, thereby decomposing the complex multi-movement mixed prediction task into independent regression tasks for single movement categories. Based on this, and considering the biomechanical characteristics of different movement categories, it differentially extracts kinematic features highly correlated with the movement from the skeletal sequence, including physical quantities with clear physiological significance such as center of gravity displacement, center of gravity velocity, center of gravity acceleration, joint angular velocity, and gait-related parameters, rather than relying on the model to implicitly learn features from raw pixels, thus enhancing the interpretability and predictive stability of the features.

[0019] This invention constructs an energy consumption prediction regression model with an independent parameter space for each action category, and the weight parameters of each model are never shared during training and inference. This design allows each independent model to focus on learning the specific mapping relationship between kinematic features and energy consumption in the corresponding action mode, avoiding parameter update conflicts and prediction biases caused by differences in feature distribution between different action modes, and significantly improving the prediction accuracy and robustness of the model in cross-action mode scenarios.

[0020] The independent energy consumption prediction regression model employed in this invention is built upon a self-attention mechanism, which can effectively capture the metabolic lag effect during the human energy consumption process. Compared with traditional recurrent neural networks or convolutional networks, the self-attention mechanism possesses a global receptive field, enabling direct modeling of the dependencies between any time steps in the input kinematic feature time series. This allows for a more accurate reflection of the physiological response delay characteristics of current energy consumption to previous motion states, further improving the temporal accuracy of energy consumption prediction.

[0021] This invention enables non-invasive, non-contact energy consumption monitoring using only ordinary camera equipment to capture video streams, eliminating reliance on expensive thermal instruments or wearable sensors. It also supports parallel monitoring of multiple individuals, facilitating its widespread application in various practical scenarios such as gyms, school physical education, community health management, and elderly care. Furthermore, its embedded primary target filtering mechanism effectively suppresses interference from multiple individuals in the frame, ensuring the stability and accuracy of posture extraction for key subjects even in complex environments.

[0022] In summary, this invention solves the problems of feature interference and accuracy decay in general energy consumption prediction models from a mechanistic perspective by using action prior-guided feature differentiation extraction and independent model parameter design, and achieves high-precision and robust prediction of fitness movement energy consumption based on ordinary video input. Attached Figure Description

[0023] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the workflow of a fitness movement energy consumption prediction method based on vision and deep learning according to the present invention. Figure 2 This is a schematic diagram of the prediction process of a fitness movement energy consumption prediction method based on vision and deep learning according to the present invention. Figure 3 This is a schematic diagram of a motion pattern recognition classification model based on a spatiotemporal graph convolutional network in a fitness motion energy consumption prediction method based on vision and deep learning according to the present invention. Figure 4This is a schematic diagram of the energy consumption prediction regression model based on a deep learning network in the energy consumption prediction method for fitness movements based on vision and deep learning of the present invention. Figure 5 This is a structural diagram of a fitness movement energy consumption prediction system based on vision and deep learning according to the present invention. Figure 6 This is a schematic diagram of the structure of a computer system used to implement the methods, systems, and electronic devices of this application. Detailed Implementation

[0024] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] The first embodiment of this invention proposes a method for predicting the energy consumption of fitness movements based on vision and deep learning, comprising: Step S10: Obtain video stream data of the subject's actions, perform pose estimation on the video stream data, extract two-dimensional coordinate sequences of human key points, and construct a skeleton sequence tensor representing the dynamic changes of human pose based on the two-dimensional coordinate sequences. Step S20: Input the skeleton sequence tensor into a pre-trained action pattern recognition classification model. The action pattern recognition classification model extracts the spatial structure features and temporal dynamic features in the skeleton sequence through a spatiotemporal graph convolutional network, and outputs the action pattern category to which the current action belongs based on the extracted features. Step S30: Based on the action pattern category, extract the kinematic feature time series associated with the current action pattern category from the skeleton sequence tensor. The kinematic feature time series contains multiple kinematic parameters with physiological significance. Step S40: According to the action pattern category, the extracted kinematic feature time series is input into an independent energy consumption prediction regression model corresponding to the action pattern category. The independent energy consumption prediction regression model has a parameter space that is completely independent of the independent energy consumption prediction regression models corresponding to other action pattern categories. Step S50: The independent energy consumption prediction regression model encodes the time-dimensional dependency of the input kinematic feature time series based on its self-attention mechanism, and outputs the predicted value of the physical activity intensity corresponding to the current action.

[0027] To more clearly explain the energy consumption prediction method for fitness movements based on vision and deep learning in this invention, the following will be combined with... Figure 1 and Figure 2 The steps in the embodiments of the present invention are described in detail below: Step S10: Obtain video stream data of the subject's actions, perform pose estimation on the video stream data, extract two-dimensional coordinate sequences of human key points, and construct a skeleton sequence tensor representing the dynamic changes of human pose based on the two-dimensional coordinate sequences. In this embodiment, constructing a skeleton sequence tensor representing the dynamic changes in human posture further includes: Step S11: Perform multi-person pose estimation frame by frame on the video stream data to obtain the coordinates of the two-dimensional key points of each of the multiple human targets in each frame. Step S12: Through a preset main target screening mechanism, calculate the area of ​​each detected human bounding box and its distance relative to the center of the image to screen out the main subject targets that meet the preset area and position conditions. Step S13: Extract the two-dimensional key point coordinate sequence of the main subject target, and perform sliding window slicing on the two-dimensional key point coordinate sequence based on a preset time window length to construct a four-dimensional skeleton sequence tensor with preset batch dimension, feature channel dimension, time dimension and key point number dimension.

[0028] In step S10, video stream data of the subject performing various fitness movements or daily activities is acquired. The video stream can be acquired using a monocular or multi-view RGB camera, with a frame rate set to 30 frames per second to ensure sufficient continuity and detail resolution in the captured action sequence over time, thus meeting the sampling density requirements for subsequent pose estimation and dynamic feature extraction. For the acquired video stream data, an embedded pose estimation network is used to perform two-dimensional human pose estimation frame by frame. In this embodiment, the pose estimation network preferably uses the YOLOv8-Pose model, which has the ability to simultaneously perform human detection and keypoint localization in a single frame image, and can directly output the two-dimensional coordinates of 17 human keypoints conforming to the COCO format definition. These 17 keypoints include joints such as the nose, both eyes, both ears, both shoulders, both elbows, both wrists, both hips, both knees, and both ankles, covering the main trunk and limb joints of the human body, and can relatively completely depict the posture configuration of the human body during movement. By running this pose estimation network frame by frame on the video stream, the two-dimensional keypoint coordinate sequence of all detected human objects in each frame can be obtained. Building upon this foundation, to construct a skeleton sequence tensor capable of representing dynamic changes in human posture, the raw two-dimensional coordinate data extracted frame by frame needs to be tensorized according to a predefined organizational structure, serving as standardized input for subsequent spatiotemporal feature extraction networks. This tensorization process involves not only the temporal arrangement of keypoint coordinate sequences but also the fusion of multi-channel feature information and the extraction of fixed time windows, thereby forming a four-dimensional skeleton sequence tensor with a unified dimensional definition.

[0029] The process of constructing the skeleton sequence tensor representing the dynamic changes in human posture described in step S10 can be further implemented by including the specific operations defined in steps S11 to S13.

[0030] In step S11, frame-by-frame multi-person pose estimation is performed on the video stream data. In real-world applications such as community activity rooms, gyms, or nursing homes, multiple people often appear simultaneously in the video frame, including the subject and bystanders or passersby. The pose estimation network may detect multiple different human targets in each frame and output the corresponding 17 key point coordinates for each. If all detection results are directly input into subsequent processing, the motion information of non-target individuals will be mixed into the action sequence, severely interfering with action pattern recognition and energy consumption prediction. Therefore, it is necessary to filter the multi-person detection results in each frame to identify the single primary subject target.

[0031] In step S12, this invention introduces a preset primary target selection mechanism to achieve robust locking of the main subject target in the multi-person detection results. Specifically, for each human target detection result output by the pose estimation network in each frame, the corresponding bounding box area and the Euclidean distance of the bounding box center point relative to the center point of the entire frame are calculated. The bounding box area reflects the imaging size of the human target in the image. Typically, the subject is an individual actively participating in the test, and their distance from the camera is relatively fixed and they are in a prominent position in the image. Therefore, their bounding box area is often larger than that of other background figures who accidentally enter the image. The center distance measures the degree of deviation of the human target from the center of the field of view. Since the subject is usually placed in the central area of ​​the image during data acquisition, the closer the target is to the center, the more likely it is to be the real subject. Based on the above two metrics, the primary target selection mechanism selects the target with the largest area and closest to the center of the image as the primary subject by comparing the area size and center distance of each target, and retains its corresponding key point coordinate sequence, while discarding the data of the other targets. This screening mechanism is executed independently in each frame, effectively avoiding the influence of background figures or interfering personnel on subsequent analysis and ensuring the temporal consistency and purity of the motion sequence data. Even when collecting data in public places with high population flow, this mechanism can stably track the movement process of the same subject, providing a reliable single-individual skeleton data stream for subsequent motion recognition and energy consumption prediction.

[0032] In step S13, after successfully extracting the frame-by-frame two-dimensional keypoint coordinate sequence of the main subject, the two-dimensional keypoint coordinate sequence is further structured and standardized into slices to construct a fixed-dimensional skeleton sequence tensor suitable for deep neural network input. Specifically, the two-dimensional coordinates of the 17 keypoints of the main subject in each frame are... Keypoints are extracted sequentially according to their index and arranged frame by frame along the time axis, forming a two-dimensional coordinate sequence with the frame number as the time step. To enhance the expressive power of motion dynamics, in addition to preserving the original position coordinate information, the motion velocity components of each keypoint in the horizontal and vertical directions are calculated through coordinate difference operations between adjacent frames. Let the coordinates of a keypoint in frame t be... The coordinates of the (t-1)th frame are Then the velocity component of the key point in frame t can be approximated as: and ,in This represents the time interval between adjacent frames. Since the video frame rate is 30 frames per second, The time is approximately 0.033 seconds. To simplify calculations and maintain numerical stability, the coordinate difference value can be directly used as the velocity feature. Therefore, in the feature channel dimension, each keypoint not only contains the original two-dimensional position coordinates... It also includes corresponding two-dimensional velocity information. This brings the total number of feature channels to four.

[0033] Considering that the energy consumption of human movement is closely related to the continuous dynamic process of the movement, and that different movement types, such as squats, jogging, and lunges, require different durations within a single complete cycle, this invention employs a preset fixed time window to perform sliding window slicing on continuous skeleton sequences in order to fully cover one or more movement cycles while avoiding the introduction of excessively long sequences that would lead to computational redundancy. The window length is set to 150 frames, corresponding to a 5-second duration in a video captured at 30 frames per second. This 5-second time window is sufficient to contain at least one complete movement cycle, or even multiple cycles, for most common periodic fitness movements such as squats, jumping jacks, and high knees, and non-periodic daily activities such as walking and climbing stairs, thus ensuring that the sliced ​​data segments can fully represent the spatiotemporal evolution of the movement. An appropriate overlap step size can be set between adjacent windows, for example, a step size of 75 frames, i.e., an overlap rate of 50%. This increases the number of samples without losing movement boundary information, thereby improving the generalization ability of the model training.

[0034] After sliding window slicing, each sample is constructed as a four-dimensional skeleton sequence tensor with a fixed-dimensional structure. The dimension definition of this four-dimensional tensor can be represented by the following notation: In this model training process, the first dimension N represents the batch size, which is typically set to 32, meaning 32 independent samples are input for each iteration. The second dimension C represents the number of feature channels, which in this embodiment includes two-dimensional position coordinates and two-dimensional velocity components, so C=4. The third dimension T represents the time series length, i.e., the number of frames within the window, where T=150. The fourth dimension V represents the number of keypoints within a single frame, where V=17. Through this dimensional organization, the original video stream undergoes a series of processes, including pose estimation, main target selection, feature channel expansion, and fixed window slicing, ultimately transforming into a skeleton sequence tensor data with a unified structure, well-defined dimensions, and filtered-out noise. This data serves as the standard input format for subsequent action pattern recognition classification models based on spatiotemporal graph convolutional networks, stably and efficiently supporting the automatic learning and extraction of spatiotemporal features by deep networks.

[0035] Step S20: Input the skeleton sequence tensor into a pre-trained action pattern recognition classification model. The action pattern recognition classification model extracts the spatial structure features and temporal dynamic features in the skeleton sequence through a spatiotemporal graph convolutional network, and outputs the action pattern category to which the current action belongs based on the extracted features. In this embodiment, the action pattern recognition classification model extracts spatial structural features and temporal dynamic features from the skeleton sequence through a spatiotemporal graph convolutional network, and further includes: Step S21: Construct the input skeleton sequence tensor into a topological graph structure of the human body, where the nodes of the topological graph structure correspond to the extracted human body key points, and the edges of the topological graph structure correspond to the natural connection relationships of the human skeleton. Step S22: The topological graph structure is processed sequentially by stacking multiple spatiotemporal graph convolutional blocks, with each spatiotemporal graph convolutional block connected in series with a spatial graph convolutional module and a temporal convolutional module. The spatial graph convolution module is used to aggregate the spatial structural features between different human key points within the same video frame, and the temporal convolution module is used to extract the dynamic change features of human key point motion along the time dimension. Residual connections are introduced in each spatiotemporal graph convolution block to prevent network gradient vanishing.

[0036] In step S20, the skeleton sequence tensor constructed in steps S10 to S13 is input into a pre-trained action pattern recognition classification model. The core of this action pattern recognition classification model employs a spatiotemporal graph convolutional network architecture. Its function is to jointly extract spatial structural features and temporal dynamic features from the input skeleton sequence tensor, and to determine the specific motion pattern category of the current action sequence based on the extracted spatiotemporal features. The motion pattern category may include, for example, various fitness movements or daily activities such as squats, lunges, walking, jogging, high knees, and jumping jacks. Because there are fundamental biomechanical differences between the energy metabolism characteristics of human movements and motion patterns, the key energy consumption prediction features corresponding to different movement types are different. Therefore, this step first accurately identifies the action category as a prerequisite for subsequently constructing an independent energy consumption regression model for each specific movement.

[0037] The hierarchical design of this invention effectively avoids the feature interference problem caused by the general model when fitting mixed motion data, enabling different motion modes to be modeled more specifically in their own dedicated parameter spaces.

[0038] The process of the action pattern recognition classification model described in this embodiment extracting spatial structural features and temporal dynamic features from the skeleton sequence through a spatiotemporal graph convolutional network can be further implemented by including the specific operations defined in steps S21 to S22.

[0039] In step S21, the input skeleton sequence tensor is constructed into a topological graph of the human body. The four-dimensional skeleton sequence tensor output from step S13 has... The dimensional organization is as follows: the batch dimension N is 32, the feature channel dimension C contains a total of 4 channels including position and velocity information, the time dimension T is 150 frames, and the number of key points dimension V is 17.

[0040] In this tensor, for each independent sample, the coordinate data of 17 key points on a single time frame are taken. These 17 key points are regarded as nodes of the topology graph. Each node is associated with a feature vector, the initial value of which is composed of the two-dimensional coordinates and velocity components of the key point in the corresponding frame. The edges of the topology graph structure are defined based on the natural connection relationship of the human skeleton.

[0041] Specifically, following the physical connections between adjacent joints in human anatomy, a set of edge connections is established between nodes. For example, edge connections are established between joint pairs such as the nose and eyes, shoulders and elbows, elbows and wrists, hips and knees, and knees and ankles. This also includes cross-truncation connections such as between the left and right shoulders and hips, forming a human skeleton graph with a fixed spatial topology. The topological connections of this skeleton graph remain consistent across all time frames, but the feature values ​​of each node change over time in each frame, thus forming a spatiotemporal graph that depicts both the spatial coupling relationships between different joints within the same frame and the motion evolution trajectory of the same joint along the time axis. After transforming the original tensor form into a graph structure representation, graph convolution operations can be used to efficiently aggregate information from adjacent nodes in non-Euclidean space, laying the data structure foundation for feature extraction from subsequent spatiotemporal graph convolutional blocks.

[0042] In step S22, the topological graph structure is processed sequentially by stacked spatiotemporal graph convolutional blocks. The spatiotemporal graph convolutional network constructed in this invention consists of several stacked spatiotemporal graph convolutional blocks. Each spatiotemporal graph convolutional block serves as a basic feature extraction unit, internally containing a spatial graph convolutional module and a temporal convolutional module, thereby achieving alternating extraction and deep fusion of spatial and temporal dimension features. The spatial graph convolutional module operates on the skeleton graph structure within a single time frame, and its computational goal is to aggregate the spatial structural features between different human keypoints within the same video frame. In each time slice, the spatial graph convolutional module, based on predefined skeletal connection edges, aggregates the features of adjacent nodes of each node through graph convolutional kernels, updating the feature representation of the current node. This process ensures that the feature vector of each joint not only contains its own kinematic information but also incorporates information from neighboring joints with which it has skeletal connections, thereby capturing the overall configuration of the human posture and the relative positional relationships between limb segments.

[0043] The temporal convolution module follows the spatial graph convolution module, extracting the dynamic changes in the motion of each joint along the time dimension. This module treats the node feature sequence after spatial feature aggregation as a one-dimensional signal arranged along the time axis, and performs convolution operations along the time direction using one-dimensional convolution. The receptive field of temporal convolution covers several consecutive frames, effectively extracting the dynamic patterns of joint position and velocity evolution over time, such as the coordinated flexion and extension rhythm of the hip and knee joints during squatting, and the alternating swing frequency of the arms and legs during walking. By sequentially executing spatial graph convolution and temporal convolution within a spatiotemporal graph convolution block, the network can simultaneously complete spatial structure modeling and temporal dynamic modeling in a single forward propagation step, thereby learning high-level motion feature representations that possess both spatial discriminative and temporal discriminative capabilities.

[0044] To improve the training stability of deep networks and alleviate the gradient vanishing problem, a residual connection mechanism is introduced within each spatiotemporal graph convolutional block. Specifically, in each spatiotemporal graph convolutional block, the feature tensor input to that block is transformed by a concatenation of spatial graph convolution and temporal convolution, and then element-wise added to the original input tensor to obtain the final output of that block. This residual connection design allows gradients to be directly propagated to earlier network layers during backpropagation through the identity mapping path, effectively avoiding the gradient decay phenomenon that may occur as network depth increases. The mathematical expression of this residual calculation process is as follows: ; in, This represents the feature tensor input to the current spatiotemporal graph convolutional block; This represents a spatial graph convolution operation, which aggregates information about adjacent nodes on a single frame skeleton graph; This represents a nonlinear activation function. In this embodiment, the ReLU activation function is used to introduce nonlinear transformation capabilities into the network. This represents a temporal convolution operation, which performs convolution operations on the node features of consecutive frames along the time dimension. This represents a residual connection branch, which is typically an identity mapping or an adjustment branch containing lightweight convolutions to maintain consistency between the input and output dimensions. This represents the output feature tensor of the current spatiotemporal graph convolutional block, which will be used as the input to the next layer or fed into a subsequent classifier.

[0045] By stacking multiple spatiotemporal graph convolutional blocks with the aforementioned structure, the network can progressively extract hierarchical spatiotemporal features, from local joint movements to coordinated whole-body actions, from shallow to deep. At the end of the network, the feature tensor output by the last spatiotemporal graph convolutional block is subjected to global pooling to compress the temporal and spatial dimensions, resulting in a fixed-length feature vector. This feature vector is then mapped to the probability distribution of each action category through a fully connected classification layer, ultimately outputting the specific motion pattern category corresponding to the current video sequence, thus completing the action recognition task.

[0046] Step S30: Based on the action pattern category, extract the kinematic feature time series associated with the current action pattern category from the skeleton sequence tensor. The kinematic feature time series contains multiple kinematic parameters with physiological significance. In this embodiment, extracting the kinematic feature time series associated with the current action pattern category further includes: Step S31: Based on the coordinate changes of key human body points in the skeleton sequence tensor, calculate and obtain multiple original kinematic features including center of gravity displacement, center of gravity velocity, center of gravity acceleration, angular velocity of specified joints, and gait-related parameters. Step S32: Standardize the calculated original kinematic features to eliminate dimensional differences between different original kinematic features. The standardization process is performed based on the mean and standard deviation of the feature set. Step S33: Based on the determined action pattern category, select a subset of features from the standardized original kinematic features that meet the preset conditions for correlation with the biomechanical characteristics of the action pattern category, and use them to construct the kinematic feature time series.

[0047] In step S30, after the action pattern recognition classification model outputs the specific action pattern category corresponding to the current video sequence in step S20, kinematic feature time series associated with the determined action pattern category are selectively extracted from the skeleton sequence tensor. This kinematic feature time series consists of multiple kinematic parameters with clear physiological and biomechanical significance, and its extraction process fully utilizes the spatiotemporal coordinate information of key human body points contained in the skeleton sequence tensor. Because different fitness movements have fundamentally different work patterns, the key kinematic variables that need to be considered for predicting energy consumption are also quite different. For example, for squatting movements where the vertical movement of the center of gravity is the primary work pattern, the displacement and velocity changes of the body's center of gravity have a strong explanatory power for energy consumption; while for jogging movements characterized by periodic strides, stride frequency, stride length, and changes in the angular velocity of the lower limb joints become more dominant energy consumption predictors. Therefore, this step selectively extracts a subset of kinematic features that are highly relevant to the current action, rather than indiscriminately including all computable kinematic parameters into the input of the energy consumption regression model, under the premise of clearly defining the action category. This design effectively avoids noise interference from irrelevant features during model training and significantly reduces the input dimension of the subsequent Transformer encoding layer, which helps to improve the accuracy and generalization ability of energy consumption prediction.

[0048] The process of extracting the kinematic feature time series associated with the current action pattern category as described in this embodiment can be further implemented by including the specific operations defined in steps S31 to S33.

[0049] In step S31, based on the coordinate changes of key human body points in the skeleton sequence tensor, multiple raw kinematic features, including center of gravity displacement, center of gravity velocity, center of gravity acceleration, angular velocity of specified joints, and gait-related parameters, are calculated. The skeleton sequence tensor provides a two-dimensional coordinate sequence and velocity components of 17 key human body points of the main subject within a continuous 150-frame time window. Using this basic data, a series of quantitative indicators reflecting the overall motion state of the human body and local joint activities can be calculated frame by frame using kinematic formulas. The coordinates of the body's center of gravity can be estimated by weighting the coordinates of each key point and the corresponding mass weights of human body segments. In a simplified implementation, the midpoint of the line connecting the left and right hip joints can also be used as a proxy variable for the center of gravity position. Let the coordinates of the left and right hip joints in frame t be... and Then the barycenter coordinates It can be approximated as: ; ; Based on the time series of the center of gravity coordinates, the velocity and acceleration of the center of gravity are further obtained through difference operations. The velocity components of the center of gravity in the horizontal and vertical directions can be obtained by dividing the difference in the center of gravity coordinates between adjacent frames by the frame interval time, and the acceleration is further obtained by the difference in velocity. The total displacement of the center of gravity can be obtained by summing the inter-frame displacement of the center of gravity coordinates within the time window. In addition to the overall kinematic parameters of the center of gravity, the angular velocity of local joints is also an important indicator of the intensity of the movement. Taking the knee joint angular velocity as an example, it is obtained by calculating the rate of change of the angle between the thigh vector (hip joint pointing to knee joint) and the lower leg vector (knee joint pointing to ankle joint). Let the coordinates of the hip, knee, and ankle in frame t be... Then the thigh vector calf vector The angle between the two From the dot product formula The angular velocity of the knee joint was obtained. Then, based on the rate of change of the included angle in adjacent frames The angular velocities of major joints such as the hip, ankle, shoulder, and elbow can be calculated similarly. For gait-related parameters, based on the recognition results of periodic movements such as walking and jogging, gait features such as stride frequency, stride length, and the proportion of support and swing phases can be extracted by analyzing the changes in the coordinates of key points on both feet over time. All of the above-mentioned original kinematic features are calculated directly or indirectly from the skeletal sequence tensor, possessing clear biomechanical meaning and providing physically interpretable input variables for subsequent energy consumption regression models.

[0050] In step S32, the calculated raw kinematic features are standardized to eliminate the influence of differences in units and numerical ranges between different raw kinematic features. Since the numerical range of center of gravity displacement may be in meters and relatively small, while the numerical range of center of gravity acceleration or joint angular velocity may reach tens or even hundreds, directly inputting unprocessed raw features into the neural network would cause the gradient update direction during network training to be dominated by large-value features, weakening the role of small-value features, thereby reducing the model's convergence speed and prediction accuracy. Therefore, this embodiment uses the Z-score standardization method to process each kinematic feature separately. The standardization process is performed based on the mean and standard deviation of the feature across the entire training dataset. Let a certain kinematic feature be... The values ​​of all samples across all time frames in the training set constitute a population distribution, and the mean of this distribution is denoted as . The standard deviation is Then the standardized eigenvalues Calculated by the following formula: ; After the above transformation, each kinematic feature is mapped to a standard normal distribution with a mean of 0 and a standard deviation of 1, giving different features a comparable numerical scale. This standardization process not only improves the training stability of the neural network but also helps avoid biases in correlation measurement caused by different units when selecting features based on their correlation with energy consumption in subsequent steps.

[0051] In step S33, based on the action pattern category determined in step S20, a subset of features that satisfy preset conditions and are correlated with the biomechanical characteristics of the action pattern category are selected from multiple raw kinematic features after standardization. This subset is used to construct the kinematic feature time series. The specific action energy consumption prediction model of the present invention has an independent parameter space, and its key support lies in reserving exclusive feature subsets for different action categories. For each type of action, the present invention predefines the feature selection scheme corresponding to the action category based on prior knowledge of exercise physiology and feature correlation analysis. Taking the squatting action as an example, this action mainly involves the coordinated flexion and extension of the hip, knee, and ankle joints of the lower limbs and the vertical rise and fall of the trunk. Therefore, the kinematic features highly correlated with it include vertical displacement of the center of gravity, vertical velocity of the center of gravity, hip joint angular velocity, knee joint angular velocity, and ankle joint angular velocity. Taking the jogging action as an example, its energy consumption is closely related to indicators such as stride frequency, stride length, horizontal velocity of the center of gravity, and peak knee joint angular velocity. Taking the lunge action as an example, the angular velocity of the front knee joint, the angular velocity of the rear hip joint extension, and the forward and backward movement of the center of gravity become key features. In practice, the Pearson correlation coefficient or mutual information value between each candidate kinematic feature and the true metabolic equivalent (MET) can be calculated in advance for each type of action, and features with a correlation higher than the set threshold can be selected to form a unique feature subset.

[0052] After filtering, for the current input sample, only the kinematic parameters in the feature subset corresponding to its action pattern category are retained, forming the kinematic feature time series of that sample. This time series can be organized as a three-dimensional input tensor, whose dimensions are represented as... ,in This is the batch size, set to 32 during the training phase; This is the time step, corresponding to the number of frames in the sliding window, which is 150. The number of filtered features can be dynamically adjusted between 10 and 20 depending on the action category; in this example, it is 14. Through the above-mentioned action-related feature filtering mechanism, the subsequent energy consumption regression model can focus on learning the motion pattern directly related to the energy metabolism of the action, eliminating noise introduced by other irrelevant or weakly correlated features, thereby significantly improving prediction accuracy and robustness across actions.

[0053] Furthermore, the step of selecting a subset of features from the standardized raw kinematic features that satisfy preset conditions and are correlated with the biomechanical characteristics of the movement pattern category further includes: From the standardized kinematic feature time series corresponding to the skeleton sequence tensor, select an anchor feature that has a clear periodicity in the current action mode, and determine the action cycle length based on the anchor feature; Based on the motion cycle length, cross-cycle waveform consistency tests are performed on each candidate kinematic feature, and candidate kinematic features that pass the waveform consistency test are selected and retained. The selected and retained candidate kinematic features are further subjected to a phase consistency test with the anchored features. The candidate kinematic features that pass the phase consistency test are included in the feature subset corresponding to the current action mode category and arranged in the original time order to form the kinematic feature time series.

[0054] The cross-cycle waveform consistency check further includes: Calculate the autocorrelation function of the anchor feature within the time window, determine the action cycle length based on the significant peak position of the autocorrelation function, and perform periodic segmentation on the anchor feature sequence according to the action cycle length to obtain multiple periodic segments resampled and aligned with a uniform length. For each candidate kinematic feature, extract its resampled sequence fragments in each period, use the sequence fragment of the first period as a reference template, calculate the dynamic time warping distance between the fragments of other periods and the reference template, and obtain the average normalized dynamic time warping distance of the candidate kinematic feature based on the dynamic time warping distance. Candidate kinematic features whose average normalized dynamic time warp distance is less than or equal to the first judgment threshold are judged to pass the waveform consistency test.

[0055] The phase consistency check further includes: For candidate kinematic features that pass the waveform consistency test, the instantaneous phase of the candidate kinematic feature and the anchoring feature are calculated respectively. The instantaneous phase is determined based on the analytical signal obtained after performing a Hilbert transform on the feature sequence. Calculate the cyclic variance of the difference between the instantaneous phase of the candidate kinematic feature and the instantaneous phase of the anchoring feature over the entire time window; Candidate kinematic features whose cyclic variance of instantaneous phase difference is less than or equal to the second decision threshold are judged to pass the phase consistency test.

[0056] In this embodiment, in step S33, based on the movement pattern category determined in step S20, a subset of features that satisfy preset conditions for correlation with the biomechanical characteristics of the movement pattern category is selected from multiple standardized original kinematic features. This process can be further realized by introducing a refined screening mechanism based on movement cycle consistency analysis. Since different fitness movements have their own inherent movement rhythms and limb coordination patterns, simply screening features based on correlation coefficient thresholds can eliminate irrelevant variables, but features that are statistically correlated with energy consumption in terms of numerical value but do not match the periodic rhythm of the movement itself in terms of temporal morphology may still cause potential interference to the training of subsequent regression models. To this end, this embodiment further provides a feature screening method that combines waveform consistency and phase consistency to ensure that the kinematic parameters included in the exclusive feature subset are not only statistically related to energy consumption, but also highly coordinated with the periodic characteristics of the corresponding movement pattern in terms of temporal structure.

[0057] In this further screening mechanism, an anchoring feature with a clear periodicity in the current movement pattern is selected from the standardized kinematic feature time series corresponding to the skeleton sequence tensor, and the movement cycle length is determined based on the anchoring feature. The selection of the anchoring feature varies depending on the movement category. For lower limb strength-based movements such as squats and lunges, which are mainly based on the vertical rise and fall of the torso, the vertical coordinate of the center of gravity can be selected as the anchoring feature, which shows a regular rise and fall over time; for periodic movement movements such as walking and jogging, the vertical coordinate of the ankle joint or the gait phase indicator signal can be selected as the anchoring feature, whose waveform has a stable gait periodicity; for full-body coordination movements such as high knees and jumping jacks, the hip joint angle or the vertical velocity of the center of gravity can be selected as the anchoring feature. Once the anchoring feature is selected, the precise movement cycle length is determined by calculating its autocorrelation function over the entire time window. The autocorrelation function measures the similarity of the same signal to itself at different time delays. Let the anchoring feature sequence be... Its autocorrelation function is defined as: ; For periodic signals, the autocorrelation function has a delay. A significant peak will appear when the value is equal to or an integer multiple of the period. The frame length of the current action period, denoted as L, can be obtained by detecting the number of delayed frames corresponding to the first significant peak of the autocorrelation function excluding zero delay.

[0058] Based on the action cycle length L, a cross-cycle waveform consistency test is performed on each candidate kinematic feature, and the candidate kinematic features that pass the waveform consistency test are selected and retained. The purpose of the cross-cycle waveform consistency test is to evaluate whether a kinematic feature maintains a stable and repeatable waveform shape across different action cycles. First, according to the determined action cycle length L, the anchor feature sequence is periodically segmented. Using the cycle start point of the anchor feature (e.g., the local lowest point of the center of gravity vertical coordinate or the landing time of the ankle joint coordinate) as the boundary, the sequence within the entire time window is divided into K consecutive cycle segments. Since the cycle length may fluctuate slightly in actual movement, the number of frames in each cycle segment is not strictly equal to L. Therefore, each cycle segment needs to be resampled and uniformly interpolated to a fixed number of frames M (e.g., M=L or preset to 30 frames) to obtain a length-aligned cycle segment sequence.

[0059] For each candidate kinematic feature, segmentation and resampling are performed based on the periodic boundaries determined by the anchoring feature to obtain the aligned sequence segments of that candidate feature within each period. Using the sequence segment of the first period as a reference template, the dynamic time warping distance between the segment of each other period and the reference template is calculated. Dynamic time warping is an algorithm for measuring the similarity between two time series. It allows the sequences to bend non-linearly along the time axis to find the optimal alignment path, thus exhibiting good robustness to small changes in period length. The dynamic time warping distance between the k-th period segment and the reference template is denoted as... After obtaining all K After calculating the dynamic time warping distance between a non-reference period segment and the reference template, its average value is calculated and normalized by dividing by the signal amplitude range or standard deviation of the reference template itself to obtain the average normalized dynamic time warping distance of the candidate kinematic feature. If the average normalized dynamic time warping distance of a candidate kinematic feature is less than or equal to a preset first judgment threshold, it indicates that the waveform morphology of the feature has good consistency and repeatability across different motion cycles, and it is retained; conversely, if the distance is too large, it indicates that the waveform variation of the feature is large across cycles, and it may be affected by noise or non-motion factors, and it is not suitable to be included in the exclusive feature subset, and it is removed.

[0060] After passing the waveform consistency test, the selected candidate kinematic features are further subjected to a phase consistency test with the anchoring features. Candidate kinematic features that pass the phase consistency test are included in the feature subset corresponding to the current action pattern category and arranged in their original chronological order to form the kinematic feature time series. The purpose of the phase consistency test is to confirm whether the relative temporal relationship between the candidate kinematic features and the anchoring features remains fixed. When the human body performs coordinated movements, there are stable temporal coupling relationships between different limb segments. For example, in jogging, there should be a specific phase difference between the arm swing and the leg stride. If the phase relationship between a candidate feature and the anchoring feature drifts in different cycles, even if its waveform morphology is repetitive, the reliability of this feature in representing that action category is questionable.

[0061] The specific implementation process of the phase consistency test is as follows: for each candidate kinematic feature that passes the waveform consistency test, the instantaneous phase between the candidate kinematic feature and the anchoring feature is calculated. The instantaneous phase is obtained based on the analytic signal obtained after performing a Hilbert transform on the feature sequence. Let a certain feature sequence be... Its Hilbert transformation is Then analyze the signal Represented as: ; Where i is the imaginary unit. The argument of the analytical signal. That is, the feature at time... Instantaneous phase: ; Its value range is usually 100%. or After obtaining the instantaneous phase sequences of the candidate kinematic features and the anchoring features respectively, the cyclic variance of the difference between their instantaneous phases over the entire time window is calculated. Cyclic variance is a statistic that measures the dispersion of angle or phase distribution; its calculation must consider the circularity of the phase. Let the instantaneous phase difference between the two features in frame t be: ; First, convert each phase difference value into a complex number on the unit circle. Calculate the average vector of all complex numbers: ; The cyclic variance V is defined as: ; in The magnitude of the average vector is given. The cyclic variance ranges from 0 to 1; a smaller value indicates that the phase difference is more concentrated at a fixed value, meaning the temporal relationship between the candidate feature and the anchoring feature is more stable. Candidate kinematic features whose instantaneous phase difference cyclic variance is less than or equal to a preset second judgment threshold are judged to pass the phase consistency test. Features that pass this test indicate that they are highly synchronized with the basic rhythm of the movement in terms of temporal phase, and are reliable kinematic variables characterizing the energy consumption of the movement.

[0062] After undergoing dual screening using waveform consistency and phase consistency tests, the retained kinematic features constitute a subset of features specific to the current action mode category. These features not only exhibit statistical correlation with the action category but also pass rigorous physiological validation in terms of waveform morphology periodicity and phase sequence stability. Subsequently, the kinematic parameters in this feature subset are arranged sequentially according to the time order of the original video frames, forming a complete kinematic feature time series, which serves as the input for subsequent independent energy consumption prediction regression models. This refined feature selection process further improves the quality and consistency of input features for models specific to different action categories, providing a more reliable data foundation for high-precision energy consumption prediction.

[0063] Step S40: According to the action pattern category, the extracted kinematic feature time series is input into an independent energy consumption prediction regression model corresponding to the action pattern category. The independent energy consumption prediction regression model has a parameter space that is completely independent of the independent energy consumption prediction regression models corresponding to other action pattern categories. The independent energy consumption prediction regression model is a deep learning regression model based on a self-attention mechanism, and the method further includes: Independent energy consumption prediction regression models are pre-built and trained for various different action mode categories. The network weight parameters of each independent energy consumption prediction regression model are not shared with other independent energy consumption prediction regression models for other action mode categories during the training and inference phases.

[0064] The training process of the action pattern recognition classification model and the independent energy consumption prediction regression model further includes: Construct a raw dataset containing video streams of various motion modes and their synchronously acquired real-world physical activity intensity values; The original dataset is divided into a training set and a test set. The action pattern recognition classification model is trained using the training set until its action classification accuracy reaches the preset condition. Based on action classification, the training set data is divided into multiple subsets according to the identified action pattern categories. Each subset is then used to independently train and adjust the hyperparameters of the corresponding independent energy consumption prediction regression model until the prediction error index of each independent energy consumption prediction regression model converges.

[0065] In step S40, after extracting the associated kinematic feature time series according to the action pattern category in step S30, the kinematic feature time series is input into an independent energy consumption prediction regression model corresponding to the action pattern category. This invention constructs and trains a dedicated independent energy consumption prediction regression model for each action pattern category. The models corresponding to different action categories have completely independent parameter spaces; that is, the network weight parameters of these models are never shared during the training and inference phases. The fundamental starting point of this design is that the biomechanical work methods and energy metabolism pathways of different fitness movements are significantly different. For example, the energy consumption of squatting is mainly determined by the concentric and eccentric contractions of the lower limb muscles against gravity, while the energy consumption of jogging involves the combined effect of the elastic potential energy storage and release of the lower limb muscles and the coordinated swinging of the whole body during the periodic gait cycle. If the kinematic features of multiple movements are mixed and input into the same regression model for training, the model will be forced to fit multiple data distributions with different mapping relationships in the same parameter space, leading to feature interference and underfitting. Specifically, a predictive feature effective for one movement often becomes noise that interferes with the prediction of another movement. By constructing energy consumption prediction regression models with completely independent parameter spaces for different action categories, each model can focus on learning the mapping relationship between the kinematic characteristics and energy consumption specific to its corresponding action, thereby fundamentally avoiding feature interference across actions and significantly improving the accuracy and robustness of energy consumption prediction.

[0066] The independent energy consumption prediction regression model is a deep learning regression model based on a self-attention mechanism. In this embodiment, the model is built using a Transformer encoder architecture. Its core lies in using a multi-head self-attention mechanism to capture the global dependencies between time frames in the kinematic feature time series, so as to fully model the metabolic lag effect of human energy consumption. The metabolic lag effect refers to the fact that after the human body starts exercising or changes the intensity of exercise, the actual energy consumption does not respond immediately, but there is a delay transition process of several seconds to tens of seconds. Traditional recurrent neural networks or one-dimensional convolutional networks are limited by their local receptive fields or recursive calculation methods, making it difficult to effectively capture temporal dependencies with large spans. However, the self-attention mechanism of Transformer can directly calculate the correlation weight between any two frames in the sequence with a constant path length, thereby accurately modeling the correlation between the current motion intensity and the motion state several seconds ago.

[0067] The input to this independent energy consumption prediction regression model is the kinematic feature time series output from step S33, which has been standardized and has undergone motion-related feature filtering. The input tensor is first mapped from the feature dimension to the embedding dimension required by the Transformer encoder through a linear projection layer, and positional encoding is superimposed to preserve frame order information. The encoding layer constructs completely independent Transformer encoder networks for different motion modes. Each encoder network consists of two stacked encoder layers, each containing two main sub-layers: a multi-head self-attention layer and a feedforward neural network layer. The multi-head self-attention layer deploys four attention heads in parallel, utilizing the global receptive field to calculate the correlation weights between time frames, achieving parallel aggregation of multi-dimensional feature information. Attention calculation follows the scaling dot product attention formula: ; Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively. The dimension of the key vector, divided by As a scaling factor to prevent the dot product result from becoming too large. Function gradient saturation occurs. The multi-head attention mechanism maps the input to multiple different subspaces, calculates attention separately for each, and then concatenates the outputs of each head before linear projection, thereby enhancing the model's ability to express dependencies between different subspaces. The feedforward neural network layer contains two fully connected layers and a non-linear activation function, used to further non-linearly transform the features aggregated by self-attention. Residual connections and layer normalization operations are introduced after each sublayer to stabilize the training process of the deep network and accelerate convergence. After layer-by-layer abstraction through two encoder layers, a high-level feature representation incorporating the temporal dependencies of the entire sequence is finally obtained. This feature representation is fed into a regression head composed of fully connected layers. The regression head directly outputs the predicted value of physical activity intensity within the current time window through linear mapping by a single neuron. This predicted value, in metabolic equivalents, characterizes the average energy expenditure level of the subject within the corresponding action cycle.

[0068] To ensure good predictive performance of each independent energy consumption prediction regression model, this invention also requires the pre-establishment and training of independent energy consumption prediction regression models for various different action pattern categories. The entire model training process involves the joint optimization of the action pattern recognition classification model and multiple independent energy consumption prediction regression models.

[0069] The training process begins by constructing a raw dataset containing video streams of various movement patterns and their simultaneously acquired real-time physical activity intensity values. Subjects sequentially performed a variety of pre-defined fitness movements and daily activities in an experimental environment, including but not limited to squats, lunges, walking, jogging, high knees, and jumping jacks. Each movement lasted for a certain duration and covered different intensity levels. During data acquisition, subjects wore indirect calorimetry devices to simultaneously record real-time metabolic equivalent data per breath or per second as the gold standard label. The video streams were captured by RGB cameras positioned in front of or to the side of the subjects at a frame rate of 30 frames per second. Video data and metabolic equivalent data were aligned using timestamps to form one-to-one corresponding sample pairs. This raw dataset forms the data foundation for all subsequent model training and evaluation.

[0070] See Figure 3 After obtaining the original dataset, it is divided into training and testing sets according to a preset ratio. Using the training set data, the action pattern recognition classification model is first trained. During training, the skeleton sequence tensors obtained from the training set videos processed in steps S10 to S13 are input into the action pattern recognition classification model based on a spatiotemporal graph convolutional network. The cross-entropy loss function is used to measure the difference between the classification prediction result and the true action label. The gradient of the loss function relative to the network weights is calculated using the backpropagation algorithm, and the network parameters are updated using a stochastic gradient descent optimizer. This training process continues until the model's action classification accuracy on the validation set reaches a preset condition, indicating that the model has a reliable action category recognition ability. Training then stops, and the model parameters are saved.

[0071] After the action pattern recognition classification model is trained, it is used to infer the predicted action pattern category for each sample in the training set. Based on the predicted action pattern category, the training set is divided into multiple subsets, each containing only samples belonging to the same action category. For each action category, an independent energy consumption prediction regression model is trained independently using its corresponding subset. The deviation between the predicted metabolic equivalent and the actual metabolic equivalent is measured using the mean squared error loss function or the mean absolute error loss function. The network weights of this dedicated model are updated through backpropagation and an optimizer. Since the data distribution and sample size of subsets for different action categories may differ, each independent model requires independent hyperparameter tuning during training, including but not limited to the learning rate, training epochs, number of encoder layers, and number of attention heads, to achieve optimal convergence on its respective data subset. The training of each independent energy consumption prediction regression model is completely isolated from each other, and weight parameters are never shared. Training continues until the prediction error index of each independent model on its respective validation subset converges to a stable level, that is, the model loss value no longer decreases significantly, indicating that the model has fully learned the mapping relationship between kinematic features and energy consumption under this action.

[0072] The predictive performance of the independent energy consumption prediction regression model was evaluated in the following ways: A dataset for constructing an energy consumption prediction regression model is provided. The dataset consists of multiple samples, each of which contains a kinematic feature time series and its corresponding real physical activity intensity value. The real physical activity intensity value is the metabolic equivalent value collected synchronously by the indirect calorimetry method or the double-labeled water method. The independent energy consumption prediction regression model is used to compare the predicted value of physical activity intensity for each sample with the actual value of physical activity intensity, and the mean absolute error is calculated. The mean absolute error is used to measure the average level of the absolute deviation between the predicted value and the actual value. It is calculated by summing the absolute values ​​of the difference between the predicted value and the actual value for each sample and then dividing by the total number of samples. The mean absolute percentage error is calculated. The mean absolute percentage error is used to measure the relative error level in percentage form. It is calculated by dividing the absolute value of the difference between the predicted value and the true value of each sample by the true value of that sample, summing the results, multiplying by 100%, and dividing by the total number of samples. The root mean square error is calculated, which is used to apply a higher penalty weight to larger errors. The calculation method is to sum the squares of the differences between the predicted value and the true value of each sample, divide by the total number of samples, and then take the arithmetic square root. The coefficient of determination is calculated to assess the proportion of variation in the predicted values ​​that explains the actual values. It is calculated by subtracting the sum of squares of the differences between the predicted and actual values ​​from the sum of squares of the differences between the actual values ​​and their mean.

[0073] Specifically, in the energy consumption regression prediction stage, the dataset of the regression model can be formally represented as: .in, This represents the time series of specific motion kinematic features extracted from the i-th sample after feature engineering. The construction process of this series has been described in detail in steps S30 to S33. This refers to the actual physical activity intensity value corresponding to the sample, i.e., the gold standard value of metabolic equivalent collected simultaneously by indirect calorimetry or double-labeled water method. To comprehensively characterize the predictive accuracy of regression models for each specific action, this invention employs four quantitative indicators—mean absolute error, mean absolute percentage error, root mean square error, and coefficient of determination—to conduct a multidimensional evaluation of model performance.

[0074] Mean absolute error (MAE) measures the average level of the absolute deviation between predicted and actual values, and is defined as follows: Where n is the sample size. Let i be the true physical activity intensity value of the i-th sample. This is the predicted value of physical activity intensity output by the model for the action category to which this sample belongs. This indicator reflects the absolute magnitude of the prediction error in the same units as the original data.

[0075] Mean Absolute Percentage Error (MAPE) is calculated by introducing the true value as the denominator to the mean absolute error (MAPE), and measures the relative error level as a percentage. Its definition is: ; This indicator can eliminate the influence of numerical scale differences under different actions or different intensity levels, making it easier to compare the relative prediction accuracy of models across action categories.

[0076] The root mean square error (RMSE) is calculated by taking the arithmetic square root of the mean square of the deviations between the predicted and actual values. Larger errors are penalized with higher weights, and it is defined as follows: ； This indicator is highly sensitive to outlier predictions and can effectively reflect the model's predictive stability under extreme motion intensities.

[0077] Coefficient of determination The proportion of variation between model predictions and actual values ​​that is explained is defined as: ; in, R² is the average of the true values ​​of all samples. The value of R² is usually between 0 and 1. The closer it is to 1, the higher the goodness of fit of the model and the stronger the correlation between the predicted value and the true value.

[0078] After all independent energy consumption prediction regression models have been trained, the entire hierarchical framework is evaluated end-to-end using a test set. Test set samples are first processed by an action pattern recognition classification model to predict their action categories, and then routed to the corresponding independent energy consumption prediction regression models to output metabolic equivalent prediction values. The accuracy of energy consumption predictions is comprehensively evaluated by calculating metrics such as mean absolute error, mean absolute percentage error, root mean square error, and coefficient of determination on the test set. The test set evaluation results can also serve as a basis for further fine-tuning of the hyperparameters of each model to continuously optimize prediction performance.

[0079] Step S50: The independent energy consumption prediction regression model encodes the time-dimensional dependency of the input kinematic feature time series based on its self-attention mechanism, and outputs the predicted value of the physical activity intensity corresponding to the current action.

[0080] In this embodiment, the independent energy consumption prediction regression model encodes the time-dimensional dependency of the input kinematic feature time series based on its self-attention mechanism, and further includes: Step S51: The kinematic parameters constituting the kinematic feature time series are embedded as input and input into the network structure composed of stacked multi-layer encoders; Step S52: In each encoder layer, the correlation weights between different time frames in the input sequence are calculated through a multi-head self-attention mechanism. The calculation of the correlation weights involves the mapping query matrix, key matrix, and value matrix of the input sequence, and the product of the query matrix and the key matrix is ​​scaled based on the dimension of the key matrix. Step S53: The features aggregated by the multi-head self-attention mechanism are input into the feedforward neural network for nonlinear transformation, and residual connections are introduced at the output of each sub-layer. Step S54: The encoded features are linearly mapped through the output layer to obtain the final predicted value of body activity intensity.

[0081] The independent energy consumption prediction regression model, when processing the kinematic feature time series, captures the metabolic lag effect in the human energy consumption process through the self-attention mechanism, and is used to model the dependency between the predicted value of physical activity intensity at the current moment and the exercise state at the previous moment.

[0082] In this embodiment, the predicted value of physical activity intensity is the metabolic equivalent value.

[0083] See Figure 4In step S50, after the kinematic feature time series is input into the independent energy consumption prediction regression model corresponding to the current action mode category in step S40, the independent energy consumption prediction regression model comprehensively encodes the temporal dependencies of the input kinematic feature time series based on its built-in self-attention mechanism, and finally outputs the predicted value of the physical activity intensity corresponding to the current action. This independent energy consumption prediction regression model adopts a Transformer encoder architecture. Its core design lies in capturing the long-distance interdependence between different time frames in the kinematic feature sequence through a multi-head self-attention mechanism, thereby effectively modeling the metabolic lag effect unique to human energy consumption. The metabolic lag effect refers to the fact that after the human body starts exercising, changes the exercise intensity, or stops exercising, the actual oxygen consumption and energy metabolism do not immediately respond synchronously with the change in exercise intensity, but rather there is a physiological delay and smooth transition process of several seconds or even tens of seconds. Traditional sequence models are limited by local receptive fields or stepwise recursive computational paradigms, making it difficult to flexibly capture temporal dependencies with large spans. In contrast, self-attention mechanisms allow each time frame in the sequence to directly interact with any other time frame in the global scope. The length of its computational path is independent of the time distance, thus enabling it to accurately characterize the intrinsic physiological relationship between the predicted value of physical activity intensity at the current moment and the state of motion a few seconds ago.

[0084] The process by which the independent energy consumption prediction regression model described in this embodiment encodes the time-dimensional dependency of the input kinematic feature time series based on its self-attention mechanism includes the specific operations defined in steps S51 to S54.

[0085] In step S51, the kinematic parameters constituting the kinematic feature time series are embedded as input and fed into a network structure composed of stacked multi-layer encoders. The kinematic feature time series has already undergone action-related feature filtering and Z-score normalization in step S30, forming a multi-dimensional time series with a fixed time step. Before entering the Transformer encoder, this sequence is first mapped from the original feature dimension to the embedding dimension required by the encoder through a linear projection layer. This embedding dimension determines the capacity of the feature representation within the self-attention mechanism. Subsequently, to compensate for the self-attention mechanism's insensitivity to sequence order, positional encoding is superimposed on the input embedding. Positional encoding can employ learnable positional embedding vectors or fixed positional encoding based on sine and cosine functions. Its purpose is to enable the model to distinguish the order of occurrence of different time frames in the sequence, thereby incorporating temporal information when calculating attention weights. After the embedding and positional encoding are superimposed, the resulting feature representation is fed into a network structure composed of stacked multi-layer encoders. Considering that the input data already consists of carefully extracted low-dimensional kinematic features with high information density and moderate sequence length, a lightweight encoder layer design is adopted. Stacking two encoder layers can ensure modeling capability while avoiding overfitting and computational redundancy. Each encoder layer has the same structure but maintains its own independent parameter set. Through multi-layer stacking, feature abstraction is achieved layer by layer and deep fusion of global context is realized.

[0086] In step S52, within each encoder layer, the correlation weights between different time frames in the input sequence are calculated using a multi-head self-attention mechanism. The internal computational logic of this multi-head self-attention mechanism is consistent with the scaled dot product attention and multi-head parallel computation methods described in the preceding model structure description; here, it is explained in conjunction with the data flow of this step. Within each encoder layer, the feature representation input to that layer is first passed to the multi-head self-attention sub-layer. This sub-layer has multiple attention heads deployed in parallel. Each attention head maps the input sequence to the corresponding query, key, and value representations using an independent linear projection matrix, and performs scaled dot product attention operations within its respective subspace to generate the attention weight distribution of each time frame relative to the global sequence. The weighted aggregation results generated by each attention head are concatenated along the feature dimension and then fused through a linear transformation to obtain the output of the multi-head self-attention sub-layer. Through this mechanism, the model can simultaneously focus on multiple different types of dependencies between different positions in the sequence. For example, some attention heads may focus on capturing motion smoothness between local adjacent frames, while others may focus on detecting metabolic lag association patterns across several seconds, thus providing rich and global temporal context information for subsequent energy consumption prediction.

[0087] In step S53, the features aggregated by the multi-head self-attention mechanism are input into the feedforward neural network for nonlinear transformation, and residual connections are introduced at the output of each sub-layer. The output of the multi-head self-attention sub-layer first undergoes residual connection and layer normalization, that is, it is normalized after being added to the original input to alleviate the gradient vanishing problem in deep network training and accelerate convergence. Subsequently, the normalized features enter the feedforward neural network sub-layer. The feedforward neural network is a position-independent fully connected network, usually composed of two linear transformations and an intermediate ReLU activation function. Its role is to introduce additional nonlinear expressive power to the model, so that the global information aggregated by self-attention can be further processed into higher-level feature representations. The output of the feedforward neural network sub-layer is also normalized through residual connection and layer normalization, and is added to the input of the sub-layer to obtain the final output of the current encoder layer. This structural design, which sets residual connections and layer normalization after each sub-layer, ensures that the gradient can be effectively transmitted to earlier network layers through the identity mapping path during backpropagation, making it possible to stack multiple encoders without encountering significant performance degradation.

[0088] In step S54, the encoded features are linearly mapped through the output layer to obtain the final predicted value of physical activity intensity. After layer-by-layer feature transformation and abstraction by the two-layer encoder, the spatiotemporal dependencies in the kinematic feature time series have been fully encoded into a high-dimensional feature representation. Before being fed into the output layer, this feature representation is first subjected to global average pooling along the time dimension or the feature vector of the last time frame is selected to compress the variable-length sequence into a fixed-length feature vector. Subsequently, this feature vector is fed into a regression head composed of fully connected layers. The regression head directly outputs a scalar value through linear mapping by a single neuron. This value is the predicted value of the subject's physical activity intensity within the current time window. In this embodiment, the predicted value of physical activity intensity is expressed in metabolic equivalents (MEEs). Metabolic equivalents are an internationally recognized indicator for measuring the level of energy expenditure during physical activity. 1 MET is defined as the energy metabolic rate in a seated position. The metabolic equivalent value output by the regression head reflects the average energy expenditure intensity of the subject within the corresponding action cycle.

[0089] Through the coordinated operation of steps S51 to S54, the independent energy consumption prediction regression model fully utilizes the self-attention mechanism to model the global dependence of kinematic feature time series in the time dimension, thereby accurately capturing the metabolic lag effect in the human energy consumption process. Specifically, when the model predicts the metabolic equivalent at a certain moment, the multi-head self-attention mechanism can automatically focus on the changes in kinematic state several seconds before the current moment based on the learned attention weights, incorporating the cumulative effect of previous exercise intensity into the calculation of the current prediction. For example, when a subject suddenly starts a rapid high leg raise from a resting state, the model uses the self-attention mechanism to identify the temporal correlation between the current high leg raise feature and the resting state feature several seconds ago, thereby outputting a gradually rising rather than abrupt metabolic equivalent prediction curve. This prediction pattern is highly consistent with the actual physiological response process of the human body.

[0090] See Figure 5 The second embodiment of the present invention provides a fitness movement energy consumption prediction system based on vision and deep learning, used to execute a fitness movement energy consumption prediction method based on vision and deep learning. The system includes: The skeleton sequence construction module is used to acquire video stream data of the subject's actions, perform pose estimation on the video stream data, extract two-dimensional coordinate sequences of human key points, and construct a skeleton sequence tensor representing the dynamic changes of human pose based on the two-dimensional coordinate sequences. An action pattern recognition module is used to receive the skeleton sequence tensor and input the skeleton sequence tensor into a pre-trained action pattern recognition classification model. The action pattern recognition classification model extracts spatial structure features and temporal dynamic features from the skeleton sequence through a spatiotemporal graph convolutional network and outputs the action pattern category to which the current action belongs based on the extracted features. The feature extraction module is used to extract a kinematic feature time series associated with the current action pattern category from the skeleton sequence tensor according to the action pattern category. The kinematic feature time series contains multiple kinematic parameters with physiological significance. The model selection and input module is used to input the extracted kinematic feature time series into an independent energy consumption prediction regression model corresponding to the action pattern category, based on the action pattern category. The independent energy consumption prediction regression model has a parameter space that is completely independent of the independent energy consumption prediction regression models corresponding to other action pattern categories. The energy consumption prediction output module is used to encode the time-dimensional dependency of the input kinematic feature time series by the independent energy consumption prediction regression model based on its self-attention mechanism, and output the predicted value of the physical activity intensity corresponding to the current action.

[0091] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related explanations of the methods described above can be found in the corresponding processes in the foregoing system embodiments, and will not be repeated here.

[0092] It should be noted that the vision and deep learning-based fitness exercise energy consumption prediction system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0093] A device according to a third embodiment of the present invention includes: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement the aforementioned vision and deep learning-based fitness movement energy consumption prediction method.

[0094] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are executed by the computer to implement the above-described method for predicting fitness movement energy consumption based on vision and deep learning.

[0095] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0096] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system for implementing embodiments of the systems, methods, and electronic devices of this application. Figure 6 The server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0097] like Figure 6As shown, the computer system includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read Only Memory (ROM) 602 or programs loaded from storage section 608 into Random Access Memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0098] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0099] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0100] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0103] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0104] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for predicting energy consumption of an exercise action based on vision and deep learning, characterized in that, include: The video stream data of the subject's actions is acquired, the pose is estimated from the video stream data, the two-dimensional coordinate sequence of human key points is extracted, and a skeleton sequence tensor representing the dynamic changes of human pose is constructed based on the two-dimensional coordinate sequence. The skeleton sequence tensor is input into a pre-trained action pattern recognition classification model. The action pattern recognition classification model extracts the spatial structure features and temporal dynamic features in the skeleton sequence through a spatiotemporal graph convolutional network, and outputs the action pattern category to which the current action belongs based on the extracted features. Based on the action pattern category, extract the kinematic feature time series associated with the current action pattern category from the skeleton sequence tensor. The kinematic feature time series contains multiple kinematic parameters with physiological significance. According to the action pattern category, the extracted kinematic feature time series is input into an independent energy consumption prediction regression model corresponding to the action pattern category. The independent energy consumption prediction regression model has a parameter space that is completely independent of the independent energy consumption prediction regression models corresponding to other action pattern categories. The independent energy consumption prediction regression model encodes the time-series dependence of the input kinematic features based on its self-attention mechanism and outputs the predicted value of the physical activity intensity corresponding to the current action.

2. The method of claim 1, wherein, The construction of the skeleton sequence tensor representing the dynamic changes in human posture further includes: Perform frame-by-frame pose estimation on the video stream data to obtain the coordinates of two-dimensional key points of multiple human targets in each frame; The system uses a pre-defined main target selection mechanism to calculate the area of ​​each detected human bounding box and its distance from the center of the image, which is then used to select the main subject targets that meet the pre-defined area and position conditions. Two-dimensional key point coordinate sequences of the main subject target are extracted, and the two-dimensional key point coordinate sequences are sliced ​​by sliding window based on a preset time window length to construct a four-dimensional skeleton sequence tensor with preset batch dimension, feature channel dimension, time dimension and key point number dimension.

3. The method according to claim 1, characterized in that, The action pattern recognition classification model extracts spatial structural features and temporal dynamic features from the skeleton sequence through a spatiotemporal graph convolutional network, and further includes: The input skeleton sequence tensor is constructed into a topological graph structure of the human body, where the nodes of the topological graph structure correspond to the extracted human body key points, and the edges of the topological graph structure correspond to the natural connection relationships of the human skeleton. The topological graph structure is processed sequentially by stacking multiple spatiotemporal graph convolutional blocks, with each spatiotemporal graph convolutional block connected in series with a spatial graph convolutional module and a temporal convolutional module. The spatial graph convolution module is used to aggregate the spatial structural features between different human key points within the same video frame, and the temporal convolution module is used to extract the dynamic change features of human key point motion along the time dimension. Residual connections are introduced in each spatiotemporal graph convolution block to prevent network gradient vanishing.

4. The method according to claim 1, characterized in that, The extraction of the kinematic feature time series associated with the current action pattern category further includes: Based on the coordinate changes of key human body points in the skeleton sequence tensor, multiple original kinematic features, including center of gravity displacement, center of gravity velocity, center of gravity acceleration, angular velocity of specified joints, and gait-related parameters, are calculated. The calculated raw kinematic features are standardized to eliminate dimensional differences between different raw kinematic features. The standardization process is performed based on the mean and standard deviation of the feature set. Based on the determined action pattern category, a subset of features that are correlated with the biomechanical properties of the action pattern category and meet preset conditions are selected from multiple raw kinematic features after standardization, and these subsets are used to construct the kinematic feature time series.

5. The method according to claim 1, characterized in that, The independent energy consumption prediction regression model is a deep learning regression model based on a self-attention mechanism, and the method further includes: Independent energy consumption prediction regression models are pre-built and trained for various different action mode categories. The network weight parameters of each independent energy consumption prediction regression model are not shared with other independent energy consumption prediction regression models for other action mode categories during the training and inference phases.

6. The method according to claim 5, characterized in that, The independent energy consumption prediction regression model encodes the time-dimensional dependencies of the input kinematic feature time series based on its self-attention mechanism, and further includes: The kinematic parameters constituting the kinematic feature time series are embedded as input and fed into a network structure composed of stacked multi-layer encoders; In each encoder layer, the correlation weights between different time frames in the input sequence are calculated through a multi-head self-attention mechanism. The calculation of the correlation weights involves the mapping query matrix, key matrix, and value matrix of the input sequence, and the product of the query matrix and the key matrix is ​​scaled based on the dimension of the key matrix. The features aggregated by the multi-head self-attention mechanism are input into the feedforward neural network for nonlinear transformation, and residual connections are introduced at the output of each sub-layer. The encoded features are linearly mapped through the output layer to obtain the final predicted value of physical activity intensity.

7. The method according to claim 6, characterized in that, When processing the kinematic feature time series, the independent energy consumption prediction regression model captures the metabolic lag effect in the human energy consumption process through the self-attention mechanism, and is used to model the dependency between the predicted value of physical activity intensity at the current moment and the exercise state at the previous moment.

8. The method according to claim 1, characterized in that, The training process of the action pattern recognition classification model and the independent energy consumption prediction regression model further includes: Construct a raw dataset containing video streams of various motion modes and their synchronously acquired real-world physical activity intensity values; The original dataset is divided into a training set and a test set. The action pattern recognition classification model is trained using the training set until its action classification accuracy reaches the preset condition. Based on action classification, the training set data is divided into multiple subsets according to the identified action pattern categories. Each subset is then used to independently train and adjust the hyperparameters of the corresponding independent energy consumption prediction regression model until the prediction error index of each independent energy consumption prediction regression model converges.

9. The method according to claim 1, characterized in that, The predicted value of physical activity intensity is the metabolic equivalent value.

10. A fitness movement energy consumption prediction system based on vision and deep learning, used to execute the fitness movement energy consumption prediction method based on vision and deep learning as described in any one of claims 1-9, characterized in that, The system includes: The skeleton sequence construction module is used to acquire video stream data of the subject's actions, perform pose estimation on the video stream data, extract two-dimensional coordinate sequences of human key points, and construct a skeleton sequence tensor representing the dynamic changes of human pose based on the two-dimensional coordinate sequences. An action pattern recognition module is used to receive the skeleton sequence tensor and input the skeleton sequence tensor into a pre-trained action pattern recognition classification model. The action pattern recognition classification model extracts spatial structure features and temporal dynamic features from the skeleton sequence through a spatiotemporal graph convolutional network and outputs the action pattern category to which the current action belongs based on the extracted features. The feature extraction module is used to extract a kinematic feature time series associated with the current action pattern category from the skeleton sequence tensor according to the action pattern category. The kinematic feature time series contains multiple kinematic parameters with physiological significance. The model selection and input module is used to input the extracted kinematic feature time series into an independent energy consumption prediction regression model corresponding to the action pattern category, based on the action pattern category. The independent energy consumption prediction regression model has a parameter space that is completely independent of the independent energy consumption prediction regression models corresponding to other action pattern categories. The energy consumption prediction output module is used to encode the time-dimensional dependency of the input kinematic feature time series by the independent energy consumption prediction regression model based on its self-attention mechanism, and output the predicted value of the physical activity intensity corresponding to the current action.

Images