Mobile skill training quality evaluation method and system

Abstract

The invention provides a mobile skill training quality evaluation method and system, and relates to the technical field of mobile skill training.The method comprises the steps that a human body joint image sequence and a face video sequence of a trainee in the mobile skill training process are collected respectively; performing skeleton recognition on the human body joint image sequence to obtain a current skeleton posture and determine a current skeleton posture category; obtaining an average heart rate based on the face video sequence; comparing the current skeleton posture with a standard skeleton posture, calculating to obtain a posture similarity and a posture related error between the current skeleton posture and the standard skeleton posture, and evaluating the standard of the current skeleton posture action; performing cross-modal interaction fusion on the current skeleton posture and the average heart rate to obtain a training intensity value; and the standard and the training intensity value of the current skeleton posture are judged, so that evaluation of the mobile skill training quality is realized. According to the invention, the problems of large error and low accuracy of skill training quality evaluation at present are solved.

Description

Technical Field

[0001] This invention relates to the field of mobile skills training technology, and in particular to a method and system for evaluating the quality of mobile skills training based on the perception of body semantic cues. Background Technology

[0002] In sports such as badminton, volleyball, and table tennis, the quality of an athlete's training has a crucial impact on their subsequent skill improvement and performance. Traditional skill training quality relies heavily on the coach's subjective judgment, lacking the support of objective and accurate data. Furthermore, for most trainees who lack coaching guidance, skill training quality depends primarily on self-observation and repetitive practice. However, complex and rapid training often makes it difficult to consciously perceive and intervene, hindering the improvement of training quality.

[0003] With the rapid development of computer vision and deep learning, human pose-based monitoring methods have opened up new avenues for addressing this problem. However, existing human pose assessment methods have many drawbacks.

[0004] On the one hand, in terms of action recognition, existing human posture assessment methods have limited accuracy and struggle to precisely capture subtle changes in joint movements. For example, the subtle twisting of the fingers during a net shot, the internal rotation of the forearm during a high clear, and the high consistency of training movements inherent in sports make it difficult to distinguish between different action categories. For instance, in badminton, high clears, drop shots, and smashes may have similar initial postures, with only subtle differences in the hitting point, limb extension, and finger movements. Existing methods lack an understanding of the deep semantic features of actions, making it difficult to effectively differentiate these seemingly similar but actually quite different action categories, easily leading to confusion in action recognition.

[0005] On the other hand, when applied to training scenarios, these methods typically rely on a single human posture assessment approach without fully considering the trainee's actual physical condition. When training intensity is high, it places significant demands on the trainee's cardiopulmonary function and other capabilities. If the trainee's physical condition cannot be monitored in real time and training intensity cannot be adjusted promptly, sports injuries and other unexpected events may occur. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a method and system for assessing the quality of mobile skills training, which solves the problems of large errors and low accuracy in current skills training quality assessments.

[0007] On one hand, this invention proposes a method for evaluating the quality of mobile skills training, the method comprising: The trainees' joint images and facial video sequences were collected separately during the movement skills training process; Perform skeletal recognition on the human joint image sequence to obtain the current skeletal pose and determine the current skeletal pose category; The average heart rate was obtained based on the facial video sequence. Obtain the standard skeletal pose of the current skeletal pose category, compare the current skeletal pose with the standard skeletal pose, calculate the pose similarity and pose correlation error between the two, and evaluate the standardization of the current skeletal pose movement based on the pose similarity and pose correlation error. The training intensity value is obtained by cross-modal interaction and fusion of the current skeletal posture and the average heart rate. According to the set heart rate health standard and training level standard, the standard of the current skeletal posture and the training intensity value are judged, thereby realizing the evaluation of the quality of mobile skill training.

[0008] Furthermore, the skeletal recognition includes: The human joint image sequence is divided into body parts according to the spatial topology of the joints and the position transformation of the joints in consecutive frames, thus constructing a human joint diagram structure. By using a multi-scale spatial-temporal graph convolutional network, the spatial relationships and long-term dependencies between local and global joints in the human joint graph structure are captured, generating a set of skeletal features that includes global features of the skeleton and local features of multiple parts. Using a large language model, global text descriptions and local text descriptions of multiple parts are generated according to a set template; and the global text descriptions and local text descriptions of multiple parts are encoded by a text encoder to obtain a set of text features including global text features and local text features of multiple parts. Align the skeletal feature set and the text feature set in the spatial dimension, and perform comparative learning to obtain the current skeletal pose and determine the current skeletal pose category.

[0009] Furthermore, the comparison learning includes: Text features that match the current skeleton pose category are used as positive samples, while text features from other similar skeleton pose categories that describe the same region are used as negative samples. Calculate the cosine similarity between the skeletal features and text features of each part to capture the directional similarity between feature vectors; bring related skeletal features and text features in the feature space closer together, while removing irrelevant skeletal features and text features.

[0010] Furthermore, the comparison learning is as follows: ; in, , These represent the encoded features of the skeleton and the text, respectively. Indicates cosine similarity; Indicates temperature parameter; Indicates the sample size.

[0011] Furthermore, the average heart rate is obtained as follows: After preprocessing, the facial video sequence is input into a neural network that connects depthwise separable convolutions and two-dimensional convolutions in parallel to extract spatial features. The spatial features are input into a long short-term memory network based on a recurrent neural network for time-domain propagation to capture temporal context features and generate a remote photoplethysmography heart rate signal. After filtering and normalizing the remote photoplethysmography heart rate signal, the heart rate peak interval is obtained, and then the average heart rate is obtained.

[0012] Furthermore, the extraction of the spatial features is as follows: The preprocessed facial feature map is first subjected to depthwise convolution, and then to pointwise convolution to obtain the first feature map. The first feature map and the feature map extracted by two-dimensional convolution are fused in the channel dimension to extract spatial features.

[0013] Furthermore, the average heart rate The method of obtaining it is: ; in, This represents the interval between heart rate peaks in the remote photoplethysmography heart rate signal. Indicates the sampling frequency.

[0014] Furthermore, the pose similarity is calculated as follows: ; in, Indicates the trainee's current skeletal posture; This indicates the standard skeleton pose corresponding to the current skeleton pose.

[0015] Furthermore, the training intensity value is obtained as follows: After mapping the encoded average heart rate to the encoding space of the current skeletal pose, the shared similarity matrix between feature vectors is calculated. The shared similarity matrix is ​​normalized to obtain multiple attention maps, which are then fused to obtain a fused attention map, thereby realizing the interaction between different modalities. The shallow features of each modality are concatenated and then input into a long short-term memory network to achieve deep fusion between different modalities and obtain the deep features of each modality. The deep features of each modality are concatenated and averaged, then input into a linear classifier to obtain the training intensity value.

[0016] In a second aspect, the present invention provides a method and system for evaluating the quality of mobile skills training, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0017] In summary, this invention provides a method and system for evaluating the quality of mobile skills training. Compared with existing technologies, the technical solution conceived in this invention can achieve the following beneficial effects: (1) This invention uses the correspondence between human joint image sequence and facial video sequence to obtain the current skeletal posture and determine the current skeletal posture category and average heart rate; it performs cross-modal interaction fusion of skeletal posture and heart rate data to generate real-time training intensity value, which can effectively take into account the influence of training intensity on the standardity of training posture, monitor the physical function status of trainees in real time, so as to adjust the training intensity in a timely manner, break through the limitation that single modal data cannot reflect the action-physiological coupling relationship, avoid overtraining or insufficient intensity, reduce the occurrence of sports injuries and other accidents, not only improve the accuracy of skill training quality assessment, but also can formulate personalized training plans for trainees based on their physical function status.

[0018] (2) By analyzing the error between skeletal posture and standard movements, this invention can identify and assess the accuracy and standardization of training postures, and accurately capture subtle changes in joint movements. This solves problems such as blurred motion images and unclear differences in technical movement categories caused by high-speed movement, transforming subjective movement assessment into quantifiable objective indicators, eliminating bias in human judgment, and improving the accuracy of skill training quality assessment. In addition, by collecting human joint image sequences and facial video sequences—multimodal analysis—dual judgment of health intensity, a closed-loop system of "data collection-quality assessment-risk intervention" is formed, significantly improving training efficiency and scientific rigor.

[0019] (3) On the one hand, this invention captures the spatial relationship and long-term dependency relationship between local joints and global joints through a multi-scale spatial-temporal graph convolutional network, and models the human skeleton structure in terms of time and space dimensions, which can generate global features of human skeleton and local features of multiple parts more accurately; on the other hand, it uses a large language model as a knowledge engine to generate global text descriptions and local text descriptions of multiple parts to obtain text features; then it compares and learns with the corresponding text features, and supervises the action skeleton recognition through action semantic text, so that the details of the trainee's actions can be captured more finely, and the accurate evaluation of the trainee's actions can be achieved. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the method steps of a mobile skills training quality assessment method and system provided by the present invention; Figure 2 This is a flowchart illustrating the principle of a mobile skills training quality assessment method and system provided by the present invention. Figure 3 This is a schematic diagram of a training scenario for a mobile skills training quality assessment method and system provided by the present invention; Figure 4 This is a schematic diagram illustrating the principle of the current skeletal posture and average heart rate of the mobile skills training quality assessment method and system provided by the present invention. Figure 5 This is a schematic diagram of the text description structure of a large language model for a mobile skills training quality assessment method and system provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of additional identical elements in the method, step, or apparatus that includes the element.

[0024] In the current context of AI-enabled sports training, this invention proposes a method and system for assessing the quality of mobile skills training. It integrates skeletal posture and heart rate data across modalities to generate real-time training intensity values. This effectively considers the impact of training intensity on the standardity of training posture and monitors the trainee's physical condition in real time, allowing for timely adjustments to training intensity. This not only improves the accuracy of skills training quality assessment but also enables the development of personalized training plans based on the trainee's physical condition.

[0025] Specifically, such as Figure 1 and 2 As shown, the method includes: S100: Collects human joint image sequences and facial video sequences of trainees during the mobile skill training process.

[0026] In training scenarios, depth sensors can be used to capture sequences of human joint images of trainees performing motion training; high-definition color cameras can be used to acquire sequences of facial videos of trainees.

[0027] like Figure 3 As shown, the depth sensor and high-definition color camera are positioned within the training area, with supplementary lighting to ensure clear acquisition of human joint image sequences and facial video sequences. The depth sensor for acquiring human joint image sequences can be positioned at the back of the training area, while the high-definition color camera and supplementary lighting are positioned on either side of the center of the training area. Both instruments are positioned 2-3 meters from the net at a height of 2-3 meters, ensuring the detection range completely covers the entire space that the trainee might be involved in during training.

[0028] For example, a depth sensor with a color resolution of 1920×1080 and a depth resolution of 512×424 can be selected to capture a sequence of human joint images of the trainee during training; a high-definition color camera with a resolution of 1280×1024 and a frame rate of 60FPS can be selected to acquire a sequence of facial video sequences of the trainee during training. Simultaneously, supplementary lighting can be installed in the training area. This supplementary lighting can be a 26cm ring LED light, consisting of 160 SMD 2835 LEDs (25lm each), with a color rendering index greater than 80, a total power of 16W, and an operating voltage of 5V.

[0029] Once the depth sensor and high-definition color camera are set up, the lighting in the training area is turned on to evenly and stably illuminate the trainee's facial skin area, ensuring that the skin can fully reflect light. The depth sensor then captures images of the human joints at a pre-set frame rate, resulting in a sequence of human joint images. A high-definition color camera captures facial videos at a set frequency, and then divides the facial videos into several non-overlapping sub-video sets at certain time intervals to obtain a facial video sequence. ;

[0030] For example, according to the acquisition sequence, the facial video is automatically divided into several non-overlapping sub-video sets in 30-second intervals to obtain a facial video sequence.

[0031] S200: Perform skeletal recognition on human joint image sequences to obtain the current skeletal pose and determine the current skeletal pose category.

[0032] The acquired human joint image sequence is input into a pre-trained GAP model based on a large visual language model, employing "text supervision-multi-site contrastive learning," for skeletal recognition to obtain the current skeletal pose and determine its category. The GAP model includes skeletal recognition in both temporal and spatial dimensions; it utilizes generated global and multi-site skeletal features to perform comparative learning with corresponding text features, thereby enabling the supervision of the trainee's skeletal pose using textual information.

[0033] As an example, such as Figure 4 As shown, skeletal recognition includes: S201: The human joint image sequence is divided into body parts according to the spatial topology of the joints and the position transformation of the joints in consecutive frames, and a human joint diagram structure is constructed.

[0034] The acquired human joint image sequences are constructed into a human joint diagram structure based on the inter-frame and intra-frame connections between joints, represented as follows: .

[0035] In a sequence of human joint images, each joint is represented as a node in the graph. Based on anatomical knowledge and the spatial topology of the joints, the natural connections between them are determined, which are called spatial edges. The positional transformations of each joint in consecutive frames are then linked, forming temporal edges.

[0036] By modeling in both time and space dimensions, the human skeletal articular structure is represented as follows: ;in, , represents all joints in the human skeletal sequence; , represents the set of edges within and between frames; Indicates the number of frames. This represents the number of nodes in the human skeleton. .

[0037] S202: Captures the spatial relationships and long-term dependencies between local and global joints in the human joint graph structure through a multi-scale spatial-temporal graph convolutional network, generating a skeletal feature set that includes global features of the skeleton and local features of multiple parts.

[0038] The constructed human joint diagram structure is processed through a multi-scale spatial-temporal graph convolutional network to extract skeletal features, generating multi-site feature vectors and global feature vectors.

[0039] In the spatial domain, to determine the region of spatial graph convolution, the sampling function is defined as: ; in, Indicates from arrive Shortest path, parameters Control the spatial extent contained within the neighborhood graph.

[0040] The direct natural connections of bones in the human joint diagram can be represented by a spatial adjacency matrix. It means that among them joint and Direct connection, otherwise .

[0041] Multi-scale temporal graph convolution models long-term dependencies by extending graph convolution to the temporal dimension. In the temporal dimension, the sampling function is defined as: ; Among them, parameters Control the time scale of aggregation.

[0042] The time adjacency matrix is ​​defined as This is used to represent joint trajectories between consecutive frames. A simple time-series-based labeling function can partition the temporal adjacency matrix into... part: .

[0043] Multi-scale spatial graph convolution captures the spatial relationships between local and non-local joints by decomposing local graph convolution; skeletal features generate multiple partial feature vectors and global feature vectors through a multi-scale spatial-temporal graph convolution network with multiple layers stacked in a cross-stack manner.

[0044] Graph convolution can be represented as: ; in, Degree matrix, The adjacency matrix represents the joint connections. Indicates the first Learnable parameters of the layer It is an activation function. Global features are obtained by average pooling the features of all joint nodes, while partial features are generated by aggregating the features of different joint groups.

[0045] Based on the defined graph, refined sampling, and labeling functions, the nodes... The graph convolution operation above is formalized as: ; in, Represents the characteristics of a node. and These represent weight functions, which are based on the labels. and Assign weights from a set of weights; This indicates the number of corresponding subsets and normalizes the feature representation. and Representing nodes respectively The outputs of spatial graph convolution and temporal graph convolution.

[0046] Global features are generated by average pooling the features of all joint nodes, while local features for multiple body parts are generated by aggregating features from different joint groups. Local features for multiple body parts include skeletal feature vectors for the head, arms, hands, hips, legs, feet, and other body parts.

[0047] Average pooling can be expressed as: ; in, and This represents the height and width of the rectangular region related to the feature map. This represents the element located within the rectangular area.

[0048] S203: Using a large language model, generate global text descriptions and local text descriptions of multiple parts according to a set template; and encode the global text descriptions and local text descriptions of multiple parts through a text encoder to obtain a set of text features including global text features and local text features of multiple parts.

[0049] The API calls a pre-trained GPT-4 language model to generate text descriptions based on pre-defined templates for motion characteristics, using a question-and-answer approach. Specifically, the pre-trained GPT-4 language model generates text descriptions such as "category labels," "label synonyms," "action description paragraphs," and "local action descriptions" for different motion scenarios according to the pre-defined templates. Then, the BERT text encoder model transforms the generated text descriptions into high-dimensional text feature vectors, represented as... .

[0050] For example, during badminton training, such as Figure 5 As shown, the relevant templates for describing text can be: Scenario-based question template: "Imagine you are a professional badminton coach, and your students are practicing badminton movements of {movement category} on the badminton court. Please answer regarding the key points of this {movement category} of badminton movements." Distinguishing similar technical movement templates: "Determine whether this {movement category} belongs to a backcourt technique (high clear, smash, drop shot) or a frontcourt technique (push, hook, push shot)? And describe the differences between this {movement category} and other similar techniques in the same area in terms of joint placement." Action category tag template: "The category tag for this badminton technique is {tag}."; Tag synonym template: "Synonyms for {action category tag} include {synonym 1}, {synonym 2}, etc."; Action description paragraph template: "In badminton, {action category} is usually represented by {description}." Partial movement description template: "In the {movement category} of badminton, beginners typically {describe} six different body parts: head, arms, hands, hips, legs, and feet."

[0051] The generated text description is input into a pre-trained text encoder to capture rich linguistic features and contextual relationships to extract text features, resulting in a set of text features including global text features and local text features from multiple parts.

[0052] For example, firstly, a pre-trained Transformer-based BERT language model can be selected as the text encoder, and the generated text description can be input into the BERT language model. This process can be represented as follows: ; Then, the sentences are tokenized through word segmentation and the addition of special tokens ([cls token] for classification tasks, [SEP] for sentence segmentation). Next, a stacked Transformer module is used to capture the long-distance dependencies of all words in the sentence through a self-attention mechanism, and the output layer of the self-attention is non-linearly transformed by an activation function (ReLU function).

[0053] Finally, the global description and the local description of each part are transformed into corresponding high-dimensional text feature vectors, which are text feature sets including global text features and local text features of multiple parts.

[0054] S204: Align the skeletal feature set and the text feature set in the spatial dimension, perform comparative learning, obtain the current skeletal pose, and determine the current skeletal pose category.

[0055] In other words, by aligning the generated skeletal features of multiple body parts with their corresponding text features in the same spatial dimension and within the same batch, the model can better understand and associate the semantic information between skeletal movements and text descriptions. By calculating the similarity between each skeletal feature and text feature and comparing positive and negative samples in two directions within a batch, the model's understanding of action semantics is enhanced. Finally, a Softmax classifier is used to classify the extracted global features and determine the corresponding skeletal pose category.

[0056] As an example, the comparison learning includes: Text features that match the current skeleton pose category are used as positive samples, while text features from other similar skeleton pose categories that describe the same region are used as negative samples. Calculate the cosine similarity between the skeletal features and text features of each part to capture the directional similarity between feature vectors; bring related skeletal features and text features in the feature space closer together, while removing irrelevant skeletal features and text features.

[0057] Furthermore, comparative learning can be described as follows: ; in, , These represent the encoded features of the skeleton and the text, respectively. Indicates cosine similarity; Indicates temperature parameter; Indicates the sample size.

[0058] Finally, the extracted global features are input into the Softmax classifier. The Softmax function maps the global features to the probability space of action categories and classifies the actions according to the probability distribution of the global features. This allows the model to output the probability of each action category based on the similarity and distribution of the features, and the category with the highest output probability is determined as the input action category, which is the current skeletal pose category.

[0059] S300: Average heart rate is obtained based on facial video sequences.

[0060] Specifically, firstly, the acquired facial video sequence is preprocessed and input into the neural network. The input is a sequence of facial video frames with RGB channels. After multiple convolution and pooling operations, a multi-channel manifold is formed to represent the spatiotemporal features. Then, the latent manifold is projected onto the signal space through a 1×1 channel convolution operation, generating a manifold of length [missing information]. The remote photoplethysmogram (RPHS) heart-rate signal is obtained. After filtering and normalization, the peak heart rate interval is obtained, thus revealing the trainee's heart rate characteristics during training, i.e., the average heart rate (HR).

[0061] As an example, such as Figure 4 As shown, the average heart rate is obtained as follows: S301: After preprocessing the facial video sequence, it is input into a neural network with depthwise separable convolution and two-dimensional convolution connected in parallel to extract spatial features.

[0062] It should be noted that the preprocessing involves first cropping and locating the facial video sequence, and then dividing it into regions of interest (FRoI).

[0063] Specifically, the Viola-Jones face detector is used to crop out the facial region in the first frame of the video, and the algorithm is used to fix the face region of the trainee in subsequent frames. The facial image is then segmented into four small regions: the forehead, the cheeks, and the middle of the face, to obtain multi-frame facial images.

[0064] For example, firstly, the acquired facial video sequence is used as input. The Viola-Jones face detector is used to initially detect faces in the video and rectangular bounding boxes are used to identify face regions. Then, the Discriminative Response Map Fitting (DRMF) method is used to find the face coordinates within the face bounding boxes, and the Kanade-Lucas-Tomasi (KLT) object tracking algorithm is used to track the face frame by frame. Finally, multiple video segments of 128 frames each and 128×128 pixels in size are obtained. Each frame is further divided into four regions: the forehead, both cheeks, and the center of the face, with each region measuring 16×42 pixels.

[0065] Spatial features are extracted by inputting preprocessed multi-frame facial images into a neural network that connects depthwise separable convolutions and two-dimensional convolutions in parallel.

[0066] As an example, spatial feature extraction is as follows: The preprocessed facial feature map is first subjected to depthwise convolution, and then to pointwise convolution to obtain the first feature map. The first feature map and the feature map extracted by two-dimensional convolution are fused in the channel dimension to extract spatial features.

[0067] The depthwise convolution operation first performs a separate convolution operation on each channel of the input facial feature map to obtain multiple single-channel feature maps; then, a pointwise convolution operation is performed to concatenate all single-channel feature maps along the channel dimension and perform a convolution operation with a 1×1 convolution kernel to finally output the first feature map.

[0068] It should be noted that a two-dimensional convolutional neural network mainly consists of multiple convolutional layers and pooling layers. First, a convolutional layer with a kernel size of 5×5 and 32 output channels is used to extract local features. Then, a max-pooling layer performs downsampling on the feature map with a stride of 2×2 to reduce the spatial dimension of the feature map. After multiple convolutional and pooling operations, a multi-channel spatial feature map is formed.

[0069] Using feature fusion module The first feature maps output by the two-dimensional convolutional neural network and the channel-wise convolutional branch network are concatenated in the channel dimension, and then dimensional transformation is performed by 1×1 convolution to capture blood flow change information in different areas of the face, thereby obtaining spatial features.

[0070] S302: Spatial features are input into a long short-term memory network based on a recurrent neural network (RNN) for time-domain propagation to capture temporal context features and generate a remote photoplethysmography heart rate signal.

[0071] It should be noted that Long Short-Term Memory (LSTM) networks capture the temporal context features of facial video sequences by propagating spatial features in the temporal domain through forward / backward information flow. This process can be represented as: ; in, Represents the spatial features of the input; This represents the temporal context features of the output; Represents the spatiotemporal model used for subspace projection; This represents the set of parameters for all convolutional filters in a Long Short-Term Memory (LSTM) network. Channel aggregation represents the final signal projection; express The parameter set.

[0072] Specifically, the calculation formula for Long Short-Term Memory (LSTM) networks is as follows: ; in, Represents the multiplication and convolution operators for LSTM; It represents the Hadamardi (or Hadama) stack; , , These represent the input gate, forget gate, and output gate, respectively. Represents a memory unit; Indicates a hidden state; Indicates learnable weights; Indicates a time step; This represents the activation function; the bias term is omitted here, and all gates, memory units, hidden states, and learnable weights are three-dimensional tensors.

[0073] S303: After filtering and normalizing the remote photoplethysmography heart rate signal, the peak heart rate interval is obtained, and then the average heart rate is obtained.

[0074] After propagation in the time domain by a recurrent neural network, a latent manifold containing spatiotemporal features is obtained. Then, channel convolution is performed using a 1×1×1 convolutional layer (1×1 kernel size, 1 output channel) to project the multi-channel latent manifold onto a one-dimensional signal space, thereby generating a remote photoplethysmography (TPM) heart rate signal with a length identical to the 128 frames of the input video. This process transforms the spatiotemporal features into a one-dimensional recurrent neural network, thus completing the mapping from the facial video sequence to the heart rate signal. Finally, the TPM heart rate signal is filtered through a bandpass filter from 0.7Hz to 4Hz and normalized to obtain the heart rate peak interval, i.e., the time between adjacent pulse peaks, thus yielding the average heart rate.

[0075] It should be noted that the human heart rate ranges from 40 to 240 beats per minute, corresponding to a frequency range of approximately 0.7 Hz to 4 Hz. Therefore, a bandpass filter of 0.7 Hz to 4 Hz is used to filter the signal to eliminate low-frequency noise caused by respiration, as well as high-frequency interference caused by changes in light intensity and human movement. Next, a standard normalization method is applied to the filtered remote photoplethysmography heart rate signal to approximate the data as normally distributed data with a mean of 0 and a standard deviation of 1, thereby reducing the impact of interference on the data.

[0076] Specifically, the standard normalization method is expressed as follows: ; in, This represents the filtered RPHS signal. This represents the mean value of the filtered RPHS signal.

[0077] As an example, average heart rate The method of obtaining it is: ; in, This indicates the interval between heart rate peaks in a remote photoplethysmography (PPG) heart rate signal. Indicates the sampling frequency.

[0078] S400: Obtain the standard skeletal pose of the current skeletal pose category, compare the current skeletal pose with the standard skeletal pose, calculate the pose similarity and pose-related error between the two, and evaluate the standardization of the current skeletal pose based on the pose similarity and pose-related error.

[0079] First, pose similarity is calculated by comparing the trainee's current skeletal pose with the standard skeletal pose. Specifically, the cosine similarity between the current skeletal pose and the relative positions of each joint at different times and the standard skeletal pose is used as the pose similarity, calculated using the following formula: ; in, Indicates the trainee's current skeletal posture; This indicates the standard skeleton pose corresponding to the current skeleton pose.

[0080] Then, posture-related errors are calculated to assess the standard of training movements and the risk of injury; specifically, posture-related errors include joint position error (UMPJPE), mean joint velocity error (MPJVE), and mean joint angle error (MPJAE).

[0081] To eliminate the differences in limb length among different trainees, the average error of each joint position is determined by calculating the joint position error (UMPJPE) between the trainee's current skeletal posture and the standard skeletal posture.

[0082] Joint position error The calculation formula is: ; in, Represents skeleton The number of joints in the middle; , The estimated first The location of each joint; , Indicates the standard skeletal pose. The location of each joint; It corresponds to the length of the limb.

[0083] The average error between the predicted joint velocity and the standard joint velocity is determined by calculating the mean joint velocity error (MPJVE) between the trainee's current skeletal posture and the standard skeletal posture.

[0084] Mean joint velocity error The calculation formula is: ; in, Represents skeleton The number of joints in the middle; Indicates the number of frames; Indicates the first Frame number Standard skeletal posture velocity of each joint; Indicates the first Frame number Predicted velocity of each joint.

[0085] The angular error between joints in the skeleton is determined by the mean joint angle error (MPJAE) between the trainee's current skeletal posture and the standard skeletal posture.

[0086] Mean joint angle error The calculation formula is: ; in, Represents skeleton The number of joints in the middle; , The estimated first The angle of each joint; , Indicates the standard skeletal pose. The angle of each joint.

[0087] The standard of movement is judged by posture similarity and average joint velocity error. The presence of sports injuries such as sprained ankle, knee hyperextension, and reverse joint is determined by calculating joint position error and average joint angle error. When the joint position error is greater than ±7° and the average joint angle error is greater than ±5°, movement adjustment is required.

[0088] S500: This system performs cross-modal interaction and fusion of the current skeletal posture and average heart rate to obtain a training intensity value. Based on set heart rate health standards and training level standards, it judges the standardization of the current skeletal posture and the training intensity value, thereby assessing the quality of mobility skill training. If the standardization is satisfactory and the exercise intensity value is appropriate, the training quality is considered good.

[0089] The current skeletal pose and average heart rate are used to calculate a shared similarity matrix in different spaces through a co-attention mechanism to achieve deep feature fusion of various modalities under different modal attention, and then the motion intensity value is obtained by passing it through a linear classifier.

[0090] As an example, such as Figure 4As shown, the training intensity value is obtained as follows: S501: After mapping the encoded average heart rate to the encoding space of the current skeletal pose, calculate the shared similarity matrix between feature vectors.

[0091] The skeletal features of the current skeletal pose are aligned with the heart rate signal in the time dimension, and a collaborative attention map is obtained by utilizing the shared similarity moments between the heart rate features and skeletal features. Through a feature fusion layer based on collaborative attention, deep features of each modality under different modal attention are obtained. The deep features are input into a linear classifier to obtain the predicted values ​​of multimodal motion intensity.

[0092] Encoded heart rate features Mapping to skeletal pose The process of encoding space is as follows: ; in, and Indicates the first The trainable weights and biases of a linear projection layer, after transformation, .

[0093] Calculate the shared similarity matrix between different spaces, which reflects the relationship between the hidden vectors of the two modalities; based on the collaborative attention mechanism, capture the correlation information between different modalities by calculating the shared similarity matrix between feature vectors.

[0094] It should be noted that the attention mechanism can be represented as: ; in, These represent the query vector, key vector, and value vector, respectively. This represents the scaling factor.

[0095] S502: Normalize the shared similarity matrix to obtain multiple attention maps and fuse them to obtain a fused attention map, thereby realizing the interaction between different modalities.

[0096] In other words, the shared similarity matrix is ​​normalized and applied to the original feature vector to realize the interaction between features; multiple views are created to consider the interaction between the two modalities, and attention maps after multiple attentions are fused by averaging.

[0097] Normalize the shared similarity matrix in the row direction to obtain the skeletal features. Heart rate features extracted from each skeletal movement change attention map , represented as: ; Normalize the shared similarity matrix in the column direction to obtain the heart rate feature. The skeletal features produced by each pulsation interval within Attention map of position transformation , represented as: ; The final attention map is obtained by averaging the multiple attention maps, and is represented as follows: ; in, ， ， and This represents the scaling factor.

[0098] Based on skeletal features Heart rate features extracted from each skeletal movement change attention map To compute the attentional context of heart rate features; based on heart rate features The skeletal features produced by each pulsation interval within Attention map of position transformation To compute the context of skeletal features; a non-linear projection layer is used to project the skeletal features. Attention context mapping to initial heart rate features The encoding space is obtained based on skeletal features. Heart rate features of attention context Obtain the skeletal feature context based on heart rate feature attention. .

[0099] That is to say: ; in, , v, after transformation , .

[0100] S503: The shallow features of each modality are concatenated and then input into the Long Short-Term Memory network to achieve deep fusion between different modalities and obtain the deep features of each modality.

[0101] In other words, the contextual features and shallow features of different attention levels are horizontally concatenated. This indicates a concatenation operation. Temporal information is fused into the features using a Long Short-Term Memory (LSTM) network for encoding, achieving deep fusion of different modalities and obtaining deep features for each modality. To prevent excessive information loss during the search for common features, the original shallow features of the modality are added.

[0102] Deep fusion between different modalities can be represented as: ; in, , , This represents the feature dimension of the LSTM hidden layer.

[0103] S504: The deep features of each modality are concatenated and the average value is taken, then input into a linear classifier to obtain the training intensity value.

[0104] Through a collaborative attention feature fusion layer, deep features for each modality based on different modal attention are obtained. The deep features of each modality are concatenated and averaged row by row, and then input into the last layer (Softmax) linear classifier to obtain the predicted value of multimodal motion intensity, which is also the training intensity value.

[0105] The training intensity value can be expressed as: ; in, and This indicates the result of splicing the fused features according to the second dimension.

[0106] Finally, based on the set heart rate health standards and training level standards, the standardization of the current skeletal posture and the training intensity value are judged, thereby realizing the evaluation of the quality of mobility skill training. If the movement standard is qualified and the exercise intensity is appropriate, the training is completed.

[0107] For example, regarding badminton training, according to the "Badminton Skill Level Assessment Standard," a badminton skill level evaluation system applicable to all people established by the Chinese Badminton Association, a score of 75 or above for each technical movement is considered passing. For scores below 75, an image of the corresponding movement posture is provided, along with the areas with the most significant deviations. The coach will provide guidance and the movement will be reassessed. Based on heart rate health standards, the maximum heart rate of badminton beginners is calculated, and heart rate is categorized accordingly. When the heart rate is less than 50% of the maximum heart rate, it indicates a relatively slow and stable heart rate; when the heart rate is 50%-80% of the maximum heart rate, it indicates a normal exercise heart rate; when the heart rate is greater than 80% of the maximum heart rate, it indicates an excessively fast exercise heart rate, and exercise should be stopped and appropriate rest provided.

[0108] On the other hand, the present invention also provides a mobile skills training quality assessment system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0109] In summary, this invention utilizes the correspondence between human joint image sequences and facial video sequences to obtain the current skeletal posture and determine the current skeletal posture category and average heart rate. By performing cross-modal interactive fusion of skeletal posture and heart rate data, a real-time training intensity value is generated. This effectively takes into account the impact of training intensity on the standardity of training posture, monitors the trainee's physical function in real time, and allows for timely adjustment of training intensity. It overcomes the limitation that single-modal data cannot reflect the action-physiological coupling relationship, avoids overtraining or insufficient intensity, and reduces the occurrence of sports injuries and other accidents. This not only improves the accuracy of skill training quality assessment but also allows for the development of personalized training plans based on the trainee's physical function.

[0110] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0111] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0113] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the quality of mobile skills training, characterized in that, The method includes: The trainees' joint images and facial video sequences were collected separately during the movement skills training process; Perform skeletal recognition on the human joint image sequence to obtain the current skeletal pose and determine the current skeletal pose category; The average heart rate was obtained based on the facial video sequence. Obtain the standard skeletal pose of the current skeletal pose category, compare the current skeletal pose with the standard skeletal pose, calculate the pose similarity and pose correlation error between the two, and evaluate the standardization of the current skeletal pose movement based on the pose similarity and pose correlation error. The training intensity value is obtained by cross-modal interaction and fusion of the current skeletal posture and the average heart rate. According to the set heart rate health standard and training level standard, the standard of the current skeletal posture and the training intensity value are judged, thereby realizing the evaluation of the quality of mobile skill training.

2. The method for evaluating the quality of mobile skills training according to claim 1, characterized in that, The skeleton recognition includes: The human joint image sequence is divided into body parts according to the spatial topology of the joints and the position transformation of the joints in consecutive frames, thus constructing a human joint diagram structure. By using a multi-scale spatial-temporal graph convolutional network, the spatial relationships and long-term dependencies between local and global joints in the human joint graph structure are captured, generating a set of skeletal features that includes global features of the skeleton and local features of multiple parts. Using a large language model, global text descriptions and local text descriptions of multiple parts are generated according to a set template; and the global text descriptions and local text descriptions of multiple parts are encoded by a text encoder to obtain a set of text features including global text features and local text features of multiple parts. Align the skeletal feature set and the text feature set in the spatial dimension, and perform comparative learning to obtain the current skeletal pose and determine the current skeletal pose category.

3. The mobile skills training quality assessment method according to claim 2, characterized in that, The comparison learning includes: Text features that match the current skeleton pose category are used as positive samples, while text features from other similar skeleton pose categories that describe the same region are used as negative samples. Calculate the cosine similarity between the skeletal features and text features of each part to capture the directional similarity between feature vectors; bring related skeletal features and text features in the feature space closer together, while removing irrelevant skeletal features and text features.

4. The mobile skills training quality assessment method according to claim 3, characterized in that, The comparison learning is as follows: ； in, , These represent the encoded features of the skeleton and the text, respectively. Indicates cosine similarity; Indicates temperature parameter; Indicates the sample size.

5. The mobile skills training quality assessment method according to claim 1, characterized in that, The average heart rate is obtained as follows: After preprocessing, the facial video sequence is input into a neural network that connects depthwise separable convolutions and two-dimensional convolutions in parallel to extract spatial features. The spatial features are input into a long short-term memory network based on a recurrent neural network for time-domain propagation to capture temporal context features and generate a remote photoplethysmography heart rate signal. After filtering and normalizing the remote photoplethysmography heart rate signal, the heart rate peak interval is obtained, and then the average heart rate is obtained.

6. The mobile skills training quality assessment method according to claim 5, characterized in that, The extraction of the spatial features is as follows: The preprocessed facial feature map is first subjected to depthwise convolution, and then to pointwise convolution to obtain the first feature map. The first feature map and the feature map extracted by two-dimensional convolution are fused in the channel dimension to extract spatial features.

7. The method for evaluating the quality of mobile skills training according to claim 5, characterized in that, The average heart rate The method of obtaining it is: ； in, This represents the interval between heart rate peaks in the remote photoplethysmography heart rate signal. Indicates the sampling frequency.

8. The method for evaluating the quality of mobile skills training according to claim 1, characterized in that, The pose similarity is calculated as follows: ； in, Indicates the trainee's current skeletal posture; This indicates the standard skeleton pose corresponding to the current skeleton pose.

9. The method for evaluating the quality of mobile skills training according to claim 1, characterized in that, The training intensity value is obtained as follows: After mapping the encoded average heart rate to the encoding space of the current skeletal pose, the shared similarity matrix between feature vectors is calculated. The shared similarity matrix is ​​normalized to obtain multiple attention maps, which are then fused to obtain a fused attention map, thereby realizing the interaction between different modalities. The shallow features of each modality are concatenated and then input into a long short-term memory network to achieve deep fusion between different modalities and obtain the deep features of each modality. The deep features of each modality are concatenated and averaged, then input into a linear classifier to obtain the training intensity value.

10. A mobile skills training quality assessment system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images