Human action feature symbol sequence encoding method and action scoring method

Abstract

The application relates to the technical field of image recognition, in particular to a human action feature symbol sequence coding method and an action scoring method. The symbol sequence coding method comprises the following steps: acquiring continuous video frames of a human action, extracting a coordinate sequence of a human key point in each frame, calculating displacement vectors of the same key point between adjacent frames, and filtering invalid displacements according to a preset displacement threshold; symbol coding is performed on the direction of the valid displacement, continuous motion directions are mapped into discrete symbols; and the symbols corresponding to each key point are spliced in time sequence to generate the action feature symbol sequence. The application organically integrates posture estimation, skeleton behavior recognition, action classification and action scoring, realizes lossless information transmission and joint optimization, avoids error accumulation and information loss caused by module fragmentation, initiates an action DNA sequence scoring mechanism, realizes fine analysis of the action, and realizes fine scoring through sequence comparison.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and more specifically, to a method for encoding human motion feature symbol sequences and a motion scoring method. Background Technology

[0002] Human motion capture and recognition technology is an important research direction in computer vision and artificial intelligence, widely used in film and television production, game development, sports training, medical rehabilitation, human-computer interaction, and many other fields. The development of this technology has evolved from wearable devices to vision-based methods. Vision-based motion capture technology has made breakthrough progress in recent years, enabling contactless human pose estimation using ordinary RGB cameras. 2D pose estimation models such as OpenPose, HRNet, and MediaPipe can detect the coordinates of key points on the human body from a single frame image, achieving a preliminary perception of human pose. With the development of deep learning technology, the YOLO-Pose series of models has achieved significant breakthroughs in balancing accuracy and inference speed. YOLO11x-Pose, a pose estimation model launched by Ultralytics, adopts a new backbone network and neck architecture design. By introducing an improved C3k2 module and C2PSA mechanism, it can more accurately capture complex human poses and maintain the stability of key point tracking in occluded environments, making it suitable for various scenarios from edge computing to offline analysis tasks.

[0003] As a representative of traditional Chinese fitness exercises, Baduanjin (Eight Pieces of Brocade) is characterized by its gentle movements, focused mind, and coordinated breathing, requiring a high degree of adherence to correct form. Traditional Baduanjin teaching and assessment often rely on on-site manual scoring, which suffers from low efficiency, strong subjectivity, and a lack of real-time feedback. Many practitioners, when learning independently or in small groups, struggle to receive effective personalized guidance, potentially impacting training effectiveness and even leading to injuries due to incorrect posture. With the rise of national fitness and intelligent health management, developing a convenient, accurate, and interpretable intelligent recognition and assessment system for Baduanjin movements has become a research hotspot.

[0004] In recent years, some progress has been made in intelligent assessment technology for Baduanjin (Eight Pieces of Brocade). Patent application CN119049127A discloses a Baduanjin teaching system based on human posture recognition, including a user terminal and a server terminal. The user terminal collects video recordings of the movement state and sends them to the server terminal. The server terminal performs three-dimensional motion state reconstruction, compares the differences in the positions of key human body points between standard Baduanjin movements and the user's movements, and generates a personalized Baduanjin teaching report. This solution achieves basic movement comparison and report generation functions, but it does not use movement continuity and coordination as independent assessment indicators, and there is still room for improvement in the depth of assessment of dynamic characteristics. Patent application CN121459426A discloses an artificial intelligence-based Baduanjin movement detail capture and comprehensive evaluation system, proposing a method for evaluating the continuity of Baduanjin based on fractal analysis. This system first acquires the human posture skeleton diagram for each frame of a Baduanjin (Eight Pieces of Brocade) practice video and annotates the movements. Based on the movement stage of each frame's human posture skeleton diagram within the corresponding movement, and the fractal dimension of the movement trajectory of each key node in the local time domain, it obtains the single-frame stuttering coefficient for each frame's human posture skeleton diagram, thereby obtaining the Baduanjin's continuity coefficient. Finally, it combines the movement standardization of each frame's human posture skeleton diagram to comprehensively evaluate Baduanjin. This method innovatively introduces fractal analysis to evaluate movement smoothness and quantifies the Baduanjin's continuity coefficient, representing the latest technological advancement in this field. However, although this method is based on the human posture skeleton diagram (including the connection relationships of basic key points), it does not fully explore and utilize human topological information (such as joint linkage and limb coordination constraints); moreover, the continuity coefficient and movement standardization are only combined using a simple weighted summation method, and it does not specify the specific location method for movement deviations, lacking the ability to finely locate movement deviations.

[0005] Existing technologies suffer from simplistic and vague scoring mechanisms, neglecting dynamic features, exhibiting poor interpretability, and lacking refined quantitative indicators. Consequently, they fail to achieve precise and interpretable quality assessments of traditional martial arts movements. Therefore, there is an urgent need to develop a motion recognition and scoring method based on existing technologies that can overcome these shortcomings and achieve end-to-end intelligent analysis from video input to refined scoring. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a first-ever combination of YOLO11x-Pose high-precision pose estimation, ST-GCN spatiotemporal feature modeling, and action DNA sequence scoring mechanism, achieving end-to-end intelligent parsing from video input to refined scoring, thus overcoming many shortcomings of existing technologies.

[0007] Specifically, this is achieved through the following technical solution: In a first aspect, this application provides a method for encoding human action feature symbol sequences, comprising the following steps: Acquire continuous video frames of human motion, extract the coordinate sequence of human key points in each frame, calculate the displacement vector of the same key point between adjacent frames, and filter invalid displacements according to a preset displacement threshold. The direction of the effective displacement is symbolically encoded, mapping the continuous motion direction to discrete symbols; The action feature symbol sequence is generated by concatenating the symbols corresponding to each key point in chronological order.

[0008] Furthermore, the number of key human body points is ≥17, and the key human body points include upper limb, lower limb and trunk nodes; Furthermore, the step of extracting the coordinate sequence of human key points in each frame includes: inputting the video frame into a pre-trained pose estimation model and outputting the two-dimensional coordinates and confidence score of each key point in the image coordinate system. Furthermore, if the confidence score of a key point is lower than 0.5, it is considered unreliable and will be repaired in subsequent processing by linear interpolation between the preceding and following frames; if the confidence score is low for multiple consecutive frames, the data segment will be discarded and the user will be prompted to re-record.

[0009] Furthermore, the pose estimation model is YOLO11x-Pose; Further, the step of extracting the coordinate sequence of human body key points in each frame and calculating the displacement vector of the same key point between adjacent frames includes: for each frame t (t=1,2,…,T-1) and each key point i (i=1,2,…,17, corresponding to 17 human body key points), calculating the displacement vector (Δx,Δy)=(x t+1,i -x t,i ,y t+1,i -y t,i ); Furthermore, the step of filtering invalid displacements according to a preset displacement threshold includes: setting a displacement threshold δ, decomposing the displacement vector of the key point into a horizontal displacement component Δx and a vertical displacement component Δy; if both |Δx| and |Δy| are less than δ, then it is determined to be an invalid displacement and is removed; the method for removing it is not to record it. Furthermore, the displacement threshold δ is set to n% of the image's horizontal width and vertical height. Furthermore, the symbol encoding of the direction of the effective displacement includes: decomposing the displacement vector of the effective displacement into a horizontal displacement component Δx and a vertical displacement component Δy. When |Δx|>|Δy| and Δx>0, it is encoded as the first symbol; When |Δx|>|Δy| and Δx<0, the encoding is the second symbol; When |Δy|≥|Δx| and Δy>0 is positive, it is encoded as the third symbol; When |Δy|≥|Δx| and Δy<0, it is encoded as the fourth symbol; Among them, the first to fourth symbols are different characters or numbers, representing the four main directions of movement: right, left, up, and down, respectively; Furthermore, the first to fourth symbols are represented by C, T, A, and G, respectively, to form a symbol sequence resembling the DNA base arrangement; Further, the step of assembling the symbols corresponding to each key point in chronological order to generate the action feature symbol sequence includes: for each key point i, assembling all valid symbols encoded in chronological order (from frame 1 to frame T-1) to form the action symbol string Si for that key point, with a length equal to the number of valid motion frames; the final user's overall action DNA sequence is composed of symbol strings corresponding to 17 key points, denoted as {S}. user,1 ,S user,2 ,…,S user,17 This data is used for subsequent detailed comparison and scoring.

[0010] Secondly, this application provides a motion scoring method based on human motion feature symbol sequences, including the following steps: For standard movements, a standard movement feature symbol sequence is constructed based on the above-mentioned encoding method for human movement feature symbol sequences, and stored in the standard movement feature symbol sequence library; Based on the above-mentioned encoding method for human action feature symbol sequences, a user action feature symbol sequence is constructed for the user's actions. A sequence alignment algorithm is used to compare the user's action feature symbol sequence with the standard action feature symbol sequence and calculate the similarity score. Based on the similarity score, generate an action score and action repair suggestions; Furthermore, both the standard action feature symbol sequence and the user's action feature symbol sequence include sub-sequences of the upper limbs, lower limbs, and trunk; Furthermore, before employing the sequence alignment algorithm, the corresponding action category in the user's practice video is identified based on the pre-trained skeleton action recognition model, and the corresponding standard action feature symbol sequence is retrieved from the standard action feature symbol sequence library. Furthermore, the skeleton motion recognition model is ST-GCN; Furthermore, the skeleton motion recognition model is pre-trained using standard motion data; Furthermore, the sequence alignment algorithm is the Needleman-Wunsch global sequence alignment algorithm; Furthermore, the comparison between the user action feature symbol sequence and the standard action feature symbol sequence involves comparing each subsequence of the user action feature symbol sequence with the corresponding subsequence of the standard action feature symbol sequence.

[0011] Furthermore, the similarity score is calculated by defining a matching score, a mismatch penalty, and a gap penalty, and then calculating the optimal alignment path and the similarity score for each part. Furthermore, the step of generating an action score based on the similarity score includes: normalizing the similarity scores of each subsequence; and weighting and summing the normalized subsequence scores according to preset subsequence weights to obtain the action score. Furthermore, the action repair suggestion is based on the mismatches and gaps that appear in the comparison path to locate the key human body points and action primitives of the user's action deviation, in order to generate a visualized "action DNA comparison report"; the action primitives include action DNA subsequences and their scores. Thirdly, this application provides an encoding system for human motion feature symbol sequences, comprising: The motion direction calculation unit is used to calculate the displacement vector of the same human key points between adjacent frames of continuous video frames; The direction determination and encoding unit is used to encode the direction of the effective displacement with symbols, mapping the continuous motion direction into discrete symbols; The sequence generation unit is used to concatenate the symbols corresponding to each key point in chronological order to generate the human motion feature symbol sequence.

[0012] Fourthly, this application provides a human motion scoring system, including a memory, a processor, and a computer program stored in the memory. When the processor executes the program, it implements the above-mentioned motion scoring method based on human motion feature symbol sequences.

[0013] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described action scoring method based on human action feature symbol sequences.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention provides a method for encoding human motion feature symbols and a method for motion scoring, aiming to automatically extract human key points from videos, identify motion categories, and provide refined scoring of motion quality. It organically integrates pose estimation, skeleton behavior recognition, motion classification, and motion scoring to achieve lossless information transfer and joint optimization, avoiding error accumulation and information loss caused by module fragmentation. Furthermore, each module can be jointly trained within a unified framework, maximizing the overall performance of the system.

[0015] 2. It pioneered an action DNA sequence scoring mechanism to achieve refined action analysis. It decomposes the actions of people in videos into "action DNA sequences" and achieves refined scoring through sequence comparison. It can accurately locate the time period of deviation, key points of the human body and action primitives, and refine the scoring granularity. It draws on DNA sequence comparison algorithms, and can accurately show which action the user's sequence and the standard sequence erroneous, just like comparing gene sequences. It can integrate sequence similarity and difference comparison to achieve multi-dimensional evaluation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the sequence coding and scoring method for this application. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application. To better understand the technical solutions of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0019] like Figure 1 As shown, the present invention provides an encoding method and a scoring method for human action feature symbol sequences. In one embodiment, the specific method steps are as follows: I. Overall Technical Process 1. Video Acquisition and Preprocessing Module: Acquires videos of users practicing Baduanjin (Eight Pieces of Brocade) using a standard RGB camera, and performs preprocessing operations such as size normalization and noise reduction on the video frames.

[0020] 2. Key point extraction module based on YOLO11x-Pose: Input the preprocessed video frames into the YOLO11x-Pose pose estimation model, and output the two-dimensional coordinates and confidence scores of 17 key points of the human body frame by frame.

[0021] 3. Spatiotemporal feature extraction and action classification module based on ST-GCN: The key point sequence of continuous frames is constructed into a spatiotemporal graph, which is input into the spatiotemporal graph convolutional network (ST-GCN) to extract spatiotemporal features. The class probability of the current action is output through a fully connected layer to identify the Baduanjin moves that the user is practicing (such as "Holding up the sky with both hands to regulate the three jiaos" and "Drawing the bow left and right like shooting an eagle").

[0022] 4. Action DNA Sequence Encoding Module: Processes the key point sequences of consecutive frames, converts the motion trajectories of different body parts of the user into "action DNA sequences" (such as A, T, C, G) according to rules, and obtains the user's action DNA sequence.

[0023] 5. Refined scoring module based on sequence alignment: Based on the action category identified by ST-GCN, the corresponding standard action DNA sequence is retrieved from the standard action DNA sequence library. The user's action DNA sequence is independently compared with the standard sequence globally for each key point, and the similarity score of each key point is calculated. Finally, based on the preset key point weights, the scores of each key point are weighted and summed to obtain the comprehensive score of the user's action.

[0024] 6. Visual Feedback Module: Generates an "Action DNA Comparison Report" that includes action category, total score, scores for each key point, deviation location, and correction suggestions, presented to the user in a graphic and textual format.

[0025] II. Keypoint Extraction Module Based on YOLO11x-Pose 1. Model Structure YOLO11x-Pose is the latest pose estimation model in the YOLO series from Ultralytics. Its backbone network adopts an improved CSPDarknet structure, introducing the C3k2 module (a cross-stage local module that integrates attention mechanisms) and the C2PSA mechanism (a parallel combination of channel attention and spatial attention), which improves keypoint localization accuracy while maintaining real-time inference speed. The model outputs the bounding box and coordinates of 17 keypoints for each detected human instance (following the keypoint definitions in the COCO dataset: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle).

[0026] 2. Input / Output Input: An RGB image with dimensions of 640×640×3 (scalable to maintain aspect ratio, with gray fill for any insufficient areas).

[0027] Output: For each frame, output N human instances (N is usually 1, i.e., the main trainee). Each instance contains the coordinates (x, y) of 17 keypoints, a confidence score c (0~1), and bounding box coordinates. If multiple people are detected, the main trainee can be selected by location or size (e.g., selecting the human body in the center of the image with the largest height).

[0028] 3. Specific implementation parameters Input frame rate: To ensure the continuity of the action, it is recommended to collect a frame rate of ≥30fps.

[0029] Confidence threshold: Key points with a confidence level below 0.5 are considered unreliable and will be repaired in subsequent processing by linear interpolation between consecutive frames; if multiple consecutive frames have low confidence levels, the data segment will be discarded and the user will be prompted to re-record.

[0030] III. Spatiotemporal Feature Extraction and Action Classification Module Based on ST-GCN This module takes 17 human keypoints (based on the COCO / MPII skeleton definition) from T consecutive frames as input, extracts the spatiotemporal features of human movements by constructing a spatiotemporal graph, and uses a stacked spatiotemporal graph convolutional network (ST-GCN) to classify the eight movements of Baduanjin. The overall design balances computational efficiency and the integrity of movement features, and is optimized for the slow-paced and low-dynamic characteristics of Baduanjin.

[0031] 1. Spatiotemporal graph construction Will The 17 key points of the frame are used to construct a spatiotemporal graph. : Node set : Include Each node Indicates the first The first frame Several key points. To balance the computational efficiency of the basic version with the expressive power of the advanced version, node features adopt scalable multi-dimensional normalized features:

[0032] in: , Key point coordinates via image width and height Normalization eliminates the influence of shooting scale; Key point confidence (raw detection output) improves the robustness of low-confidence nodes; , The key point is speed in the time dimension ( ), capturing the dynamic sequence of actions.

[0033] The base version can be retained only. To reduce computational load, an advanced version stacks velocity features to improve temporal modeling capabilities. This embodiment preferably uses normalized coordinates. As input, it can be flexibly extended in the feature input layer.

[0034] Edge set : Divided into two categories: spatial and temporal, comprehensively covering the spatial relationships of the human skeleton and the temporal relationships of movements: Spatial edge: in the same frame Internally, based on the predefined adjacency matrix of the human natural skeleton topology. , Indicate key points and Physical connections exist (e.g., head-neck, left shoulder-left elbow, left elbow-left wrist, etc., strictly matching the 17 keypoint skeleton connection rules). Additionally, spatial distance edges can be selectively added: weak connections can be set for non-skeleton-connected keypoints within the same frame that are spatially close (e.g., left and right hands, left and right feet). This enhances the modeling of the overall spatial features of the human body.

[0035] Time edge: Same key point In adjacent frames and Building strong connections (adjacency matrix) , ). Optional supplementary cross-frame time edges (e.g.) and ), used to capture the long temporal dependencies of slow-motion Eight-Section Brocade, the adjacency weight can decay with the frame interval (e.g. ).

[0036] 2. ST-GCN Network Structure The ST-GCN core is composed of multiple spatiotemporal convolutional blocks stacked together. Each block contains two branches: spatial graph convolution and temporal convolution, supplemented by residual connections, batch normalization (BN), and dropout to adapt to the slow pace and low dynamic characteristics of Baduanjin movements, while preventing overfitting.

[0037] (1) Spatial graph convolution For input feature tensors ( For the number of channels, For frame number, To address the insufficient expressive power of a single adjacency matrix, a multi-adjacency subset normalized graph convolution is employed (to determine the number of key points).

[0038] in: (Multi-subset strategy): Divide the adjacency matrix into 3 subsets—self-connected subsets. Centripetal subset (Neighbors pointing to the centroid of the skeleton), centrifugal subset (Neighbors far from the skeleton's centroid), each subset corresponds to an independently learnable weight matrix. ; For the identity matrix, self-connection is introduced to avoid loss of node features; For degree matrix, It is used for adjacency matrix normalization to solve the gradient explosion / vanishing problem; The LeakyReLU activation function (as an alternative to ReLU) is used, with a slope of 0.2 to alleviate gradient sparsity.

[0039] (2) Temporal convolution Spatial convolution output The temporal dimension performs standard one-dimensional convolution to capture temporal dependencies between adjacent frames. To adapt to the slow motion of Baduanjin (a traditional Chinese exercise) and reduce the number of parameters, the convolution kernel size is set to... (Optimized to be reduced from 9), stride = 1, padding = 3 (keeping the temporal length unchanged). Each convolutional time step is followed by a BN layer and dropout (dropout rate = 0.2) to reduce the risk of overfitting.

[0040] (3) Network configuration Input layer: The shape of the input tensor is ,in (Normalized coordinates) ). Set according to the actual frame rate (e.g., 150 frames).

[0041] Hidden layers: Nine spatiotemporal convolutional blocks are stacked, with the number of output channels in each block being 64, 64, 128, 128, 256, 256, 512, 512, and 512 respectively. Each block contains both spatial graph convolutions and temporal convolutions, and residual connections are used (to compensate for differences in the number of input and output channels). (Convolution adjustment).

[0042] Global average pooling: Average pooling is performed on both the time dimension and the node dimension to obtain a 512-dimensional feature vector.

[0043] Fully connected classification layer: The output dimension is 8 (corresponding to the 8 moves of Baduanjin), followed by the Softmax function to obtain the probability of each category.

[0044] 3. Action Classification Output and Optimization Strategies Based on the probability distribution of Softmax output, classification logic is designed to improve the practicality and robustness of Baduanjin movement recognition.

[0045] Basic classification rules: The category corresponding to the maximum probability output by Softmax is taken as the recognition result.

[0046] Preset probability threshold (Optimized and increased from 0.7 to better suit the low dynamic characteristics of the Eight Pieces of Brocade movements), if the maximum value If the output is unclear, the message "Action unclear" will be displayed, along with the prompt "Keep the action stable / Re-record".

[0047] Output format: Successful identification: Returns "Move Name + Match Probability" (e.g., "Holding Up the Sky with Both Hands to Regulate the Three Burners: 0.82").

[0048] Recognition failed: Returns "Action unclear / Move switching" + guidance prompt (such as "Please complete the move slowly and keep your body stable").

[0049] IV. Action DNA Sequence Encoding Module The core objective of the action DNA sequence encoding module is to transform continuous keypoint motion trajectories into discrete symbol sequences, enabling subsequent refined scoring using bioinformatics sequence alignment algorithms. This module borrows from gene sequence expression methods, encoding the movements of different parts of the human body as "bases" (such as A, T, C, G), thereby constructing a quantifiable and traceable action representation.

[0050] 1. Module Input and Output enter: The keypoint sequence extracted by YOLO11x-Pose is in the form of a T×17×2 tensor (T is the number of frames, 17 keypoints, and each point has (x,y) coordinates).

[0051] It also receives the current action category (such as "holding up the sky with both hands to regulate the three jiaos") identified by the ST-GCN module, so as to call the corresponding standard sequence.

[0052] Output: The user action DNA sequence Suser is a set of 17 symbol strings generated independently for each key point.

[0053] 2. Action DNA Sequence Mapping Rules Given a keypoint sequence P={Pt,i|t=1..T,i=1..17} of length T frames, where Pt,i=(Xt,i,Yt,i) are the coordinates of the keypoints, the mapping rule converts the trajectory of each keypoint into a symbol string through the following steps: Motion direction calculation: For each keypoint i, calculate the displacement vector (Δx,Δy) = (Xt+1,i-Xt,i,Yt+1,i-Yt,i) between adjacent frames. Set a displacement threshold δ (e.g., 1% of the image width and height, i.e., 0.01) to filter out minor jitters. Movements below the displacement threshold δ are considered stationary and are not included in the DNA sequence.

[0054] (1) Direction determination and coding: Encoding based on the direction of main motion: If |Δx|>|Δy| and Δx>0, the encoding is C (right shift). If |Δx|>|Δy| and Δx<0, the encoding is T (left shift). If |Δy|≥|Δx| and Δy>0, the encoding is G (shift down). If |Δy|≥|Δx| and Δy<0, the encoding is A (shifted up). (2) Sequence generation: For each frame t, a symbol is generated for each keypoint i. These symbols are concatenated in chronological order to obtain the symbol string Si for each keypoint (length T-1, derived from inter-frame shifts). The final user action DNA sequence consists of 17 symbol strings, denoted as {Suser,i}.

[0055] Symbol mapping table:

[0056] 3. Construction of a standard action DNA sequence library The standard movement DNA sequence for each move of Baduanjin is established, and the specific process is as follows: (1) Expert video collection: Several Baduanjin experts were invited to record standard demonstration videos in a controlled environment to ensure that the movements are standardized and the perspective is fixed.

[0057] (2) Key point extraction: The YOLO11x-Pose model was used to extract the coordinates of 17 key points in each frame of the video.

[0058] (3) Action DNA Sequence Extraction: According to the mapping rules, the key point coordinates of the expert video are converted into symbol strings for each key point. For data from multiple experts, the final standard sequence Sstd,i can be obtained by majority voting or averaging the trajectory before encoding. If the duration of the experts' actions is inconsistent, the trajectory needs to be time-normalized (e.g., interpolated to a fixed number of frames) before encoding.

[0059] 4. Generation of User Action DNA Sequences When a user practices, their video is captured and the same steps as those used in building the standard library are performed to obtain the user action DNA sequence {Suser,i}.

[0060] V. Refined Scoring Module Based on Sequence Alignment This module draws on sequence alignment algorithms from bioinformatics to compare user sequences with standard sequences, calculate similarity scores, and locate differences.

[0061] 1. Algorithm selection This embodiment employs the Needleman-Wunsch global alignment algorithm because it can perform a globally optimal match between two complete sequences, making it suitable for evaluating overall action quality. For cases where length discrepancies arise due to differences in action speed, the algorithm automatically handles these discrepancies by introducing a gap penalty.

[0062] 2. Parameter Settings Define the scoring matrix: Match score: +2 points for matching the same symbol.

[0063] Mismatch penalty: -1 point is awarded for a mismatch between different symbols.

[0064] Gap penalty: -1 point is awarded for inserting or deleting a symbol.

[0065] 3. Comparison process Assuming a standard sequence Length is User sequence Length is Build Score matrix ,initialization The first row and first column are filled with gap penalty points. , Then recursively fill the matrix:

[0066] in like ,otherwise .

[0067] The optimal comparison path is obtained by backtracking. If a mismatch (diagonal movement but different signs) or a gap (vertical / horizontal movement) occurs in the path, the corresponding action deviation position is determined.

[0068] 4. Key Point Score Calculation and Normalization For each key point Perform the above comparison independently to obtain the optimal comparison score for that point. To ensure the score remains stable within the [0, 100] interval, minimum-maximum normalization is used: Highest score in computational theory Assuming all matching positions are matched, the remaining length is filled with gaps, i.e. (because Simplified to ).

[0069] Calculate the lowest theoretical score Assuming all positions are mismatched or empty, i.e. .

[0070] Normalized score: If (Meaningless scenario), then ;otherwise:

[0071] 5. Overall score calculation for the action Based on the characteristics of the Eight Pieces of Brocade movements, different weights are assigned to each key point (this can be obtained through expert scoring or data analysis). The following is an example weight allocation; adjustments can be made according to specific needs in actual application:

[0072] The overall score for the action is the weighted sum of the normalized scores for each key point:

[0073] 6. Visual Feedback Module The final "Motion DNA Comparison Report" displays the motion category, total score, scores for each key point, deviation locations (such as mismatches or gaps), corresponding correction suggestions, and a score distribution trend chart in a graphical format. This helps users intuitively understand motion quality issues and make improvements.

[0074] In some embodiments, besides YOLO11x-Pose, the present invention can employ other mature 2D pose estimation models to extract human key points, for example: OpenPose: Based on a multi-branch convolutional network, it can simultaneously detect key points of the human body, hands, and face, making it suitable for multi-person scenarios.

[0075] HRNet maintains high-resolution representation through parallel multi-resolution subnets, resulting in high accuracy in keypoint localization, making it particularly suitable for fine motion analysis.

[0076] MediaPipe: A lightweight pose estimation solution from Google that supports real-time inference on mobile devices and is easy to deploy on portable devices such as smartphones.

[0077] ViTPose: A pose estimation model based on the Transformer architecture, which uses a self-attention mechanism to capture global dependencies and is more robust in occluded scenarios.

[0078] If a binocular or depth camera is used, 3D pose estimation models (such as VideoPose3D and PoseFormer) can be used to directly extract the coordinates of 3D key points. The 3D information can more accurately reflect the spatial motion trajectory and further improve the accuracy of DNA encoding.

[0079] In some embodiments, for key points with low confidence, in addition to linear interpolation repair, the following methods can also be used: Kalman filtering to predict the location of missing points; discarding low-confidence frames and encoding only valid frames (while maintaining time alignment); and introducing an attention mechanism to adaptively reduce the weight of low-confidence nodes in ST-GCN.

[0080] In some embodiments, in addition to the standard ST-GCN, the following improved graph convolutional networks may be used: 2s-AGCN: A two-stream adaptive graph convolutional network that learns the implicit dependencies between joints through an adaptive adjacency matrix, thereby improving robustness to individual differences.

[0081] MS-G3D: Multi-scale Spatiotemporal Graph Convolutional Network, which uses multi-scale graph convolution to capture joint dependencies at different ranges, while combining 3D convolution to enhance temporal modeling capabilities.

[0082] DSTA-Net: A dynamic spatiotemporal attention network that combines graph convolution and Transformer to dynamically aggregate spatiotemporal information through a self-attention mechanism.

[0083] CTR-GCN: Channel-level topology refinement graph convolution learns different spatial topologies in the channel dimension, enhancing feature representation capabilities.

[0084] If the graph structure is not explicitly constructed, the Transformer and its variants can be used to directly process the sequence of key points, for example: PoseFormer: Embeds keypoint coordinates as tokens and models global dependencies using a spatiotemporal Transformer encoder.

[0085] TimeSformer: Introduces segmented spatiotemporal attention in video understanding, which can be transferred to skeleton sequence processing.

[0086] In some embodiments, in addition to the Softmax algorithm outputting the maximum probability, the classification strategy may also employ: Multi-label classification: When an action may contain transition states, output multiple possible moves and their probabilities.

[0087] Temporal segmentation classification: The long video is divided into multiple action segments, classified separately, and then the results are merged.

[0088] In some embodiments, in addition to the Needleman-Wunsch global comparison, the following may also be used: Smith-Waterman local comparison: suitable for scenarios where only local action quality is of concern, such as finding the most similar parts of a user's actions.

[0089] Edit distance (Levenshtein distance): Calculates the minimum number of edit operations required to transform one sequence into another, and normalizes it as a score.

[0090] Multiple sequence alignment: Simultaneously compares user sequences with multiple standard templates (such as different expert versions) and selects the best match.

[0091] In some embodiments, the matching score, mismatch penalty, and gap penalty can be dynamically adjusted, for example: Different matching scores are assigned to different primitives based on the importance of the action.

[0092] Introducing affine gap penalties (gap open and gap extend) to more realistically simulate the cost of action insertion / absence.

[0093] In some embodiments, keypoint weights are not limited to expert scoring, but can also be determined through: data-driven: automatically determining weights based on correlation analysis (such as PCA, regression coefficients) between keypoints in training data and the final score; learnable weights: introducing an attention mechanism into the scoring network to allow the model to automatically learn the contribution of each keypoint to the overall score; and move-adaptive weights: different weight allocations are used for different moves, and the corresponding weight vectors are dynamically called based on the ST-GCN classification results.

[0094] In some embodiments, the overall score may not be limited to weighted summation, but may also employ: geometric mean, harmonic mean; comprehensive evaluation based on fuzzy logic; or machine learning regression model to directly predict the total score based on the comparative features of each key point.

[0095] In some embodiments, the movements are not limited to Baduanjin (Eight Pieces of Brocade) and can be extended to other traditional fitness exercises (such as Tai Chi, Wuqinxi, and Yijinjing) and sports training (such as gymnastics, yoga, and dance) for intelligent evaluation. The human body standard movement evaluation method and system proposed in this invention can be widely used in home and community entertainment, professional sports competitions, and online teaching.

[0096] Embodiments of this application also provide an encoding system for human motion feature symbol sequences, including: The motion direction calculation unit is used to calculate the displacement vector of the same human key point between adjacent frames of consecutive video frames; The direction determination and encoding unit is used to encode the direction of the effective displacement with symbols, mapping the continuous motion direction into discrete symbols; The sequence generation unit is used to concatenate the symbols corresponding to each key point in chronological order to generate the human motion feature symbol sequence.

[0097] Embodiments of this application also provide a human motion assessment system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the scoring method described in the above embodiments.

[0098] Embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scoring method described in the above embodiments.

[0099] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for encoding human motion feature symbol sequences, characterized in that, Including the following steps: Acquire continuous video frames of human motion, extract the coordinate sequence of human key points in each frame, calculate the displacement vector of the same key point between adjacent frames, and filter invalid displacements according to a preset displacement threshold. The direction of the effective displacement is symbolically encoded, mapping the continuous motion direction to discrete symbols; By arranging the symbols corresponding to each key point in chronological order, a sequence of human motion feature symbols is generated.

2. The human motion feature symbol sequence encoding method according to claim 1, characterized in that, The step of extracting the coordinate sequence of human key points in each frame includes: inputting the video frame into a pre-trained pose estimation model and outputting the two-dimensional coordinates of each key point in the image coordinate system.

3. The human motion feature symbol sequence encoding method according to claim 1 or 2, characterized in that, The step of filtering invalid displacements according to a preset displacement threshold includes: setting a displacement threshold δ, decomposing the displacement vector of the key point into a horizontal displacement component Δx and a vertical displacement component Δy, and if both |Δx| and |Δy| are less than δ, then the displacement is determined to be invalid and is removed.

4. The human motion feature symbol sequence encoding method according to claim 3, characterized in that, The symbol encoding of the direction of the effective displacement includes: decomposing the displacement vector of the effective displacement into a horizontal displacement component Δx and a vertical displacement component Δy; if |Δx|>|Δy| and Δx>0, then it is encoded as a first symbol; if |Δx|>|Δy| and Δx<0, then it is encoded as a second symbol; if |Δy|≥|Δx| and Δy>0, then it is encoded as a third symbol; if |Δy|≥|Δx| and Δy<0, then it is encoded as a fourth symbol.

5. A motion scoring method based on human motion feature symbol sequences, characterized in that, Including the following steps: For standard actions, according to the encoding method of any one of claims 1-4, a standard action feature symbol sequence is constructed and stored; For user actions, a user action feature symbol sequence is constructed according to the encoding method described in any one of claims 1-4; A sequence alignment algorithm is used to compare the user's action feature symbol sequence with the standard action feature symbol sequence and calculate the similarity score. Based on the location and type of symbol differences in the sequence alignment results and the similarity score, corresponding joint movement correction suggestions are generated.

6. The action scoring method according to claim 5, characterized in that, Both the standard action feature symbol sequence and the user action feature symbol sequence include sub-sequences of the upper limbs, lower limbs, and trunk.

7. The action scoring method according to claim 5 or 6, characterized in that, Before using the sequence alignment algorithm, the user action category in the user action video is identified based on the pre-trained skeleton action recognition model, and the corresponding standard action feature symbol sequence is retrieved from the standard action feature symbol sequence library.

8. The action scoring method according to claim 6, characterized in that, The user action feature symbol sequence is compared with the standard action feature symbol sequence, and each subsequence of the user action feature symbol sequence is compared with the corresponding subsequence of the standard action feature symbol sequence.

9. A human motion scoring system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the program, it implements the scoring method according to any one of claims 5-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the scoring method according to any one of claims 5-8.

Images