A method and system for visual capture and evaluation of human motion

Abstract

The application provides a human action visual capture and evaluation method and system, wherein the method comprises: performing video frame decomposition on a human action video to obtain a plurality of single-frame images; using a first Visual Transformer model to detect each single-frame image to obtain the 2D human joint position of each single-frame image; generating the 2D human skeleton corresponding to each single-frame image according to the 2D human joint position of each single-frame image; using a Transformer decoder to process the 2D human skeleton corresponding to each single-frame image to obtain the 3D human posture data corresponding to each single-frame image; obtaining the 3D human model corresponding to each single-frame image according to the 3D human posture data; and comparing the 3D human model corresponding to each single-frame image with a set standard action to obtain a human action score. The application can improve the accuracy of human action evaluation.

Description

Technical Field

[0001] This application relates to the fields of visual analysis and human motion recognition technology, and in particular to a method and system for visual capture and evaluation of human motion. Background Technology

[0002] Existing intelligent evaluation methods for human motion generally score the accuracy of human movements based on the location of joints. For example, they use convolutional neural networks (CNNs) to extract 2D joint positions and perform simple pose analysis based on these positions to obtain evaluation results. However, this method has low accuracy in joint position detection and lacks consideration for pose deviation analysis, so the overall accuracy of the evaluation needs to be improved.

[0003] Therefore, how to improve the accuracy of human motion evaluation is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method for visually capturing and evaluating human movements, which can improve the accuracy of movement evaluation. This application also provides a system for visually capturing and evaluating human movements, achieving the same technical effect.

[0005] The first objective of this application is to provide a method for visual capture and evaluation of human movements.

[0006] The aforementioned objective of this application is achieved through the following technical solution:

[0007] A method for visually capturing and evaluating human movements, comprising:

[0008] The human motion video to be evaluated is acquired, and the human motion video is decomposed into video frames to obtain multiple single-frame images.

[0009] Obtain a pre-built first Visual Transformer model, wherein the first Visual Transformer model is configured to detect 2D human joint positions from the single-frame image;

[0010] Using the first Visual Transformer model, each of the single-frame images is detected to obtain the 2D human joint positions of each single-frame image;

[0011] Based on the 2D human joint positions of each single frame image, a 2D human skeleton corresponding to each single frame image is generated.

[0012] Obtain a pre-built Transformer decoder, wherein the Transformer decoder is configured to predict 3D human pose data as input to the 2D human skeleton;

[0013] Using the Transformer decoder, the 2D human skeleton corresponding to each single frame image is processed to obtain the 3D human pose data corresponding to each single frame image.

[0014] Based on the 3D human pose data corresponding to each single frame image, 3D human reconstruction is performed to obtain a 3D human model corresponding to each single frame image.

[0015] The 3D human body model corresponding to each single frame image is compared with the set standard action to obtain the comparison result, and a human body action score is obtained based on the comparison result.

[0016] Preferably, in the visual capture and evaluation method for human body movements, the 3D human posture data includes 3D human joint positions, human joint rotation angles, movement amplitude, speed characteristics, and overall body posture.

[0017] Preferably, in the visual capture and evaluation method for human motion, the step of performing 3D human reconstruction based on the 3D human pose data corresponding to each single-frame image to obtain a 3D human model corresponding to each single-frame image includes:

[0018] Based on the 3D human pose data corresponding to each single frame image, 3D human reconstruction is performed using the SMPL model to obtain a 3D human model corresponding to each single frame image.

[0019] Preferably, the method for visually capturing and evaluating human movements further includes:

[0020] Obtain a pre-built MediaPipe Hands model, wherein the MediaPipe Hands model is configured to detect 2D hand joint positions from the single-frame image;

[0021] Using the MediaPipe Hands model, the 2D hand joint positions of each frame image are detected to obtain the 2D hand joint positions of each frame image.

[0022] Obtain a pre-built second Visual Transformer model, wherein the second Visual Transformer model is configured to predict the hand rotation angle based on the 2D hand joint position as input;

[0023] Using the second Visual Transformer model, the 2D hand joint position corresponding to each single frame image is processed to obtain the hand rotation angle corresponding to each single frame image;

[0024] The 2D hand joint position and hand rotation angle corresponding to each single frame image are compared with the set standard hand gesture to obtain the hand gesture comparison result, and a hand gesture score is obtained based on the hand gesture comparison result.

[0025] Preferably, the method for visually capturing and evaluating human movements further includes:

[0026] Obtain the joint position corresponding to each of the single-frame images, wherein the joint position is the 2D human joint position or the 3D human joint position;

[0027] Obtain a pre-built comment generation model, wherein the comment generation model is configured to take the joint position as input and output human action comments, and the comment generation model adopts an end-to-end Transformer architecture, including: an embedding layer, an encoder position encoder, an encoder, a decoder position encoder, a decoder and an output layer;

[0028] Using the aforementioned comment generation model, human motion comments are generated based on the joint positions corresponding to each single-frame image.

[0029] Preferably, in the visual capture and evaluation method for human motion, the embedding layer is used for:

[0030] For each of the single-frame images, the joint positions are flattened to obtain flattened coordinate vectors. ,in, This indicates the number of frames in the single-frame image;

[0031] According to the flattened coordinate vector Each joint Corresponding coordinate vector With each joint Corresponding learnable embedding vectors Perform element-wise multiplication to obtain the first embedding vector. ;

[0032] According to the flattened coordinate vector Each joint Corresponding coordinate vector Perform 1D convolution to obtain the second embedding vector. ;

[0033] According to the first embedding vector and the second embedding vector The gating coefficient was calculated. The calculation formula is:

[0034] ;

[0035] In the formula, This represents the activation function. Represents the weight matrix. Indicates the bias term;

[0036] According to the first embedding vector The second embedding vector and the gate coefficient The calculation yields each joint in each of the single-frame images. The corresponding final embedding vector The calculation formula is:

[0037] .

[0038] Preferably, in the visual capture and evaluation method for human motion, the encoder position encoder is used for:

[0039] Rotational position encoding and relative position encoding are used for each of the final embedding vectors. Perform position encoding to obtain the corresponding first position code;

[0040] The encoder is used for:

[0041] For each joint in each of the single-frame images The corresponding final embedding vector The first feature is obtained by processing the corresponding first position code;

[0042] The encoder includes an M-layer encoder layer, wherein the encoder layers from layer 1 to layer L employ a local self-attention mechanism, and the encoder layers from layer L+1 to layer M employ a global multi-head self-attention mechanism.

[0043] Preferably, in the visual capture and evaluation method for human motion, the decoder position encoder is used for:

[0044] By employing rotational position encoding and relative position encoding, position encoding is performed on each of the first features to obtain the corresponding second position encoding;

[0045] The decoder comprises N decoder layers, wherein each decoder layer includes a masked self-attention layer, a cross-attention layer, and a feedforward neural network. The decoder is used for:

[0046] Global average pooling is performed on the first feature and the corresponding second position encoding to obtain the action intent vector;

[0047] Using the mask self-attention layer and the feedforward neural network in the decoder layer of the first K layers, a comment outline skeleton is generated based on the action intent vector and the historical output of the decoder.

[0048] By utilizing the masked self-attention layer, the cross-attention layer, and the feedforward neural network in the decoder layer of the post-NK layer, the comment outline skeleton and the first feature are fused through gating fusion to obtain the output vector of the decoder.

[0049] Preferably, in the visual capture and evaluation method for human motion, the output layer is used to map the output vector of the decoder into human motion comments.

[0050] The second objective of this application is to provide a visual capture and evaluation system for human motion.

[0051] The second objective of this application is achieved through the following technical solution:

[0052] A visual capture and evaluation system for human motion, comprising:

[0053] The first acquisition unit is used to acquire the human motion video to be evaluated, and to decompose the human motion video into video frames to obtain multiple single-frame images.

[0054] The second acquisition unit is used to acquire a pre-built first Visual Transformer model, wherein the first Visual Transformer model is configured to detect 2D human joint positions from the single-frame image;

[0055] The detection unit is used to detect each of the single-frame images using the first Visual Transformer model to obtain the 2D human joint positions of each single-frame image.

[0056] The generation unit is used to generate a 2D human skeleton corresponding to each single frame image based on the 2D human joint positions of each single frame image.

[0057] The third acquisition unit is used to acquire a pre-built Transformer decoder, wherein the Transformer decoder is configured to predict 3D human pose data with the 2D human skeleton as input.

[0058] The processing unit is used to process the 2D human skeleton corresponding to each single frame image using the Transformer decoder to obtain 3D human pose data corresponding to each single frame image.

[0059] The reconstruction unit is used to perform 3D human body reconstruction based on the 3D human body pose data corresponding to each single frame image, so as to obtain a 3D human body model corresponding to each single frame image.

[0060] The evaluation unit is used to compare the 3D human body model corresponding to each single frame image with the set standard action, obtain the comparison result, and obtain the human body action score based on the comparison result.

[0061] The above technical solution acquires a human motion video to be evaluated, decomposes the human motion video into multiple single-frame images, and performs visual capture based on the single-frame images to extract human joint data. Specifically, it acquires a pre-constructed first Visual Transformer model and uses the first Visual Transformer model to detect each single-frame image to obtain the 2D human joint position of each single-frame image. The first Visual Transformer model is configured to detect the 2D human joint position from the single-frame image. By applying the first Visual Transformer model, the position of key points of the human body can still be accurately captured even in fast movements, occlusion, or blurry conditions. Furthermore, based on the 2D human joint positions of each single-frame image, a 2D human skeleton is generated for each single-frame image. A pre-built Transformer decoder is then acquired and used to process the 2D human skeleton for each single-frame image, obtaining 3D human pose data for each single-frame image. Based on the 3D human pose data for each single-frame image, 3D human reconstruction is performed to obtain a 3D human model for each single-frame image. The Transformer decoder is configured to predict 3D human pose data using the 2D human skeleton as input. Based on the 3D human pose data predicted by the Transformer decoder, an accurate 3D human model can be generated. Further, the 3D human model for each single-frame image is compared with a set standard action to obtain the comparison results. By comparing frame by frame, the deviation between each frame image and the standard action can be detected. Finally, based on the comparison results, an accurate human action score can be obtained. In summary, the above technical solution can improve the accuracy of human action evaluation. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart illustrating a method for visually capturing and evaluating human movements provided in an embodiment of this application.

[0064] Figure 2 This is an application diagram illustrating the mobile phone recording scenario provided in the embodiments of this application;

[0065] Figure 3 This is a schematic diagram illustrating the application of XR glasses in the embodiments of this application;

[0066] Figure 4 This is an application diagram of the dance studio camera scene provided in the embodiments of this application;

[0067] Figure 5 This is a schematic diagram of the structure of a visual capture and evaluation system for human motion provided in an embodiment of this application. Detailed Implementation

[0068] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are merely illustrative. For example, the division of units and modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or modules can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0070] In addition, each functional unit in the various embodiments of this application can be integrated into a single processor, or each unit can be a separate device, or two or more units can be integrated into a single device; each functional unit in the various embodiments of this application can be implemented in hardware or in the form of hardware plus software functional units.

[0071] Those skilled in the art will understand that all or part of the steps of the following method embodiments can be implemented by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, they perform the steps of the following method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0072] It should be understood that the use of terms such as "system," "device," "unit," and / or "module" in this application is merely one method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0073] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0074] If a flowchart is used in this application, it is used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0075] It should also be noted that, in this document, terms such as “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the article or device that includes the aforementioned element.

[0076] The embodiments in this application are written in a progressive manner.

[0077] like Figure 1 As shown in the figure, this application provides a method for visual capture and evaluation of human movements, including:

[0078] S101. Obtain the human motion video to be evaluated, and decompose the human motion video into video frames to obtain multiple single-frame images;

[0079] In step S101, specifically, a video recording device can be used to capture the human body movements of the target object, obtaining a video of the human body movements to be evaluated. Then, the video of the human body movements to be evaluated is decomposed into multiple single-frame images, which can be used for subsequent feature extraction and comparison. Different application scenarios can correspond to different target objects. For example, in a construction operation scenario, the target object can be construction workers; in a technical training scenario, the target object can be trainees; and in a dance training scenario, the target object can be dancers learning dance. This application is not limited to these examples.

[0080] S102. Obtain the pre-built first Visual Transformer model;

[0081] In S102, specifically, Visual Transformer (ViT) is a vision model based on the Transformer architecture, proposed by the Google team in 2020. ViT segments an image into fixed-size patches, each patch being converted into a vector representation and used as an input sequence. These vectors are processed by a Transformer encoder, utilizing a self-attention mechanism to capture contextual dependencies at different locations in the image. Finally, specific vision tasks are accomplished by classifying or regressing the encoder output.

[0082] The first Visual Transformer model is configured to detect 2D human joint positions from a single frame image. These 2D joint positions can be the coordinates of 2D human joints, such as the 2D coordinates of the head, shoulder, elbow, wrist, hip, knee, and ankle. The first Visual Transformer model can be pre-trained using a large number of image samples and existing model training methods; this application does not impose specific limitations on this.

[0083] Due to the rapid and complex pose changes in human movements, the first Visual Transformer model was designed with dynamic adaptability to different human movements and complex poses in mind. Specifically, the first Visual Transformer model divides a single frame image into fixed-size patches (such as 16x16 pixel patches) and inputs each patch as an independent input unit into the Transformer structure for processing. Each patch is embedded in a high-dimensional vector space, mapped to a specific dimension through a linear transformation, and then encoded with position information, enabling the first Visual Transformer model to understand the positional relationships of each patch. Furthermore, the self-attention mechanism of the first Visual Transformer model helps it capture the dependencies between patch images, allowing the model to detect key points even when parts of the human body are occluded, based on image context and known structural features. For example, if the arm is occluded by the torso, the first Visual Transformer model can infer the approximate position of the wrist based on the relative positions of the shoulder and elbow. For blurred images during rapid motion, the attention mechanism of the First Visual Transformer model associates different parts of an image patch, inferring the actual joint positions based on the positions of surrounding joints and the overall skeleton. In this way, the First Visual Transformer model maintains stable detection performance when handling high-speed human motion. Therefore, applying the First Visual Transformer model can accurately capture the positions of key points on the human body even in fast motion, occlusion, or blurred conditions.

[0084] S103. Using the first Visual Transformer model, detect each single-frame image to obtain the 2D human joint position of each single-frame image;

[0085] In S103, specifically, each single-frame image is input into the pre-built first Visual Transformer model to obtain the 2D human joint positions of each single-frame image, so that a 2D skeleton can be generated based on the 2D human joint positions.

[0086] S104. Generate a 2D human skeleton corresponding to each single frame image based on the 2D human joint positions of each single frame image.

[0087] In S104, specifically, the 2D human joint position can be the coordinates of the 2D human joint. Based on the coordinates of the 2D human joint in each single frame image, the 2D human skeleton corresponding to each single frame image can be generated. The 2D human skeleton is used to connect the human joints, forming the relative position of the human joints and the overall structure of the human body, which is used for subsequent 3D pose reconstruction.

[0088] In some embodiments, a preset 2D skeleton generation algorithm can be used to generate a 2D human skeleton for each single frame image based on the 2D human joint positions. Specifically, the 2D skeleton generation algorithm can employ the existing OpenPose algorithm. This algorithm calculates the relative positions between adjacent joints in each single frame image based on the 2D human joint positions, thereby determining the relative angles and positional relationships between the joints. In each single frame image, the skeleton structure is corrected based on the relative proportions and angles of the skeleton connections to maintain a natural human form, ultimately establishing a complete 2D human skeleton. This allows for accurate construction of the target object's pose even during rapid movements, deflections, or body occlusion.

[0089] S105. Obtain the pre-built Transformer decoder;

[0090] In S105, specifically, the Transformer's decoder is mainly composed of multiple identical decoder layers stacked together. Each layer contains three main sub-layers: a masked multi-head self-attention layer, an encoder-decoder attention layer, and a feed-forward network (FFN). These sub-layers are connected through residual connections and layer normalization to ensure efficient information transfer and model stability.

[0091] The Transformer decoder is configured to predict 3D human pose data from a 2D human skeleton. The Transformer decoder can be pre-trained using existing model training methods based on prior joint structures and training data; this application does not impose specific limitations on this. Using the Transformer decoder, 2D joint information can be mapped to 3D space to obtain 3D human pose data.

[0092] In some embodiments, the training process of the Transformer decoder includes:

[0093] First, a dataset containing a 2D human skeleton and its corresponding 3D pose data was obtained. The 2D human skeleton in the dataset has been standardized to ensure that the model can maintain good generalization ability under different image resolutions and human body sizes. To enhance the robustness of the model, the dataset underwent data augmentation, including operations such as rotation, scaling, and horizontal flipping of joint positions.

[0094] Then, the initial Transformer decoder is trained using the dataset to obtain a trained Transformer decoder. In terms of model architecture, the Transformer decoder adopts a standard multi-layer self-attention mechanism, with each layer containing multiple attention heads to capture the relationships between different joints. The input data for the Transformer decoder is a 2D human skeleton, and spatial information of the joints is preserved through position encoding. Based on the 2D human skeleton and pre-trained joint structures and angle priors, the Transformer decoder reconstructs 3D pose and outputs 3D human pose data. To ensure that the output 3D human pose data conforms to the natural structure of the human skeleton, the model embeds spatial relationships and angle priors between joints during training. The training process uses the Adam optimizer and the Mean Squared Error (MSE) as the loss function, aiming to minimize the difference between the 2D joint position predictions and the true 3D pose. During training, the batch size can be set to a predetermined value, and the model's performance is evaluated through cross-validation after each training epoch. In the evaluation process, metrics such as 3D joint reconstruction error are mainly used to quantify the model's prediction accuracy. Ultimately, after multiple rounds of training, the Transformer decoder can accurately predict the corresponding 3D human pose data based on the input 2D human skeleton, and can maintain high accuracy and robustness, especially in dynamic environments or complex scenes.

[0095] In some embodiments, 3D human posture data may specifically include 3D human joint positions and joint rotation angles. For example, 3D human joint positions may include the 3D coordinate positions of the head, shoulders, elbows, wrists, hips, knees, and ankles, and joint rotation angles may include the pitch, internal / external rotation, and other angles of the shoulder and knee joints. This allows for the subsequent comparison of the 3D human model constructed based on the 3D human joint positions and rotation angles with a set standard movement, detection of deviations in joint positions and rotation angles, and calculation of a human movement score based on these deviations.

[0096] In other embodiments, 3D human posture data also includes motion amplitude, velocity features, and overall body posture. Specifically, using a Transformer decoder, the motion amplitude and velocity features of the target object in 3D space are extracted from the 2D human skeleton to construct a 3D human model for comparison with standard movements. For example, in a rapid kicking motion, the Transformer decoder can capture changes in the angular velocity of the leg, thereby retrieving the velocity deviation during subsequent comparison and calculating a human motion score based on the velocity deviation. Using the Transformer decoder, the overall posture of the target object, including the body's center of gravity, balance, and coordination, is analyzed based on the 2D human skeleton to obtain overall body posture information, which is used to construct a 3D human model for comparison with standard movements.

[0097] To predict motion amplitude, velocity features, and overall body posture, the Transformer decoder, during training, not only learns the mapping from 2D human skeleton to 3D joint positions but also improves prediction accuracy by incorporating motion features and velocity information. Specifically, during training, the Transformer decoder introduces time-series information and the rate of change of joint motion into the input, enabling the model to capture the motion amplitude and velocity features of the joints. Furthermore, during training, the Transformer decoder utilizes prior knowledge of the relative positions and angles between joints to predict the overall body posture. By using a multi-task joint optimization objective function (including 3D joint position error, joint motion velocity error, and posture constraints based on biomechanical priors), the relationship between motion amplitude, velocity features, and overall body posture is optimized, allowing the Transformer decoder to accurately predict these dynamic features from the 2D human skeleton. During training, the Transformer decoder continuously adjusts model parameters to minimize prediction errors and achieves accurate estimation of motion amplitude and velocity features by comparing posture changes at different time steps. Through the Transformer decoder, richer 3D human posture data can be obtained from the 2D human skeleton, enabling the construction of more accurate 3D human models.

[0098] S106. Using the Transformer decoder, process the 2D human skeleton corresponding to each single frame image to obtain the 3D human pose data corresponding to each single frame image.

[0099] In S106, specifically, the 2D human skeleton corresponding to each single frame image is input into the pre-built Transformer decoder to obtain the 3D human pose data corresponding to each single frame image, so as to build a 3D human model based on the 3D human pose data.

[0100] S107. Based on the 3D human pose data corresponding to each single frame image, perform 3D human reconstruction to obtain the 3D human model corresponding to each single frame image.

[0101] In S107, specifically, existing 3D human body reconstruction methods can be used to perform 3D human body reconstruction based on the 3D human body pose data corresponding to each single frame image, thereby obtaining a 3D human body model corresponding to each single frame image, which can then be compared with standard actions.

[0102] In some embodiments, one implementation of this step specifically includes: using the SMPL model to reconstruct the 3D human body based on the 3D human pose data corresponding to each single frame image, thereby obtaining the 3D human body model corresponding to each single frame image.

[0103] Specifically, the SMPL (Skinned Multi-Person Linear Model) is a statistical model used for 3D human body modeling and motion capture, capable of describing the posture and shape of the human body. Using 3D human posture data, the SMPL model can be driven to reconstruct the 3D human body, outputting an accurate 3D human body model. For example, using the SMPL model, modeling and optimization can be performed based on the 3D human body joint positions and rotation angles from the 3D human posture data. By minimizing posture and skeletal errors, accurate joint angles can be obtained, thereby constructing a precise 3D human body model.

[0104] S108. Compare the 3D human body model corresponding to each single frame image with the set standard action to obtain the comparison result, and obtain the human body action score based on the comparison result.

[0105] In S108, specifically, the 3D human body model corresponding to each single frame image is compared frame by frame with the set standard action to obtain the comparison result corresponding to each single frame image. Based on the comparison result and the preset scoring rules, the human body action score can be obtained.

[0106] For example, a 3D human body model is compared with a set standard movement to detect joint position deviations and joint rotation angle deviations. Based on these deviations, a human movement score is calculated according to preset scoring rules. Joint position deviations can be measured using Euclidean distance, while joint rotation angle deviations can be measured using angular differences or quaternion distances. For each joint deviation, including joint position and rotation angle deviations, it can be determined whether the deviation is within the allowable error range. If the deviation exceeds the allowable error range, the movement is considered inaccurate. According to the preset scoring rules, different weights can be pre-assigned to each joint deviation, and the overall human movement score is calculated by combining the deviations of all joints. A specific rule could be: deducting a set score when the deviation of each joint exceeds the allowable error range, with the final human movement score being a weighted average of the joint scores; this application is not limited to this.

[0107] In the construction operation scenario, for example, the standard movements can be a pre-set sequence of standard movements based on construction videos that meet safety operation specifications. Correspondingly, the human movement scores obtained based on the above implementation steps can help construction workers assess whether their construction movements meet safety operation specifications, reminding them to make timely improvements. Similarly, in the technical training scenario, the standard movements can be a pre-set sequence of standard movements based on videos of technicians with strict movement requirements. Correspondingly, the human movement scores obtained based on the above implementation steps can help trainees assess whether their technical movements meet technical training requirements, facilitating timely problem identification. In the dance training scenario, the standard movements can be a pre-set sequence of standard movements based on dance videos of dance instructors. These movements can be provided by the dancers or use dance tutorials or templates from dance teaching software. Correspondingly, the human movement scores obtained based on the above implementation steps can help dancers assess the gap between their dance movements and those of the dance instructor, contributing to improved dance skills. This application does not impose specific limitations on these aspects.

[0108] In some embodiments, prompts can be output based on joint position deviations and joint rotation angle deviations. For example, warnings can be issued at points with significant joint and limb position deviations, such as "left shoulder deviation" or "insufficient arm angle," to mark the non-standard movement parts and help the target subject make focused adjustments. In other embodiments, the human movement video to be evaluated can be segmented by time or structure and compared with preset standard movements to obtain segmented human movement scores, providing an accurate evaluation of each movement segment.

[0109] Existing intelligent evaluation methods for human motion generally score the accuracy of human movements based on the location of joints. For example, they extract 2D joint positions using convolutional neural networks and perform simple posture analysis based on these positions to obtain evaluation results. However, this method has low accuracy in joint position detection and lacks consideration for posture deviation analysis, so the overall accuracy of the evaluation needs to be improved.

[0110] The above embodiments involve acquiring a human motion video to be evaluated, decomposing the human motion video into multiple single-frame images, and performing visual capture based on the single-frame images to extract human joint data. Specifically, a pre-constructed first Visual Transformer model is acquired, and the first Visual Transformer model is used to detect each single-frame image to obtain the 2D human joint position of each single-frame image. The first Visual Transformer model is configured to detect 2D human joint positions from single-frame images. By applying the first Visual Transformer model, the positions of key points of the human body can still be accurately captured even in fast movements, occlusion, or blurry conditions. Furthermore, based on the 2D human joint positions of each single-frame image, a 2D human skeleton corresponding to each single-frame image is generated. A pre-built Transformer decoder is obtained, and the 2D human skeleton corresponding to each single-frame image is processed using the Transformer decoder to obtain 3D human pose data corresponding to each single-frame image. Based on the 3D human pose data corresponding to each single-frame image, 3D human reconstruction is performed to obtain a 3D human model corresponding to each single-frame image. The Transformer decoder is configured to predict 3D human pose data using the 2D human skeleton as input. Based on the 3D human pose data predicted by the Transformer decoder, an accurate 3D human model can be generated. Further, the 3D human model corresponding to each single-frame image is compared with a set standard action to obtain the comparison result. By comparing frame by frame, the deviation between each frame image and the standard action can be detected. Finally, based on the comparison result, an accurate human action score can be obtained. In summary, the above embodiments can improve the accuracy of human action evaluation.

[0111] In other embodiments of this application, the above-described visual capture and evaluation method for human motion further includes:

[0112] S201. Based on each single-frame image and the preset multi-target tracking algorithm, obtain the multi-target tracking result for each single-frame image.

[0113] Specifically, the human motion video to be evaluated may contain multiple target objects. In order to continuously track the position and posture of the target objects in the video images and ensure accurate capture of every key detail in the entire human motion, in this step, a preset multi-target tracking algorithm is used to analyze and process each single frame image to obtain the multi-target tracking result of each single frame image. Based on the multi-target tracking result of each single frame image, the first VisualTransformer model and Transformer decoder are used to obtain the 3D human model corresponding to each target object in each single frame image. This allows for independent standard motion comparison of each target object in the human motion video to obtain an independent human motion score for each target object.

[0114] Multi-Object Tracking (MOT) is an important research direction in the field of computer vision. The goal of MOT is to detect multiple targets in a video sequence and assign a unique identifier to each target, maintaining tracking even when targets are occluded, moving rapidly, or the camera is moving. In some embodiments, the preset multi-object tracking algorithm can adopt the 4D human tracking algorithm in the paper "Goel S, Pavlakos G, Rajasegaran J, et al. Humans in 4D: Reconstructing and tracking humans with transformers[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2023: 14783-14794". This algorithm can achieve multi-object tracking based on the 3D position, motion features, and appearance information of the target objects in the human motion video, so that even when multiple people are occluded or their positions are intertwined, the pose of each target object can still be stably tracked, ensuring the continuity of tracking and the integrity of the pose.

[0115] In other embodiments of this application, in order to achieve fine-grained analysis of hand movements, the above-mentioned visual capture and evaluation method for human movements further includes:

[0116] S301. Obtain the pre-built MediaPipe Hands model;

[0117] In S301, specifically, the MediaPipe Hands model is configured to detect 2D hand joint positions from a single frame image. The MediaPipe Hands model is a solution for hand and finger tracking that uses machine learning techniques to detect 2D hand joint positions from a single frame image, specifically including the coordinates of 21 key points of the hand, such as the wrist and finger joints, to describe the hand's posture and gestures.

[0118] S302. Using the MediaPipe Hands model, detect each single-frame image to obtain the 2D hand joint position of each single-frame image;

[0119] In S302, specifically, each single-frame image is input into the MediaPipe Hands model to obtain the 2D hand joint position of each single-frame image.

[0120] S303. Obtain the pre-built second Visual Transformer model;

[0121] In S303, specifically, the second Visual Transformer model is configured to predict the hand rotation angle based on the 2D hand joint positions as input; the hand rotation angle may include the finger rotation angle and the palm rotation angle. The second Visual Transformer model can be pre-trained using a large number of image samples and existing model training methods; this application does not impose any specific limitations on this.

[0122] In some embodiments, the training process of the second Visual Transformer model includes:

[0123] First, obtain an image dataset containing labeled 2D hand joint positions and their corresponding hand rotation angles; these image datasets can come from publicly available gesture recognition or hand motion datasets, such as the EgoHands dataset.

[0124] Then, the image dataset is preprocessed to obtain the preprocessed image dataset; the data preprocessing includes standardizing the 2D hand joint positions so that the model can adapt to different hand sizes and image resolutions.

[0125] Finally, the initial second Visual Transformer model is trained using the preprocessed image dataset to obtain a trained second Visual Transformer model. The architecture of the second Visual Transformer model includes a multi-layer self-attention mechanism, which effectively extracts the spatial relationships between hand joints, especially capturing relative positional changes between joints under complex movements. During training, the model aims to predict the rotation angles of the hand (including finger and palm rotation angles), and the loss function can be the mean squared error (MSE) to minimize the difference between the predicted angles and the true values. The optimization method can employ the Adam optimizer, setting an appropriate learning rate, and validating the model through methods such as cross-validation. Data augmentation techniques can also be introduced during training, such as performing rotation, translation, and scaling transformations on the hand, to increase the model's robustness. Finally, when evaluating the model's performance, the accuracy of the model under different gesture and movement conditions can be ensured through rotation angle prediction errors on the test set and other evaluation metrics (such as the root mean square value of the angle error), but this application is not limited to these.

[0126] S304. Using the second Visual Transformer model, process the 2D hand joint position corresponding to each single frame image to obtain the hand rotation angle corresponding to each single frame image;

[0127] In S304, specifically, the 2D hand joint position corresponding to each single frame image is input into the pre-built second Visual Transformer model to obtain the hand rotation angle corresponding to each single frame image, so as to perform action comparison based on the hand rotation angle later.

[0128] S305. Compare the 2D hand joint position and hand rotation angle corresponding to each single frame image with the set standard hand gesture to obtain the hand gesture comparison result, and obtain the hand gesture score based on the hand gesture comparison result.

[0129] In S305, specifically taking a dance training scenario as an example, the set standard hand gestures can be a pre-set sequence of standard hand gestures based on the dance teacher's dance video. The 2D hand joint position and hand rotation angle corresponding to each single frame image are compared with the set standard hand gestures frame by frame to obtain the hand gesture comparison result corresponding to each single frame image. Based on the hand gesture comparison result and the preset hand gesture scoring rules, the hand gesture score can be obtained.

[0130] For example, the 2D hand joint positions and hand rotation angles corresponding to each single-frame image are compared with a set standard gesture to detect deviations in 2D hand joint positions and hand rotation angles. Based on these deviations, a gesture score for each single-frame image is calculated according to a preset gesture scoring rule. The gesture scores of all single-frame images are then weighted and averaged to obtain the final gesture score. In the deviation comparison process, firstly, for the 2D hand joint positions in each single-frame image, the position deviation is measured by calculating the Euclidean distance between the current frame's 2D hand joint position and the corresponding joint position in the standard gesture. If the position deviation exceeds a set tolerance threshold, the joint position is considered to have an error. For the hand rotation angle, the rotation angle deviation between the current frame's hand rotation angle and the corresponding hand rotation angle in the standard gesture is calculated. If the rotation angle deviation exceeds a set range, it is considered inaccurate. The preset gesture scoring rule evaluates the score of each frame image based on the weighted average of the position and rotation angle deviations. Specifically, positional deviation and rotation angle deviation can be weighted according to their respective importance (e.g., positional deviation accounts for 70%, rotation angle deviation accounts for 30%), and then the score for each frame is calculated using a formula. Each frame's score is within the range [0,1], with scores closer to 1 indicating smaller deviations and more precise movements. Finally, a weighted average of the scores from all frames is used to obtain the overall gesture score. If the 2D hand joint position deviation and hand rotation angle deviation of all individual frames are within the set error range, the score is higher; if any individual frame's 2D hand joint position deviation and hand rotation angle deviation exceed the error range, the score will decrease, reflecting inaccurate or incomplete gestures. This application is not limited to this.

[0131] In some embodiments, prompts can be output based on 2D hand joint position deviations and hand rotation angle deviations, especially for common hand movements such as waving, spreading fingers, and clenching fists. If deviations are found, prompts such as "finger angle too small" or "wrist offset" can be output to facilitate understanding and improvement by the target audience, i.e., the dancer. In other embodiments, movement annotations can be performed based on 2D hand joint position deviations and hand rotation angle deviations to mark non-standard gestures and help dancers gradually standardize their gestures. Exemplarily, the annotation method uses color highlighting, with red indicating a large deviation, yellow indicating a small deviation, and green indicating compliance with standards; however, this application is not limited to this.

[0132] In this embodiment, based on the MediaPipe Hands model and the second Visual Transformer model, an accurate gesture score is obtained, which enables fine-grained analysis of hand movements and helps the target object understand the shortcomings of its own gestures.

[0133] Existing intelligent evaluation methods for human motion still involve using multimodal large models to analyze human motion videos and generate natural language comments. However, current multimodal large models typically analyze complete frame data directly when processing human motion videos, resulting in extremely high demands on computing resources. Model training and deployment rely heavily on high-performance GPUs, and are also time-consuming. However, for videos with human motion as the core and minimal background changes, it is completely unnecessary to input all the information from each frame into the model.

[0134] To improve the efficiency of generating natural language comments on human movements, in other embodiments of this application, the above-mentioned visual capture and evaluation method for human movements further includes:

[0135] S401. Obtain the joint position corresponding to each single frame image;

[0136] In S402, specifically, based on the visual capture technology described in S101-S106 above, the joint positions corresponding to each single frame image in the human motion video can be obtained. These joint positions can be 2D or 3D human joint positions. 2D joint positions can be the coordinates of 2D human joints, such as the 2D coordinates of the head, shoulder, elbow, wrist, hip, knee, and ankle. 3D joint positions can include the 3D coordinates of the head, shoulder, elbow, wrist, hip, knee, and ankle; this application does not impose specific limitations on this. A joint coordinate sequence can be constructed based on the joint coordinates corresponding to each joint in each single frame image, serving as the initial input for subsequent models. These joint coordinates can represent the human body's motion trajectory in the time series.

[0137] S402. Obtain the pre-built comment generation model;

[0138] In S402, specifically, the comment generation model is configured to take joint positions as input and output human action comments. The comment generation model adopts an end-to-end Transformer architecture, including: Embedding Layer, encoder position encoder, encoder, decoder position encoder, decoder and output layer.

[0139] The Transformer architecture, proposed by Vaswani et al. in 2017, is entirely based on the attention mechanism, abandoning the recurrent neural network structure, and has performed excellently in many natural language processing tasks such as text generation. In the comment generation model, the embedding layer takes the joint coordinate sequence of a single frame image as input and converts it into a fixed-dimensional vector representation. The encoder, at the bottom layer, directly adds positional encoding to the embedding vector of the input sequence. The encoder then transforms the vector representation of the input sequence into its output vector. The decoder, at the bottom layer, adds positional encoding to the encoder's output vector. The decoder transforms the encoder's output vector into a query vector and calculates the decoder's output vector by combining the query vector and the encoder's output vector. The output layer maps the decoder's output vector to the human action comment corresponding to the single frame image. The comment generation model can be pre-trained using existing model training methods. The Transformer architecture is end-to-end, meaning that the mapping from the input sequence to the output sequence is accomplished through the collaborative training of the entire network. During the training of the comment generation model, the network parameters can be updated by minimizing the difference between the predicted output sequence and the real output sequence (e.g., using the cross-entropy loss function). The parameters of the entire network are updated simultaneously, including the parameters of the encoder and decoder. This application does not impose any specific restrictions on this.

[0140] S403. Using a comment generation model, generate human motion comments based on the joint positions corresponding to each single-frame image.

[0141] In S402, specifically, a joint coordinate sequence can be constructed based on the joint position corresponding to each single frame image, input into the comment generation model, and human motion comments can be obtained.

[0142] In this embodiment, an end-to-end Transformer architecture-based comment generation model is adopted. Based on the joint position corresponding to a single frame image, a comment on the human body action of the target object is generated. Compared with the traditional method based on full-frame video, the above comment generation model focuses only on human joint data, which greatly reduces computational overhead and improves robustness, and can effectively improve the generation efficiency of natural language comments on human body actions.

[0143] In other embodiments of this application, to further improve the coherence and professionalism of the generated comments, the embedding layer is used for:

[0144] (1) Flatten the joint positions corresponding to each single frame image to obtain flattened coordinate vectors. ;

[0145] Specifically, This represents the frame number of a single frame image, for the th frame... A single frame of an image, assuming it corresponds to a frame. Joint coordinates The vector is flattened and integrated into a single vector, resulting in a flattened coordinate vector. ,in, Specifically, it depends on the dimension of the joint coordinates. If each joint coordinate is two-dimensional, then... If each joint coordinate is three-dimensional, then By flattening the data, multidimensional data can be processed by machine learning models. These coordinates represent the movement trajectory of the human body in a time series.

[0146] (2) Based on the flattened coordinate vector Each joint Corresponding coordinate vector With each joint Corresponding learnable embedding vectors Perform element-wise multiplication to obtain the first embedding vector. ;

[0147] Specifically, each joint Corresponding learnable embedding vectors The comment generation model can be continuously updated during training using the backpropagation algorithm to capture the specific semantic and functional features of the joint in the action sequence. This is achieved through joint-specific embedding, i.e., flattening the coordinate vector. Each joint Corresponding coordinate vector With the corresponding learnable embedding vector Element-wise multiplication is performed to obtain the first embedding vector. This approach emphasizes the importance of different joints, enabling the comment generation model to capture the unique semantic features of each joint in the action. This method is similar to word embedding techniques in natural language processing, which helps the comment generation model understand the functional differences between different joints.

[0148] (3) Based on the flattened coordinate vector Each joint Corresponding coordinate vector Perform 1D convolution to obtain the second embedding vector. ;

[0149] Specifically, this is achieved through temporal context embedding, i.e., embedding the flattened coordinate vector. Each joint Corresponding coordinate vector Using 1D convolution (kernel width = 3, number of channels = d), short-term motion dynamics are captured to obtain the second embedding vector. By applying one-dimensional convolution in the time dimension, the comment generation model can extract the motion trends and change patterns of joints in a short period of time. This helps to capture the dynamic characteristics of the action and improve the comment generation model's ability to model time dependencies.

[0150] (4) Based on the first embedding vector Second embedding vector The gating coefficient was calculated. The calculation formula is:

[0151] ;

[0152] In the formula, This represents the activation function. Represents the weight matrix. Indicates the bias term;

[0153] According to the first embedding vector Second embedding vector and gating coefficient The calculation yields the joints in each single frame image. The corresponding final embedding vector The calculation formula is:

[0154] ;

[0155] Specifically, by calculating the gating coefficient And merged into the final embedding vector The natural balance between joint-specific embeddings and temporal context is achieved through a gating mechanism that learns a weight to perform a weighted fusion between the joint-specific embeddings and the temporal context embeddings. This allows the comment generation model to dynamically adjust its focus on static positions and dynamic changes based on specific circumstances, thereby more accurately modeling complex human movements.

[0156] In other embodiments of this application, the encoder position encoder is used for:

[0157] Rotational position encoding and relative position encoding are used for each final embedding vector. Perform position encoding to obtain the corresponding first position code.

[0158] Specifically, Rotary Positional Encoding (RoPE) is a positional encoding method for Transformer models that encodes positional information by associating positional codes with each dimension of the input vector. This method can better capture the relative positional relationships between elements in a sequence, thereby improving the model's sensitivity to positional information.

[0159] For each final embedding vector Apply rotation position encoding to Multiply by the rotation matrix:

[0160] ;

[0161] In the formula, , The dimension of the vector is used to obtain the rotational position code. In the Transformer model, the rotational position code is usually added to the input vector and then fed into the model, thus effectively combining absolute and relative positional information. In this way, RoPE can capture the relative positional information between elements in a sequence without adding additional parameters, enhancing the model's ability to model time-series data.

[0162] Specifically, relative positional encoding is an improved positional encoding method that encodes not only the absolute position of elements but also their relative positions. This method can better capture the relative order and distance of elements in a sequence, thereby improving the model's understanding of positional information. Relative position bias is an implementation of relative positional encoding. It directly encodes relative positional information by adding a bias term to the self-attention mechanism. First, it defines the relative position of any two elements in the sequence, and for each possible relative position, it generates a corresponding relative position bias. When calculating self-attention, the relative position bias is added to the attention score to reflect the relative positional relationship between elements, for example, based on joints in a single frame image. and shortest path distance Add a learnable bias to the attention score To enhance spatial adjacency perception between joints in the graph structure, this bias term is added to the attention score in the self-attention mechanism, enabling the comment generation model to better utilize the structural information of the skeleton and enhance the modeling of relationships between joints.

[0163] Understandably, by using an encoder position encoder, rotational position encoding and relative position encoding mechanisms can be applied simultaneously in the model.

[0164] In other embodiments of this application, the encoder is used to: process each joint in each single frame image The corresponding final embedding vector The first feature is obtained by processing the corresponding first position code;

[0165] Specifically, each joint in each single frame image can be... The corresponding final embedding vector The first feature is obtained by adding the first position code to the first feature and inputting the result into the encoder. The encoder is then used to perform calculations to obtain the first feature.

[0166] The encoder consists of M encoder layers, where M is a positive integer that can be determined based on the specific application requirements; for example, M=8. The encoder is composed of multiple identical encoder layers stacked together. Each layer contains two sub-layers: an attention mechanism sub-layer and a feedforward neural network sub-layer. Residual connections and layer normalization can be used after each sub-layer to avoid the vanishing gradient problem and stabilize the training process.

[0167] In this embodiment, in the M-layer encoder layer, the encoder layers from layer 1 to layer L adopt the local self-attention mechanism, and the encoder layers from layer L+1 to layer M adopt the global multi-head self-attention mechanism.

[0168] Where L is a positive integer, which can be determined according to the actual application requirements; layers 1 to L adopt a local self-attention mechanism, and the maximum attention span of each layer's self-learning is... Only calculate inter-frame distance The attention mechanism focuses only on the temporal and spatial neighborhoods of each joint, concentrating on short-term spatiotemporal patterns. Through local self-attention, the comment generation model efficiently captures local spatiotemporal dependencies, reducing computational complexity. Layers L+1 to M employ standard multi-head global self-attention (8 heads). Integrate long-range dependencies and output mapping back. Global self-attention enhances the ability of comment generation models to model long-distance dependencies and helps to capture complex action patterns.

[0169] In other embodiments of this application, the decoder position encoder is used for:

[0170] By employing rotational position encoding and relative position encoding, position encoding is performed on each first feature to obtain the corresponding second position encoding.

[0171] Specifically, for the implementation details of rotational position encoding and relative position encoding, please refer to the section on encoder position encoders mentioned above, which will not be repeated here; it is understandable that by using decoder position encoders, rotational position encoding mechanism and relative position encoding mechanism can be applied simultaneously in the model.

[0172] The decoder consists of N decoder layers, where N is a positive integer and can be determined according to the actual application requirements, for example, N=6. The decoder is composed of multiple identical decoder layers stacked together. Each decoder layer includes three sub-layers: a masked self-attention layer, a cross-attention layer (also known as an encoder-decoder attention layer), and a feedforward neural network.

[0173] The decoder is used for:

[0174] (1) Global average pooling is performed on the first feature and the corresponding second position encoding to obtain the action intent vector;

[0175] Specifically, global average pooling (GAP) is performed on the first feature and the corresponding second position encoding of the encoder's final output to obtain a fixed-length action intent vector. This vector summarizes the macroscopic semantic features of the entire action and can be transformed into the initial hidden state of the first layer of the decoder through a linear mapping, so that the generator has the ability to perceive the overall structure from the beginning.

[0176] (2) Using the masked self-attention layer and feedforward neural network in the decoder layer of the first K layers, a comment outline skeleton is generated based on the action intention vector and the historical output of the decoder.

[0177] Specifically, K is a positive integer and can be determined according to actual application requirements, for example, K=2. In this embodiment, the decoder adopts a two-stage generation. Stage 1 (Coarse): The first two layers of the decoder only use mask self-attention, relying on the action intent vector. The historical output of the decoder (e.g., generated historical vocabulary) is used to quickly generate a comment outline skeleton through autoregression, outlining key phrases and the overall framework. During this process, future moments are strictly masked to ensure the sequentiality and consistency of the autoregressive generation.

[0178] (3) Using the masked self-attention layer, cross-attention layer and feedforward neural network in the decoder layer of the post-NK layer, the skeleton of the comment outline and the first feature are fused by gating fusion to obtain the output vector of the decoder;

[0179] Specifically, taking N=6 and K=2 as an example, in stage two (Fine): the last four layers of the decoder simultaneously perform cross-attention and mask self-attention, and combine the comment outline skeleton generated by the first two layers with the first feature output by the encoder through gating fusion, refine the comment content word by word, and ensure that while maintaining global coherence, it accurately reflects the details and professionalism of the action.

[0180] In other embodiments, the comment generation model uses Beam Search to generate a candidate set during the decoding process. Specifically, at each generation time step, all candidate words are sorted according to their cumulative log probability, and the five candidate sequences with the highest probability (Beam Size=5) are retained. In the next step, the model continues to expand based on these sequences until all sequences are generated, ending the process. <eos>This strategy balances search space and generation quality, effectively improving decoding efficiency and text readability.

[0181] The output layer is used to map the decoder's output vectors to human motion comments.

[0182] Specifically, the output layer is typically a linear transformation layer that maps the decoder's output vector to a dimension equal to the vocabulary size. Then, a softmax function is used to calculate the probability distribution for each word or token. For example, the decoder's top-level output is mapped to the vocabulary size V, and then softmax is applied to obtain the word probability distribution. This process is repeated when continuously generating... <eos>or end decoding when a maximum length (e.g., 50 words) is reached, obtaining a final human motion comment.

[0183] In the embodiment, a comment generation model based on an end-to-end Transformer architecture is adopted, joint position corresponding to a single frame image is taken as input, through adaptive embedding, mixed position coding, hierarchical local-global encoder and two-stage decoder, a human motion comment of a target object is generated, the calculation overhead is greatly reduced and the robustness is improved, the temporal-semantic mapping is jointly optimized in the self-attention framework, and the coherence and professionalism of the generated comment text are effectively ensured.

[0184] In other embodiments of the present application, for the dance training scene, in order to evaluate the fluency and rhythm of dance movements, the above-mentioned human motion visual capture and evaluation method further comprises:

[0185] S401. Obtain a continuity detection result according to the human motion video and a preset fluency detection model;

[0186] S402. Obtain a rhythm matching result according to the human motion video and a preset rhythm matching algorithm;

[0187] S403. Obtain a fluency score according to the continuity detection result and the rhythm matching result.

[0188] Specifically, the human motion video is specifically a dance video of a target object, i.e., a dancer, the fluency detection model is used to detect the continuity between dance movements in the dance video, such as the speed, direction and consistency of joint movement, to obtain a continuity detection result; wherein, by analyzing each frame of movement in the dance video, combining the changes of joint speed and direction, a time series-based model is constructed as a fluency detection model, for example, using an optical flow method or a convolutional neural network, the motion trajectory of the joint is extracted, the speed change and direction consistency of the joint between adjacent frames are detected, so as to evaluate the continuity between dance movements, and obtain a continuity detection result.

[0189] Wherein, the rhythm matching algorithm is used to analyze the matching degree of dance movements in the dance video and the set standard movements in terms of movement rhythm, to obtain a rhythm matching result. Specifically, the rhythm matching algorithm can compare the time sequence characteristics of the movements in the dance video and the set standard movements (such as dance templates), calculate the rhythm difference of the movements, use dynamic time warping (Dynamic Time Warping, DTW) or similarity matching algorithm to analyze the matching degree in terms of rhythm, judge the speed of dance movements and the degree of coincidence with standard movements, and obtain a rhythm matching result.

[0190] According to the continuity detection result and the rhythm matching result, a fluency score is obtained according to a set fluency scoring rule. For example, a weight can be set for each index, and a weighted average method is used to obtain the final score. The score range can be 0 to 1, and the closer to 1 indicates that the dance movement is more fluent and the rhythm matching degree is high. In some embodiments, improvement suggestions can also be generated according to the fluency score, such as appropriately reducing the action pause time or adjusting the action speed, to help the dancer improve the continuity and rhythm of the dance.

[0191] In other embodiments of the present application, for the dance training scene, in order to evaluate the emotional expression and artistic appeal of the dance, the above-mentioned visual capture and evaluation method of human body movement further includes:

[0192] S501. According to the human body movement video, the facial expression and emotion of the target object are identified.

[0193] S502. According to the facial expression and emotion, an emotional expression score is obtained.

[0194] Specifically, the human body movement video is specifically a dance video of the target object, i.e., the dancer. According to the dance video, the facial expression and emotion of the dancer are identified and compared with the set standard facial expression and emotion, and an emotional expression score is obtained according to the comparison result and the set emotional expression scoring rule. The facial expression and emotion recognition can be realized through computer vision technology, especially a facial expression recognition model based on deep learning, such as OpenCV combined with a convolutional neural network or a pre-trained facial expression recognition model (such as a model on the FER-2013 dataset), to analyze each frame in the dance video, extract facial features and identify the emotional state (such as happy, sad, surprised, etc.) of the dancer, and obtain the facial expression and emotion of the dancer. Then the facial expression and emotion of the dancer are compared with the standard facial expression and emotion, and a similarity calculation method (such as cosine similarity or Euclidean distance) can be used to measure the difference between the two, to obtain the comparison result. The emotional expression score can be calculated according to the comparison result and the preset scoring rule, for example, if the facial expression matches the standard emotion well, the score is close to 1, and if the matching degree is low, the score is reduced.

[0195] In some embodiments, according to the emotional expression score, prompt information can also be given to help the dancer perform the dance more vividly. For example, according to the matching degree between the facial expression and emotion of the dancer and the standard facial expression and emotion, specific prompt information can be provided, such as suggesting the dancer to enhance certain emotional expression, to improve the expressiveness and artistic appeal of the dance.

[0196] In other embodiments of this application, for dance training scenarios, in order to achieve systematic evaluation of dancer performance and generation of personalized training plans, the above-mentioned visual capture and evaluation method for human movements further includes:

[0197] S601. Based on human motion videos and a pre-trained visual generative model, obtain multi-dimensional scores and evaluation reports.

[0198] S602. Obtain historical training data of the target object, and generate a training plan for the target object based on the historical training data and the visual generative pre-trained model.

[0199] Specifically, the human motion video refers to the dance video of the target subject, namely the dancer; the historical training data refers to the dancer's historical dance training data; the training plan refers to the dancer's dance training plan; and the visual generative pre-training model can adopt large models such as GPT-4-vision. Through the multimodal analysis capabilities of the GPT-4-vision model, detailed comments are made from multiple dimensions such as the accuracy, fluency, and rhythm of the movements, resulting in multi-dimensional scores and evaluation reports. Based on the dancer's historical dance training data, personalized dance training calculations are provided to help the dancer improve their dance performance in all aspects.

[0200] Specifically, the GPT-4-vision model extracts features such as dancers' postures, movement continuity, and rhythmic variations from dance video clips through a visual processing module. It analyzes the dancers' performance in accuracy, fluency, rhythm, and emotional expression. The GPT-4-vision model compares the dancers' movement characteristics with set standard movements, identifying errors and providing multi-dimensional scores for "movement accuracy," "fluency," and "rhythm mastery." Based on the scores, it generates a multi-dimensional evaluation report detailing the dancers' strengths and weaknesses in each movement. For example, "Arm movements are in place, but footwork needs further adjustment," or "Rhythm mastery is good, but emotional expression is weak." Dancers can review the evaluation report after completing a movement and adjust their movements under the guidance of the GPT-4-vision model to help them steadily improve in actual practice.

[0201] Furthermore, historical dance training data from dancers is collected and analyzed using the GPT-4-vision model to generate personalized dance training plans. This historical training data can be generated by dancers using dance training software. For example, the GPT-4-vision model can generate phased practice suggestions based on the dancer's historical performance, outlining daily training tasks and key points for each segment. Each suggestion includes detailed movement guidance, such as "arm posture correction" and "weight maintenance," as well as specific practice goals, such as "practice footwork stability for 10 minutes." In subsequent practice sessions, the GPT-4-vision model dynamically adjusts the dance training plan based on the dancer's progress.

[0202] In a specific embodiment, taking a dance training scenario as an example, the above-mentioned visual capture and evaluation method for human movements can be applied to smart terminals such as mobile phones to perform offline analysis based on smart terminal recording and split-screen comparison.

[0203] Specifically, such as Figure 2 As shown, after completing their dance, the dancers record videos using their mobile phones. The phone's camera captures the dancers' movements and stores the complete dance video. The dance instructor's standard movement video is pre-stored on the phone for later comparative analysis. Using a pre-set dance movement evaluation method on the phone, offline 3D reconstruction and human posture detection are performed on the recorded dance video. This process includes the following steps: detecting 2D human joint positions from each frame of the dance video using a first Visual Transformer model and generating a 2D human skeleton; using a Transformer decoder to predict 3D human posture data and construct a 3D human model; comparing the 3D human model with the set standard movements to obtain the comparison results; and obtaining a dance movement score based on the comparison results.

[0204] Furthermore, videos of the dancer and the dance teacher can be displayed in a split-screen format on the phone screen for side-by-side comparison to enhance visual appeal. In the split-screen comparison, color coding is used to mark differences in the dancer's and teacher's movements. Red indicates a larger deviation, yellow indicates a smaller deviation, and green indicates the movement meets the standard. During playback, scores and errors are displayed frame by frame, allowing the dancer to clearly observe areas for improvement. Finally, based on the comparison results, an evaluation report is automatically generated for each dance move, including consistency, gesture accuracy, and rhythm. The evaluation report highlights the score, deviations, and improvement suggestions for each move. The dancer can review the evaluation report in detail after practice to focus on areas for improvement in the next session. This embodiment is suitable for scenarios where dancers want to review their performance and gradually optimize after dancing. It can be used directly with a mobile phone without needing a device connection, facilitating playback and frame-by-frame analysis of movement differences, and is suitable for high-precision offline analysis.

[0205] In other embodiments, the above-described visual capture and evaluation method for human motion can also be applied to smart wearable devices such as XR glasses to perform real-time motion analysis based on smart wearables.

[0206] Specifically, such as Figure 3 As shown, the practitioner enters the dance practice environment wearing XR glasses. The camera on the XR glasses captures the practitioner's dance movements in real time, providing a full-body view. The evaluation method for the aforementioned dance movements is run on the built-in computing unit of the XR glasses, achieving real-time reconstruction of 3D posture and dynamic capture of joint positions and rotation angles. The standard posture of the dance instructor can be projected in real-time into the XR view as a transparent virtual guidance image (ghost figure), displayed synchronously with the practitioner's posture. The practitioner can clearly see a comparison between their own movements and the dance instructor's in the XR view. The XR glasses compare the joint rotation angles and positions of the dance instructor and the practitioner, and if the practitioner's movements deviate, corresponding real-time prompts are displayed in the view (e.g., "Please raise your arms slightly" or "Maintain your foot position") to help the practitioner adjust immediately.

[0207] Furthermore, the system can analyze joint position errors, and areas exceeding a specific deviation threshold will be color-coded in real time (e.g., red for joints indicates a larger error). Dynamic prompts are provided within the XR view, allowing dancers to instantly perceive shortcomings in their movements. Automatic feedback can be generated at the end of each movement, such as "Arm not raised high enough, try raising the angle again" or "Movement in place." The XR glasses record the dancer's entire performance data and calculate scores in real time, finally displaying an overall score and improvement suggestions. For example, a summary of the overall performance will be generated, along with a comprehensive evaluation of multiple dimensions such as fluency, rhythm, and emotional expression. Dancers can choose to continue practicing in the XR environment or review after exiting it. This embodiment is suitable for scenarios requiring real-time correction of movements and rapid improvement of movement standards. It is particularly suitable for dance beginners and dancers who need frequent practice. Real-time feedback can greatly improve practice efficiency, allowing dancers to immediately correct their movements, contributing to rapid optimization and stability.

[0208] In other embodiments, the above-described visual capture and evaluation method for human movements can also be applied to offline analysis and personalized report generation of multiple dancers based on a dance studio camera.

[0209] Specifically, such as Figure 4 As shown, using a single camera in the dance studio combined with a multi-target tracking algorithm, the dance movements of multiple dancers in the same space are simultaneously captured, analyzed offline, and provided with personalized feedback. Through posture detection, 3D reconstruction, and multi-target tracking technology, the movements of multiple dancers can be distinguished and tracked, generating personalized feedback reports. After completing their practice, dancers can view detailed reports on smart terminals in the dance studio, such as tablets, thus helping each dancer to improve their movements in a targeted manner.

[0210] A single camera can be mounted at a slightly elevated position at the front of the dance studio, covering the entire space and ensuring that the dance movements of each dancer are clearly captured. A standard video of the dance instructor's movements is pre-stored on the tablet device, serving as a reference standard for comparison among multiple dancers and providing a benchmark for subsequent frame-by-frame analysis. After video acquisition, the first VisualTransformer model is used to detect the 2D joint positions of each dancer frame-by-frame, generating an independent 2D human skeleton for each dancer. This process ensures the differentiation of postures among different individuals within the same frame. Based on each dancer's 2D human skeleton, a Transformer decoder generates individual 3D human posture data for each dancer, including joint positions and rotation angles, and performs 3D human reconstruction to obtain a corresponding 3D human model for each dancer. Using the 4D human tracking algorithm presented in the paper, the posture information of multiple dancers is correlated frame-by-frame. Utilizing the dancers' 3D position, movement features, and appearance information, multi-target tracking is achieved, ensuring stable tracking of each dancer's posture even under conditions of multiple dancers occlusion and overlapping positions. The 4D human tracking algorithm can predict and fill in pose gaps when multiple people overlap or are moving rapidly by associating data from previous and subsequent frames, ensuring the continuity of tracking and the integrity of pose.

[0211] After 3D pose reconstruction and tracking, the 3D human model of each dancer is compared frame-by-frame with the standard movements of the dance teacher. The key joint positions, angles, and posture deviations of each dancer are calculated. Based on the comparison results for each frame, an independent dance movement score is generated for each dancer. The scoring dimensions include movement consistency, gesture accuracy, rhythm, and fluidity. By calculating movement deviations and posture stability frame-by-frame, independent score data for multiple dancers is generated.

[0212] In addition, for each dancer, the areas with significant movement deviations are recorded, and errors in joint position and posture are marked, such as incorrect arm angles or knee positions. Based on each dancer's movement score and deviation analysis, a personalized assessment report is automatically generated. Each assessment report includes, but is not limited to, the following:

[0213] Dance movement scoring: Provides an overall score for each dancer's consistency, fluency, and rhythm mastery; Deviation point analysis: Marks in detail the deviations and deficiencies of each dancer in specific movements; Improvement suggestions: Automatically generates personalized improvement suggestions for each dancer's specific errors, pointing out the parts and movements that need adjustment, such as "left hand needs to be raised higher" or "insufficient squatting range".

[0214] After each dancer finishes practice, an evaluation report is automatically uploaded to the dance studio's tablet, allowing each dancer to view their performance individually. Each dancer can select their own report on the tablet, which provides a simple interface displaying scores, deviation analysis, and improvement suggestions, ensuring everyone can quickly access their report. Based on the generated evaluation report, dancers can gradually improve specific movements, systematically enhancing the accuracy and consistency of their posture.

[0215] like Figure 5 As shown, in another embodiment of this application, a visual capture and evaluation system for human motion is also provided, comprising:

[0216] The first acquisition unit 10 is used to acquire the human motion video to be evaluated, and to decompose the human motion video into video frames to obtain multiple single-frame images.

[0217] The second acquisition unit 11 is used to acquire a pre-built first Visual Transformer model, wherein the first Visual Transformer model is configured to detect 2D human joint positions from a single frame image.

[0218] The detection unit 12 is used to detect each single frame image using the first Visual Transformer model to obtain the 2D human joint position of each single frame image.

[0219] The generation unit 13 is used to generate a 2D human skeleton corresponding to each single frame image based on the 2D human joint positions of each single frame image.

[0220] The third acquisition unit 14 is used to acquire a pre-built Transformer decoder, wherein the Transformer decoder is configured to predict 3D human pose data with 2D human skeleton as input.

[0221] The processing unit 15 is used to process the 2D human skeleton corresponding to each single frame image using the Transformer decoder to obtain the 3D human pose data corresponding to each single frame image.

[0222] The reconstruction unit 16 is used to perform 3D human body reconstruction based on the 3D human body pose data corresponding to each single frame image, so as to obtain the 3D human body model corresponding to each single frame image.

[0223] Evaluation unit 17 is used to compare the 3D human body model corresponding to each single frame image with the set standard action, obtain the comparison result, and obtain the human body action score based on the comparison result.

[0224] In other embodiments of this application, the 3D human posture data in the above-mentioned visual capture and evaluation system for human movements includes 3D human joint positions, human joint rotation angles, movement amplitude, speed characteristics, and overall body posture.

[0225] In other embodiments of this application, in the above-described visual capture and evaluation system for human motion, when the reconstruction unit 16 performs 3D human reconstruction based on the 3D human pose data corresponding to each single frame image to obtain a 3D human model corresponding to each single frame image, it is specifically used for:

[0226] Based on the 3D human pose data corresponding to each single frame image, the SMPL model is used to reconstruct the 3D human body, resulting in a 3D human body model corresponding to each single frame image.

[0227] In other embodiments of this application, the above-described visual capture and evaluation system for human motion also includes a tracking unit, used to: obtain the multi-target tracking result of each single-frame image based on each single-frame image and a preset multi-target tracking algorithm.

[0228] In other embodiments of this application, the above-mentioned visual capture and evaluation system for human motion further includes:

[0229] The fourth acquisition unit is used to acquire a pre-built MediaPipe Hands model, wherein the MediaPipeHands model is configured to detect 2D hand joint positions from a single frame image;

[0230] The hand detection unit is used to detect each single frame image using the MediaPipe Hands model to obtain the 2D hand joint position of each single frame image;

[0231] The fifth acquisition unit is used to acquire a pre-built second Visual Transformer model, wherein the second Visual Transformer model is configured to predict the hand rotation angle with 2D hand joint positions as input.

[0232] The hand processing unit is used to process the 2D hand joint position corresponding to each single frame image using the second Visual Transformer model to obtain the hand rotation angle corresponding to each single frame image.

[0233] The gesture evaluation unit is used to compare the 2D hand joint position and hand rotation angle corresponding to each single frame image with the set standard gesture action to obtain the gesture comparison result, and to obtain the gesture action score based on the gesture comparison result.

[0234] In other embodiments of this application, the above-mentioned visual capture and evaluation system for human motion further includes:

[0235] The sixth acquisition unit is used to acquire the joint position corresponding to each single frame image, wherein the joint position is either a 2D human joint position or a 3D human joint position.

[0236] The seventh acquisition unit is used to acquire a pre-built comment generation model, wherein the comment generation model is configured to take joint positions as input and output human action comments. The comment generation model adopts an end-to-end Transformer architecture, including: an embedding layer, an encoder position encoder, an encoder, a decoder position encoder, a decoder, and an output layer.

[0237] The comment generation unit is used to generate human motion comments based on the joint positions corresponding to each single frame image using a comment generation model.

[0238] In other embodiments of this application, in the above-described visual capture and evaluation system for human motion, the embedding layer is used for:

[0239] For each single-frame image, the joint positions are flattened to obtain flattened coordinate vectors. ,in, Indicates the number of frames in a single image;

[0240] Based on flattened coordinate vectors Each joint Corresponding coordinate vector With each joint Corresponding learnable embedding vectors Perform element-wise multiplication to obtain the first embedding vector. ;

[0241] Based on flattened coordinate vectors Each joint Corresponding coordinate vector Perform 1D convolution to obtain the second embedding vector. ;

[0242] According to the first embedding vector Second embedding vector The gating coefficient was calculated. The calculation formula is:

[0243] ;

[0244] In the formula, This represents the activation function. Represents the weight matrix. Indicates the bias term;

[0245] According to the first embedding vector Second embedding vector and gating coefficient The calculation yields the joints in each single frame image. The corresponding final embedding vector The calculation formula is:

[0246] .

[0247] In other embodiments of this application, in the above-described visual capture and evaluation system for human motion, the encoder position encoder is used for:

[0248] Rotational position encoding and relative position encoding are used for each final embedding vector. Perform position encoding to obtain the corresponding first position code;

[0249] Encoders are used for:

[0250] For each joint in each single frame image The corresponding final embedding vector The first feature is obtained by processing the corresponding first position code;

[0251] The encoder includes an M-layer encoder layer. The encoder layers from layer 1 to layer L adopt a local self-attention mechanism, and the encoder layers from layer L+1 to layer M adopt a global multi-head self-attention mechanism.

[0252] In other embodiments of this application, in the above-described visual capture and evaluation system for human motion, the decoder position encoder is used for:

[0253] By employing rotational position encoding and relative position encoding, position encoding is performed on each first feature to obtain the corresponding second position encoding;

[0254] The decoder consists of N decoder layers, where each decoder layer includes a masked self-attention layer, a cross-attention layer, and a feedforward neural network. The decoder is used for:

[0255] Global average pooling is performed on the first feature and the corresponding second position encoding to obtain the action intent vector;

[0256] By utilizing the masked self-attention layer and feedforward neural network in the first K layers of the decoder layer, a comment outline skeleton is generated based on the action intent vector and the historical output of the decoder.

[0257] By utilizing the masked self-attention layer, cross-attention layer, and feedforward neural network in the decoder layer of the post-NK layer, the skeleton of the comment outline and the first feature are fused through gating fusion to obtain the output vector of the decoder.

[0258] In other embodiments of this application, in the above-described visual capture and evaluation system for human motion, the output layer is used to map the output vector of the decoder into human motion comments.

[0259] In other embodiments of this application, the above-described visual capture and evaluation system for human motion further includes a fluency evaluation unit, used for:

[0260] Based on the human motion video and the preset smoothness detection model, the coherence detection results are obtained;

[0261] Based on the human motion video and the preset rhythm matching algorithm, the rhythm matching result is obtained;

[0262] A fluency score is obtained based on the coherence test results and rhythm matching results.

[0263] In other embodiments of this application, the above-described visual capture and evaluation system for human motion further includes an emotion expression evaluation unit, used for:

[0264] Based on human motion videos, facial expressions and emotions of the target object can be identified.

[0265] An emotional expression score is obtained based on facial expressions and emotions.

[0266] In other embodiments of this application, the above-described visual capture and evaluation system for human motion further includes a comprehensive evaluation unit, used for:

[0267] Based on human motion videos and a pre-trained visual generative model, multi-dimensional scores and evaluation reports are obtained.

[0268] In other embodiments of this application, in the above-described visual capture and evaluation system for human motion, the comprehensive evaluation unit is further used for:

[0269] Obtain historical training data for the target object;

[0270] Based on historical training data and a visual generative pre-trained model, a training plan for the target object is generated.

[0271] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.< / eos> < / eos>

Claims

1. A method for visually capturing and evaluating human movements, characterized in that, include: The human motion video to be evaluated is acquired, and the human motion video is decomposed into video frames to obtain multiple single-frame images. Obtain a pre-built first Visual Transformer model, wherein the first Visual Transformer model is configured to detect 2D human joint positions from the single-frame image; Using the first Visual Transformer model, each of the single-frame images is detected to obtain the 2D human joint positions of each single-frame image; Based on the 2D human joint positions of each single frame image, a 2D human skeleton corresponding to each single frame image is generated. Obtain a pre-built Transformer decoder, wherein the Transformer decoder is configured to predict 3D human pose data with the 2D human skeleton as input, the 3D human pose data including 3D human joint positions, human joint rotation angles, range of motion, velocity features and overall body pose. Using the Transformer decoder, the 2D human skeleton corresponding to each single frame image is processed to obtain the 3D human pose data corresponding to each single frame image. Based on the 3D human pose data corresponding to each single frame image, 3D human reconstruction is performed to obtain a 3D human model corresponding to each single frame image. The 3D human body model corresponding to each single frame image is compared with the set standard action to obtain the comparison result, and a human body action score is obtained based on the comparison result. Obtain the joint position corresponding to each of the single-frame images, wherein the joint position is the 2D human joint position or the 3D human joint position; Obtain a pre-built comment generation model, wherein the comment generation model is configured to take the joint position as input and output human action comments, and the comment generation model adopts an end-to-end Transformer architecture, including: an embedding layer, an encoder position encoder, an encoder, a decoder position encoder, a decoder and an output layer; Using the aforementioned comment generation model, human motion comments are generated based on the joint positions corresponding to each single-frame image.

2. The method as described in claim 1, characterized in that, The step of reconstructing a 3D human body based on the 3D human pose data corresponding to each single frame image to obtain a 3D human body model corresponding to each single frame image includes: Based on the 3D human pose data corresponding to each single frame image, 3D human reconstruction is performed using the SMPL model to obtain a 3D human model corresponding to each single frame image.

3. The method as described in claim 1, characterized in that, Also includes: Obtain a pre-built MediaPipe Hands model, wherein the MediaPipe Hands model is configured to detect 2D hand joint positions from the single-frame image; Using the MediaPipe Hands model, the 2D hand joint positions of each frame image are detected to obtain the 2D hand joint positions of each frame image. Obtain a pre-built second Visual Transformer model, wherein the second Visual Transformer model is configured to predict the hand rotation angle based on the 2D hand joint position as input; Using the second Visual Transformer model, the 2D hand joint position corresponding to each single frame image is processed to obtain the hand rotation angle corresponding to each single frame image; The 2D hand joint position and hand rotation angle corresponding to each single frame image are compared with the set standard hand gesture to obtain the hand gesture comparison result, and a hand gesture score is obtained based on the hand gesture comparison result.

4. The method as described in claim 1, characterized in that, The embedding layer is used for: For each of the single-frame images, the joint positions are flattened to obtain flattened coordinate vectors. ,in, This indicates the number of frames in the single-frame image; According to the flattened coordinate vector Each joint Corresponding coordinate vector With each joint Corresponding learnable embedding vectors Perform element-wise multiplication to obtain the first embedding vector. ; According to the flattened coordinate vector Each joint Corresponding coordinate vector Perform 1D convolution to obtain the second embedding vector. ; According to the first embedding vector and the second embedding vector The gating coefficient was calculated. The calculation formula is: ； In the formula, This represents the activation function. Represents the weight matrix. Indicates the bias term; According to the first embedding vector The second embedding vector and the gate coefficient The calculation yields each joint in each of the single-frame images. The corresponding final embedding vector The calculation formula is: 。 5. The method as described in claim 4, characterized in that, The encoder position encoder is used for: Rotational position encoding and relative position encoding are used for each of the final embedding vectors. Perform position encoding to obtain the corresponding first position code; The encoder is used for: For each joint in each of the single-frame images The corresponding final embedding vector The first feature is obtained by processing the corresponding first position code; The encoder includes an M-layer encoder layer, wherein the encoder layers from layer 1 to layer L employ a local self-attention mechanism, and the encoder layers from layer L+1 to layer M employ a global multi-head self-attention mechanism.

6. The method as described in claim 5, characterized in that, The decoder position encoder is used for: By employing rotational position encoding and relative position encoding, position encoding is performed on each of the first features to obtain the corresponding second position encoding; The decoder comprises N decoder layers, wherein each decoder layer includes a masked self-attention layer, a cross-attention layer, and a feedforward neural network. The decoder is used for: Global average pooling is performed on the first feature and the corresponding second position encoding to obtain the action intent vector; Using the mask self-attention layer and the feedforward neural network in the decoder layer of the first K layers, a comment outline skeleton is generated based on the action intent vector and the historical output of the decoder. By utilizing the masked self-attention layer, the cross-attention layer, and the feedforward neural network in the decoder layer of the post-NK layer, the comment outline skeleton and the first feature are fused through gating fusion to obtain the output vector of the decoder.

7. The method as described in claim 6, characterized in that, The output layer is used to map the output vector of the decoder into human motion comments.

8. A visual capture and evaluation system for human movements, characterized in that, include: The first acquisition unit is used to acquire the human motion video to be evaluated, and to decompose the human motion video into video frames to obtain multiple single-frame images. The second acquisition unit is used to acquire a pre-built first Visual Transformer model, wherein the first Visual Transformer model is configured to detect 2D human joint positions from the single-frame image; The detection unit is used to detect each of the single-frame images using the first Visual Transformer model to obtain the 2D human joint positions of each single-frame image. The generation unit is used to generate a 2D human skeleton corresponding to each single frame image based on the 2D human joint positions of each single frame image. The third acquisition unit is used to acquire a pre-built Transformer decoder, wherein the Transformer decoder is configured to predict 3D human pose data with the 2D human skeleton as input, and the 3D human pose data includes 3D human joint positions, human joint rotation angles, range of motion, velocity features and overall body posture. The processing unit is used to process the 2D human skeleton corresponding to each single frame image using the Transformer decoder to obtain 3D human pose data corresponding to each single frame image. The reconstruction unit is used to perform 3D human body reconstruction based on the 3D human body pose data corresponding to each single frame image, so as to obtain a 3D human body model corresponding to each single frame image. The evaluation unit is used to compare the 3D human body model corresponding to each single frame image with the set standard action, obtain the comparison result, and obtain the human body action score based on the comparison result. The sixth acquisition unit is used to acquire the joint position corresponding to each single frame image, wherein the joint position is the 2D human joint position or the 3D human joint position. The seventh acquisition unit is used to acquire a pre-built comment generation model, wherein the comment generation model is configured to take the joint position as input and output human action comments. The comment generation model adopts an end-to-end Transformer architecture, including: an embedding layer, an encoder position encoder, an encoder, a decoder position encoder, a decoder, and an output layer. The comment generation unit is used to generate human motion comments based on the joint positions corresponding to each single frame image using the comment generation model.

Images