Visual capture and evaluation method and system for human body actions

Abstract

The invention provides a human body motion visual capture and evaluation method and system, and the method comprises the steps: carrying out the video frame decomposition of a human body motion video, and obtaining a plurality of single-frame images; using a first Visual Transform model to detect each single-frame image, and obtaining a 2D human body joint position of each single-frame image; according to the 2D human body joint position of each single-frame image, generating a 2D human body skeleton corresponding to each single-frame image; processing the 2D human body skeleton corresponding to each single-frame image by using a Transform decoder to obtain 3D human body posture data corresponding to each single-frame image; obtaining a 3D human body model corresponding to each single-frame image according to the 3D human body posture data; and comparing the 3D human body model corresponding to each single-frame image with a set standard action to obtain a human body action score. The accuracy of human body action evaluation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual analysis and human motion recognition, and particularly relates to a human motion visual capturing and evaluation method and system. BACKGROUND

[0002] The existing human motion intelligent evaluation method generally scores the accuracy of human motion based on the position of the joint point. For example, a 2D joint position is extracted through a convolutional neural network (CNN), and a simple posture analysis is performed based on the 2D joint position to obtain an evaluation result. However, the joint position detection accuracy of this method is low, and the posture deviation analysis is also lacking in consideration, and the overall evaluation accuracy needs to be improved.

[0003] Therefore, how to improve the accuracy of human motion evaluation is a technical problem to be solved by those skilled in the art. SUMMARY

[0004] To solve the above technical problems, the present application provides a human motion visual capturing and evaluation method, which can improve the accuracy of motion evaluation. The present application also provides a human motion visual capturing and evaluation system, which has the same technical effect.

[0005] The first object of the present application is to provide a human motion visual capturing and evaluation method.

[0006] The above application object of the present application is achieved by the following technical scheme: A human motion visual capturing and evaluation method, comprising: obtaining a human motion video to be evaluated, and performing video frame decomposition on the human motion video to obtain a plurality of single-frame images; obtaining a first Visual Transformer model constructed in advance, wherein the first Visual Transformer model is configured to detect 2D human joint positions from the single-frame images; detecting each single-frame image using the first Visual Transformer model to obtain the 2D human joint positions of each single-frame image; generating a 2D human skeleton corresponding to each single-frame image according to the 2D human joint positions of each single-frame image; obtaining a Transformer decoder constructed in advance, wherein the Transformer decoder is configured to input the 2D human skeleton to predict 3D human posture data; processing, by the Transformer decoder, the 2D human skeleton corresponding to each of the single-frame images to obtain 3D human posture data corresponding to each of the single-frame images; reconstructing a 3D human body according to the 3D human posture data corresponding to each of the single-frame images to obtain a 3D human model corresponding to each of the single-frame images; comparing the 3D human model corresponding to each of the single-frame images with a set of standard actions to obtain a comparison result, and obtaining a human action score according to the comparison result.

[0007] Preferably, in the human action visual capture and evaluation method, the 3D human posture data includes 3D human joint positions, human joint rotation angles, action amplitudes, speed characteristics, and overall body postures.

[0008] Preferably, in the human action visual capture and evaluation method, the 3D human model corresponding to each of the single-frame images is obtained by reconstructing a 3D human body according to the 3D human posture data corresponding to each of the single-frame images, including: reconstructing a 3D human body according to the 3D human posture data corresponding to each of the single-frame images by using an SMPL model to obtain a 3D human model corresponding to each of the single-frame images.

[0009] Preferably, in the human action visual capture and evaluation method, it further includes: obtaining a pre-constructed MediaPipe Hands model, wherein the MediaPipe Hands model is configured to detect 2D hand joint positions from the single-frame images; detecting each of the single-frame images by using the MediaPipe Hands model to obtain 2D hand joint positions of each of the single-frame images; obtaining a pre-constructed second Visual Transformer model, wherein the second Visual Transformer model is configured to input the 2D hand joint positions to predict hand rotation angles; processing the 2D hand joint positions corresponding to each of the single-frame images by using the second Visual Transformer model to obtain hand rotation angles corresponding to each of the single-frame images; comparing the 2D hand joint positions and the hand rotation angles corresponding to each of the single-frame images with a set of standard hand gesture actions to obtain a hand gesture comparison result, and obtaining a hand gesture action score according to the hand gesture comparison result.

[0010] Preferably, the method for visually capturing and evaluating human motion further comprises: Acquire a joint position corresponding to each of the single-frame images, wherein the joint position is a 2D human body joint position or a 3D human body joint position; Obtain a pre-built comment generation model, wherein the comment generation model is configured to take the joint positions as input and output human action comments, and the comment generation model adopts an end-to-end Transformer architecture, including: an embedding layer, an encoder position encoder, an encoder, a decoder position encoder, a decoder, and an output layer; The comment generation model is used to generate human body action comments according to the joint positions corresponding to each single-frame image.

[0011] Preferably, in the method for visually capturing and evaluating human motion, the embedding layer is used to: Flatten the joint position corresponding to each single frame image to obtain a flattened coordinate vector ,in, Indicates the frame number of the single-frame image; According to the flattened coordinate vector Each joint The corresponding coordinate vector , with each joint The corresponding learnable embedding vector Perform element-by-element multiplication to obtain the first embedding vector ; According to the flattened coordinate vector Each joint The corresponding coordinate vector , perform 1-dimensional convolution calculation to obtain the second embedding vector ; According to the first embedding vector and the second embedding vector , calculate the gating coefficient , the calculation formula is: ; Where, represents the activation function, represents the weight matrix, represents the bias term; According to the first embedding vector , the second embedding vector and the gating coefficient , calculate each joint in each single frame image The corresponding final embedding vector , the calculation formula is: .

[0012] Preferably, in the human action visual capture and evaluation method, the encoder position encoder is configured to: perform position encoding on each of the final embedding vectors to obtain corresponding first position encodings; the encoder is configured to: perform processing on each joint corresponding final embedding vectors and corresponding first position encodings to obtain first features; wherein the encoder comprises M encoder layers, wherein the encoder layers from the 1st layer to the Lth layer employ a local self-attention mechanism, and the encoder layers from the L+1th layer to the Mth layer employ a global multi-head self-attention mechanism.

[0013] Preferably, in the human action visual capture and evaluation method, the decoder position encoder is configured to: perform position encoding on each of the first features to obtain corresponding second position encodings using rotational position encoding and relative position encoding; the decoder comprises N decoder layers, wherein each of the decoder layers comprises a masked self-attention layer, a cross-attention layer, and a feed-forward neural network, and the decoder is configured to: perform global average pooling on the first features and corresponding second position encodings to obtain an action intention vector; generate an outline skeleton of a comment based on the action intention vector and historical outputs of the decoder using the masked self-attention layer and the feed-forward neural network in the first K decoder layers; fuse the outline skeleton of the comment and the first features through gated fusion using the masked self-attention layer, the cross-attention layer, and the feed-forward neural network in the last N-K decoder layers to obtain an output vector of the decoder.

[0014] Preferably, in the human action visual capture and evaluation method, the output layer is configured to map the output vector of the decoder to a human action comment.

[0015] A second object of the present application is to provide a human action visual capture and evaluation system.

[0016] The above-mentioned second object of the present application is achieved by the following technical solutions: A human action visual capture and evaluation system, comprising: The first acquisition unit is configured to acquire a human action video to be evaluated, and perform video frame decomposition on the human action video to obtain a plurality of single-frame images. The second acquisition unit is configured to acquire a first Visual Transformer model constructed in advance, wherein the first Visual Transformer model is configured to detect 2D human joint positions from the single-frame images. The detection unit is configured to detect each single-frame image by using the first Visual Transformer model to obtain the 2D human joint positions of each single-frame image. The generation unit is configured to generate a 2D human skeleton corresponding to each single-frame image according to the 2D human joint positions of each single-frame image. The third acquisition unit is configured to acquire a Transformer decoder constructed in advance, wherein the Transformer decoder is configured to take the 2D human skeleton as input to predict 3D human pose data. The processing unit is configured to process the 2D human skeleton corresponding to each single-frame image by using the Transformer decoder to obtain 3D human pose data corresponding to each single-frame image. The reconstruction unit is configured to perform 3D human reconstruction according to the 3D human pose data corresponding to each single-frame image to obtain a 3D human model corresponding to each single-frame image. The evaluation unit is configured to compare the 3D human model corresponding to each single-frame image with a set standard action to obtain a comparison result, and obtain a human action score according to the comparison result.

[0017] The technical solution described above, by obtaining a human action video to be evaluated, and performing video frame decomposition on the human action video to obtain a plurality of single-frame images; and based on the single-frame images, visual capture is performed to extract human joint data, specifically, by obtaining a first Visual Transformer model constructed in advance, the first Visual Transformer model is used to detect each single-frame image to obtain the 2D human joint position of each single-frame image; wherein the first Visual Transformer model is configured to detect the 2D human joint position from the single-frame image, and by applying the first Visual Transformer model, even in the case of fast action, occlusion or blur, the human key point position can still be accurately captured. Further, according to the 2D human joint position of each single-frame image, a 2D human skeleton corresponding to each single-frame image is generated; a Transformer decoder constructed in advance is obtained, and the Transformer decoder is used to process the 2D human skeleton corresponding to each single-frame image to obtain 3D human posture data corresponding to each single-frame image; according to the 3D human posture data corresponding to each single-frame image, 3D human reconstruction is performed to obtain a 3D human model corresponding to each single-frame image; wherein the Transformer decoder is configured to take the 2D human skeleton as input to predict the 3D human posture data, and based on the 3D human posture data predicted by the Transformer decoder, an accurate 3D human model can be generated. Further, the 3D human model corresponding to each single-frame image is compared with a set standard action to obtain a comparison result, by the frame-by-frame comparison method, the deviation of each frame image from the standard action can be detected, and finally according to the comparison result, an accurate human action score can be obtained. In summary, the technical solution described above can improve the accuracy of human action evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 A flowchart of a human action visual capture and evaluation method provided in an embodiment of the present application; Figure 2 An application schematic diagram of a mobile phone recording scene provided in an embodiment of the present application; Figure 3 An application schematic diagram of an XR eye scene provided in an embodiment of the present application; Figure 4An application diagram of a dance camera scene provided in an embodiment of the present application; Figure 5 A structure diagram of a human action visual capture and evaluation system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order for those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0021] In the embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are only illustrative. For example, the division of units and modules is only a logical function division. In actual implementation, there can be another division manner, such as: a plurality of units or modules can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0022] In addition, each functional unit in each embodiment of the present application can be integrated into one processor, or each unit can be a separate device, or two or more units can be integrated into one device. Each functional unit in each embodiment of the present application can be realized in the form of hardware or hardware plus software functional unit.

[0023] Those skilled in the art can understand that all or part of the steps of the following method embodiments can be completed by program instructions and related hardware. The aforementioned program instructions can be stored in a computer readable storage medium. When the program instructions are executed, the steps of the method embodiments are executed. The aforementioned storage medium includes mobile storage devices, read-only memories (ROM), magnetic discs or optical discs, and various storage media that can store program codes.

[0024] It should be understood that if "system", "device", "unit" and / or "module" are used in the present application, it is only a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0025] In addition, the terms "first", "second", etc. are used only for descriptive purposes and should not be construed as implying or implying relative importance or implying the number of the technical features indicated. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" or "several" is two or more, unless otherwise explicitly specified.

[0026] If a flowchart is used in the present application, the flowchart is used to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or subsequent operations are not necessarily performed in sequence. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0027] It should also be noted that in this document, terms such as "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such article or device. Without more limitations, the element defined by the statement "including one" does not exclude the presence of another identical element in the article or device including the above element.

[0028] The embodiments of the present application are written in a progressive manner.

[0029] As shown in Figure 1 The embodiments of the present application provide a visual capture and evaluation method of human action, which comprises: S101. Obtain the human action video to be evaluated, and perform video frame decomposition on the human action video to obtain a plurality of single frame images; In S101, specifically, the human action of the target object can be captured by a video shooting device to obtain the human action video to be evaluated, and then the human action video to be evaluated is decomposed into a plurality of single frame images for subsequent feature extraction and comparison according to the single frame images. Different application scenarios can correspond to different target objects. Taking a work operation scene as an example, the target object can be a work operation personnel, taking a technical training scene as an example, the target object can be a trainee participating in training, and taking a dance training scene as an example, the target object can be a dancer learning dance, and the present application is not limited thereto.

[0030] S102. Obtain a first Visual Transformer model constructed in advance; In S102, specifically, Visual Transformer (ViT) is a visual model based on Transformer architecture, proposed by Google team in 2020. ViT divides an image into fixed-size patches, each patch is converted into a vector representation and taken as an input sequence. These vectors are processed by a Transformer encoder, which captures the context-dependent relationships between different positions in the image using self-attention mechanisms. Finally, a specific visual task is completed by classifying or regressing the encoder output.

[0031] wherein the first Visual Transformer model is configured to detect 2D human joint positions from a single frame image, the 2D human joint positions can be coordinates of 2D human joints, for example, 2D coordinate positions of head, shoulder, elbow, wrist, hip, knee and ankle. The first Visual Transformer model can be pre-trained by a large number of image samples using existing model training methods, and the present application does not make specific limitations thereto.

[0032] Due to the existence of complex poses in human actions, the first Visual Transformer model considers the dynamic adaptability to different human actions and complex poses when designing. Specifically, the first Visual Transformer model divides a single frame image into fixed-size small blocks (such as 16x16 pixel patches), and inputs each image block as an independent input unit into the Transformer structure for processing, wherein each image block is embedded into a high-dimensional vector space (embedding), mapped to a specific dimension through linear transformation, and added with position information coding (position encoding), so that the first Visual Transformer model can understand the positional relationship of each image block; in addition, the self-attention mechanism of the first Visual Transformer model can help the first Visual Transformer model to capture the dependency relationship between image blocks, allowing the model to detect the joint points through image context and known structural features even when the human body is partially occluded, for example, if the arm is occluded by the torso, the first Visual Transformer model will infer the approximate position of the wrist through the relative position of the shoulder and elbow. For blurred situations in fast motion, the attention mechanism of the first Visual Transformer model will associate different parts of the image block, and infer the actual position of the joint based on the position of the surrounding joints and the overall skeleton. In this way, the first Visual Transformer model can maintain stable detection effect when processing high-speed human actions. It can be known that by applying the first Visual Transformer model, the position of the human body key point can be accurately captured even in the case of fast action, occlusion or blur.

[0033] S103. Utilize the first Visual Transformer model to detect each single frame image to obtain the 2D human joint position of each single frame image; In S103, specifically, each single frame image is input into the first Visual Transformer model constructed in advance to obtain the 2D human joint position of each single frame image, so as to generate a 2D skeleton according to the 2D human joint position subsequently.

[0034] S104. Generate a 2D human skeleton corresponding to each single frame image according to the 2D human joint position of each single frame image; In S104, specifically, the 2D human joint positions can be coordinates of the 2D human joints. According to the coordinates of the 2D human joints in each single-frame image, a 2D human skeleton corresponding to each single-frame image can be generated, wherein the 2D human skeleton is used to connect the human joints to form the relative positions of the human joints and the overall structure of the human body, for subsequent 3D pose reconstruction.

[0035] In some embodiments, a preset 2D skeleton generation algorithm can be used to generate a 2D human skeleton corresponding to each single-frame image according to the 2D human joint positions of each single-frame image. Specifically, the 2D skeleton generation algorithm can use an existing OpenPose algorithm. Using the 2D skeleton generation algorithm, the relative positions between adjacent joints in each single-frame image can be calculated according to the 2D human joint positions of each single-frame image, and then the relative angles and positional relationships between the joints can be determined. In each single-frame image, the skeleton structure is corrected according to the relative proportions and angles of the skeleton lines, so that it maintains a natural human form, and finally a complete 2D human skeleton is established. In this way, even in fast action, deflection, or body occlusion, the pose of the target object can be accurately constructed according to the 2D human skeleton.

[0036] S105. Obtain a pre-constructed Transformer decoder; In S105, specifically, the decoder of the Transformer mainly consists of multiple layers of the same decoder stacked together, each layer including three main sub-layers: a masked multi-head self-attention layer, an encoder-decoder attention layer, and a feed-forward neural network layer (FFN). These sub-layers are connected through residual connections and layer normalization to ensure effective information transmission and model stability.

[0037] The Transformer decoder is configured to take the 2D human skeleton as input and predict 3D human pose data. The Transformer decoder can be pre-trained using existing model training methods based on prior joint structures and training data, and the present application does not make specific limitations on this. Using the Transformer decoder, 2D joint information can be mapped to 3D space to obtain 3D human pose data.

[0038] In some embodiments, the training process of the Transformer decoder includes: First, a dataset containing 2D human skeletons and their corresponding 3D pose data is obtained; among them, the 2D human skeletons in the dataset have been standardized to ensure that the model can maintain good generalization ability under different image resolutions and human sizes. In order to enhance the robustness of the model, the dataset is processed by data augmentation, including rotating, scaling and horizontally flipping the joint positions, etc.

[0039] Then, the initial Transformer decoder is trained using the dataset to obtain the trained Transformer decoder. In terms of model architecture, the Transformer decoder adopts a standard multi-layer self-attention mechanism structure, each layer contains multiple attention heads to capture the mutual relationship between different joints. The input data of the Transformer decoder is the 2D human skeleton, and the spatial information of the joints is preserved through position encoding. The Transformer decoder reconstructs the 3D pose according to the 2D human skeleton and the pre-trained joint structure and angle prior, and outputs the 3D human pose data. In order to ensure that the output 3D human pose data conforms to the natural structure of the human skeleton, the model embeds the spatial relationship and angle prior between joints during training. The training process can use the Adam optimizer and use the Mean Squared Error (MSE) as the loss function, the goal is to minimize the difference between the 2D joint position prediction and the true 3D pose. During training, the batch size can be set to a predetermined value, and after each training round, the model's performance is evaluated through cross-validation. During the evaluation process, mainly use 3D joint reconstruction error and other indicators to quantify the prediction accuracy of the model. Finally, after multiple rounds of training, the Transformer decoder can accurately predict the corresponding 3D human pose data from the input 2D human skeleton, especially in dynamic environments or complex scenarios, it can still maintain high precision and robustness.

[0040] In some embodiments, the 3D human pose data can specifically include 3D human joint positions and human joint rotation angles, for example, the 3D human joint positions can include the 3D coordinate positions of the head, shoulders, elbows, wrists, hips, knees and ankles, and the human joint rotation angles can include the pitch, roll and yaw angles of the shoulder joint and knee joint. In order to subsequently construct a 3D human model based on the 3D human joint positions and human joint rotation angles, compare the 3D human model with the set standard action, detect the joint position and joint rotation angle deviation, and calculate the human action score according to the joint position and joint rotation angle deviation.

[0041] In some embodiments, the 3D human pose data further includes motion amplitude, speed features, and overall body posture. Specifically, the motion amplitude and speed features of the target object in 3D space are extracted from the 2D human skeleton using the Transformer decoder, which are used to build the 3D human model and standard action comparison. For example, in a fast kicking action, the Transformer decoder can capture the angular velocity change of the leg, so that the speed deviation of the action can be retrieved during subsequent comparison, and the human action score can be calculated according to the speed deviation. The overall body posture of the target object is analyzed from the 2D human skeleton using the Transformer decoder, including the center of gravity, balance, and coordination of the body, to obtain the overall body posture information, which is used to build the 3D human model and standard action comparison.

[0042] To achieve the prediction of motion amplitude, speed features, and overall body posture, the Transformer decoder learns not only the mapping from 2D human skeleton to 3D joint position during training, but also improves the prediction accuracy by adding motion features and speed information. Specifically, the Transformer decoder introduces time series information and joint motion change rate in the input during training, so that the model can capture the motion amplitude and speed features of the joints. In addition, the Transformer decoder also uses the prior knowledge of the relative position and angle between joints to predict the overall body posture during training. By using a multi-task joint optimization objective function (including 3D joint position error, joint motion speed error, and posture constraint term based on biomechanics prior), the relationship between motion amplitude, speed features, and overall body posture is optimized, so that the Transformer decoder can accurately predict these dynamic features from the 2D human skeleton. During training, the Transformer decoder continuously adjusts the model parameters to minimize the prediction error, and accurately estimates the motion amplitude and speed features by comparing the posture changes at different time steps. Through the Transformer decoder, more rich 3D human pose data can be obtained from the 2D human skeleton, which can build a more accurate 3D human model.

[0043] S106. Utilize the Transformer decoder to process the 2D human skeleton corresponding to each single-frame image to obtain 3D human pose data corresponding to each single-frame image; In S106, specifically, the 2D human skeleton corresponding to each single-frame image is input into the pre-constructed Transformer decoder to obtain 3D human pose data corresponding to each single-frame image, so as to subsequently establish a 3D human model according to the 3D human pose data.

[0044] S107. According to the 3D human pose data corresponding to each single-frame image, 3D human reconstruction is performed to obtain a 3D human model corresponding to each single-frame image. In S107, specifically, an existing 3D human reconstruction method can be used to perform 3D human reconstruction according to the 3D human pose data corresponding to each single-frame image, to obtain a 3D human model corresponding to each single-frame image, which is used for subsequent comparison with the standard action.

[0045] In some embodiments, one implementation of the present step specifically includes: performing 3D human reconstruction using an SMPL model according to the 3D human pose data corresponding to each single-frame image, to obtain a 3D human model corresponding to each single-frame image.

[0046] Specifically, the SMPL (Skinned Multi-Person Linear Model) model is a statistical model for three-dimensional human modeling and motion capture, which can describe the pose and shape of the human body. Through 3D human pose data, the SMPL model can be driven to perform 3D human reconstruction and output an accurate 3D human model. For example, using the SMPL model, modeling and optimization can be performed based on the 3D human joint positions and human joint rotation angles in the 3D human pose data, and accurate joint angles can be obtained by minimizing the pose and skeleton error, to construct an accurate 3D human model.

[0047] S108. Comparing the 3D human model corresponding to each single-frame image with the set standard action to obtain a comparison result, and obtaining a human action score according to the comparison result.

[0048] In S108, specifically, frame-by-frame comparison of the 3D human model corresponding to each single-frame image with the set standard action can obtain a comparison result corresponding to each single-frame image, and according to the comparison result and the pre-set scoring rule, a human action score can be obtained.

[0049] For example, the 3D human body model is compared with the set standard action, the joint position deviation and the joint rotation angle deviation are detected, and the human body action score is calculated according to the joint position deviation and the joint rotation angle deviation according to the preset scoring rule. The joint position deviation can be measured by the Euclidean distance, and the joint rotation angle deviation can be measured by the angle difference or the distance of the quaternion. For the deviation of each joint, including the joint position deviation and the joint rotation angle deviation, it can be judged whether the deviation is within the allowable error range. If the deviation exceeds the allowable error range, it means that the action is not accurate. According to the preset scoring rule, different weights can be set for each joint deviation based on the deviation of each joint, and the overall human body action score is calculated by comprehensively considering the deviations of each joint. The specific rule can be that a set score is deducted when the deviation of each joint exceeds the allowable error range, and the final human body action score is the weighted average of the joint scores. The application is not limited to this.

[0050] For example, in the construction operation scene, the set standard action can be a standard action sequence preset based on the construction video meeting the safety operation specification. Correspondingly, the human body action score obtained based on the above implementation steps can help the construction operation personnel to evaluate whether their own construction action meets the safety operation specification, so as to remind the construction personnel to improve in time. For example, in the technical training scene, the set standard action can be a standard action sequence preset based on the action video of the technician who has strict requirements on the action. Correspondingly, the human body action score obtained based on the above implementation steps can help the trainee participating in the training to evaluate whether their own technical action meets the technical training requirement, so as to help the trainee to find problems in time. For example, in the dance training scene, the set standard action can be a standard action sequence preset based on the dance video of the dance teacher. It can be provided by the dancer, or the dance course or dance template in the dance teaching software can be used. Correspondingly, the human body action score obtained based on the above implementation steps can help the dancer participating in the dance to evaluate the gap between their own dance action and the dance teacher, so as to help improve the dance level of the dancer. The application does not make specific limitations in this regard.

[0051] In some embodiments, prompt information can also be output according to the joint position deviation and the joint rotation angle deviation, for example, warnings such as "left shoulder deviation" and "insufficient arm angle" are made at the points with large joint and limb position deviation, so as to mark the non-standard action part and help the target object to concentrate on adjustment. In other embodiments, the human body action video to be evaluated can be segmented by time or structure, and compared with the preset standard action respectively to obtain the segmented human body action score, so as to provide accurate evaluation for each segment of action.

[0052] The existing intelligent evaluation method of human action is generally based on the position of the joint node to score the accuracy of human action. For example, the 2D joint position is extracted by a convolutional neural network, and a simple posture analysis is performed based on the 2D joint position to obtain an evaluation result. However, the joint position detection accuracy of this method is low, and the posture deviation analysis is also lacking in consideration, and the overall evaluation accuracy needs to be improved.

[0053] In the above embodiment, the human action video to be evaluated is obtained, and the human action video is decomposed into video frames to obtain a plurality of single-frame images. Visual capture is performed based on the single-frame images to extract human joint data. Specifically, a first Visual Transformer model is obtained in advance, and the first Visual Transformer model is used to detect each single-frame image to obtain the 2D human joint position of each single-frame image. The first Visual Transformer model is configured to detect the 2D human joint position from the single-frame image. Even in the case of fast action, occlusion or blur, the first Visual Transformer model can still accurately capture the human key point position. Further, the 2D human skeleton corresponding to each single-frame image is generated according to the 2D human joint position of each single-frame image. A Transformer decoder is obtained in advance, and the Transformer decoder is used to process the 2D human skeleton corresponding to each single-frame image to obtain the 3D human posture data corresponding to each single-frame image. The 3D human reconstruction is performed according to the 3D human posture data corresponding to each single-frame image to obtain the 3D human model corresponding to each single-frame image. The Transformer decoder is configured to input the 2D human skeleton to predict the 3D human posture data. Based on the 3D human posture data predicted by the Transformer decoder, an accurate 3D human model can be generated. Further, the 3D human model corresponding to each single-frame image is compared with the set standard action to obtain a comparison result. By the frame-by-frame comparison method, the deviation of each frame image from the standard action can be detected. Finally, according to the comparison result, an accurate human action score can be obtained. In summary, the above embodiment can improve the accuracy of human action evaluation.

[0054] In other embodiments of the present application, the above-mentioned human action visual capture and evaluation method further comprises: S201. According to each single-frame image and a preset multi-target tracking algorithm, a multi-target tracking result of each single-frame image is obtained.

[0055] Specifically, in the human action video to be evaluated, multiple target objects can be included. In order to continuously track the positions and postures of the target objects in the video images and ensure accurate capture of each key detail in the entire human action, in this step, a preset multi-target tracking algorithm is used to analyze and process each single-frame image to obtain a multi-target tracking result of each single-frame image. According to the multi-target tracking result of each single-frame image, a 3D human body model corresponding to each target object in each single-frame image is obtained by using a first VisualTransformer model and a Transformer decoder, so as to independently compare the standard actions of each target object in the human action video and obtain an independent human action score of each target object.

[0056] In some embodiments, the preset multi-target tracking algorithm can use the 4D human tracking algorithm in the paper “Goel S, Pavlakos G, Rajasegaran J, et al. Humans in 4D: Reconstructing and tracking humans with transformers [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 14783-14794”. In this way, multi-target tracking can be achieved according to the 3D positions, action features and appearance information of the target objects in the human action video, so that the postures of each target object can be stably tracked in the case of multi-person occlusion, position interlacing and the like, ensuring the continuity of tracking and the integrity of the postures.

[0057] In other embodiments of the present application, in order to realize the fineness analysis of the hand action, the above-mentioned human action visual capture and evaluation method further comprises: S301. Obtain a pre-constructed MediaPipe Hands model; In S301, specifically, the MediaPipe Hands model is configured to detect 2D hand joint positions from single-frame images, where the MediaPipe Hands model is a solution for hand and finger tracking that detects 2D hand joint positions, including 21 key point coordinates of the hand such as wrist, finger joint, etc., from single-frame images using machine learning technology, to describe the posture and gesture of the hand.

[0058] S302. Detect each single-frame image using the MediaPipe Hands model to obtain the 2D hand joint positions of each single-frame image. In S302, specifically, each single-frame image is input into the MediaPipe Hands model to obtain the 2D hand joint positions of each single-frame image.

[0059] S303. Obtain a pre-constructed second Visual Transformer model. In S303, specifically, the second Visual Transformer model is configured to take 2D hand joint positions as input to predict hand rotation angles; the hand rotation angles can include finger rotation angles and palm rotation angles. The second Visual Transformer model can be pre-trained using existing model training methods based on a large number of image samples, and the present application does not make specific limitations.

[0060] In some embodiments, the training process of the second Visual Transformer model includes: First, obtain an image dataset containing labeled 2D hand joint positions and their corresponding hand rotation angles; where these image datasets can come from public gesture recognition or hand action datasets such as the EgoHands dataset.

[0061] Then, data preprocessing is performed on the image dataset to obtain a preprocessed image dataset; where data preprocessing includes standardizing 2D hand joint positions so that the model can adapt to different hand sizes and image resolutions.

[0062] Finally, the pre-processed image dataset is used to train the initial second Visual Transformer model, obtaining a trained second Visual Transformer model. The architecture of the second Visual Transformer model includes a multi-layer self-attention mechanism, which can effectively extract the spatial relationship between hand joints, especially in complex actions to capture the relative position changes between joints. During the training process, the goal of the model is to predict the rotation angle of the hand (including the rotation angle of the fingers and the palm), and the mean square error (MSE) can be selected as the loss function to minimize the difference between the predicted angle and the true value. The optimization method can use the Adam optimizer, set an appropriate learning rate, and verify it through cross-validation and other methods. Data augmentation techniques such as rotating, translating, and scaling the hand can also be introduced during training to increase the robustness of the model. Finally, to evaluate the performance of the model, the rotation angle prediction error and other evaluation indicators (such as the root mean square value of the angle error) on the test set can be used to ensure the accuracy of the model under different gesture and action conditions, and the present application is not limited thereto.

[0063] S304. Utilize the second Visual Transformer model to process the 2D hand joint positions corresponding to each single frame image to obtain the hand rotation angle corresponding to each single frame image. In S304, specifically, the 2D hand joint positions corresponding to each single frame image are input into the pre-constructed second Visual Transformer model to obtain the hand rotation angle corresponding to each single frame image, so as to subsequently compare the hand gestures according to the hand rotation angle.

[0064] S305. Compare the 2D hand joint positions and hand rotation angle corresponding to each single frame image with the set standard gesture action to obtain a gesture comparison result, and obtain a gesture action score according to the gesture comparison result.

[0065] In S305, specifically, taking the dance training scene as an example, the set standard gesture action can be a standard gesture action sequence pre-set based on a dance video of a dance teacher. Comparing the 2D hand joint positions and hand rotation angle corresponding to each single frame image with the set standard gesture action frame by frame can obtain a gesture comparison result corresponding to each single frame image, and according to the gesture comparison result and the pre-set gesture scoring rule, a gesture action score can be obtained.

[0066] For example, the 2D hand joint position and hand rotation angle corresponding to each single frame image are compared with the set standard gesture action, the 2D hand joint position deviation and hand rotation angle deviation are detected, and the gesture score of each single frame image is calculated according to the 2D hand joint position deviation and hand rotation angle deviation of each single frame image according to the preset gesture scoring rule. The gesture scores of all single frame images are weighted and averaged to obtain the final gesture action score. In the deviation comparison process, first, for the 2D hand joint position in each single frame image, the position deviation is measured by calculating the Euclidean distance between the 2D hand joint position of the current frame and the corresponding joint position in the standard gesture action. If the position deviation exceeds the set tolerance threshold, it is considered that the joint position has an error. For the hand rotation angle, the rotation angle deviation between the hand rotation angle of the current frame and the corresponding hand rotation angle in the standard gesture action is calculated. If the rotation angle deviation exceeds the set range, it is considered to be inaccurate. The preset gesture scoring rule can evaluate the score of each frame according to the weighted average of the position deviation and the rotation angle deviation. Specifically, the position deviation and the rotation angle deviation can be weighted according to their importance (for example, the position deviation accounts for 70%, and the rotation angle deviation accounts for 30%), and then the score of each frame is calculated by the formula. The score of each frame is in the interval [0, 1], and the closer to 1 indicates that the deviation is smaller and the action is more accurate. Finally, the gesture action score of the entire gesture action is obtained by weighted averaging the scores of all frames. If the 2D hand joint position deviation and the hand rotation angle deviation of all single frame images are within the set error range, the score is higher. If the 2D hand joint position deviation and the hand rotation angle deviation of a single frame image exceed the error range, the score will decrease, reflecting that the gesture action is inaccurate or incomplete. The present application is not limited thereto.

[0067] In some embodiments, prompt information can also be output according to the 2D hand joint position deviation and the hand rotation angle deviation, especially for common hand actions such as waving, finger spreading, and fist clenching. If the deviation is found, prompt information such as "finger angle is too small" and "wrist deviation" can be output to facilitate the understanding and improvement of the target object, i.e. the dancer. In other embodiments, action annotation can also be performed according to the 2D hand joint position deviation and the hand rotation angle deviation to mark non-standard gestures and help the dancer standardize the gestures gradually. Exemplarily, the annotation method uses color highlighting, with red indicating a large deviation, yellow indicating a small deviation, and green indicating compliance with the standard. The present application is not limited thereto.

[0068] In the present embodiment, based on the MediaPipe Hands model and the second Visual Transformer model, an accurate gesture action score is obtained, which can realize the fine analysis of hand actions and help the target object understand the shortcomings of their own gestures.

[0069] The existing intelligent evaluation method of human action still uses a multi-modal large model to analyze the human action video and generate a natural language comment. However, the current multi-modal large model usually directly analyzes the complete frame data when processing the human action video, resulting in a very high demand for computing resources. The model training and deployment rely on a large number of high-performance GPUs, and the time consumption is also long. However, for videos with human action as the core and small background changes, it is completely unnecessary to input all the information of each frame into the model.

[0070] In order to improve the generation efficiency of the natural language comment of the human action, in other embodiments of the present application, the above-mentioned visual capture and evaluation method of human action further comprises: S401. Obtain the joint position corresponding to each single frame image; In S402, specifically, the joint position corresponding to each single frame image in the human action video can be obtained based on the visual capture technology in S101-S106. The joint position can be a 2D human joint position or a 3D human joint position. The 2D human joint position can be the coordinates of the 2D human joints, such as the 2D coordinate positions of the head, shoulder, elbow, wrist, hip, knee, and ankle. The 3D human joint position can include the 3D coordinate positions of the head, shoulder, elbow, wrist, hip, knee, and ankle. The present application does not make specific limitations on this. The joint coordinate sequence can be constructed according to the joint coordinate corresponding to each joint in each single frame image as the initial input of the subsequent model. These joint coordinates can represent the motion trajectory of the human body in the time sequence.

[0071] S402. Obtain a pre-constructed comment generation model; In S402, specifically, the comment generation model is configured to input the joint position and output the human action comment. The comment generation model adopts an end-to-end Transformer architecture, including an embedding layer (Embedding Layer), an encoder position encoder, an encoder (Encoder), a decoder position encoder, a decoder (Decoder), and an output layer (Output Layer).

[0072] The Transformer architecture is proposed by Vaswani et al. in 2017, which is completely based on the attention mechanism and abandons the structure of the recurrent neural network, and performs well in many natural language processing tasks such as text generation tasks. The embedding layer in the comment generation model can convert the joint coordinate sequence of a single frame image into a fixed-dimensional vector representation as an input sequence. The position encoder at the bottom layer of the encoder can directly add the position encoding to the embedding vector of the input sequence. The encoder can convert the vector representation of the input sequence into an output vector of the encoder. The position encoder at the bottom layer of the decoder can add the position encoding to the output vector of the encoder. The decoder can convert the output vector of the encoder into a query vector and calculate the query vector and the output vector of the encoder to obtain an output vector of the decoder. The output layer can map the output vector of the decoder to the human action comment corresponding to the single frame image. The comment generation model can be pre-trained using existing model training methods. The Transformer architecture is end-to-end, which means that the mapping from the input sequence to the output sequence is completed through the collaborative training of the entire network. In the training process of the comment generation model, the network parameters can be updated by minimizing the difference between the predicted output sequence and the real output sequence (for example, using a cross-entropy loss function). The parameters of the entire network are updated simultaneously, including the parameters of the encoder and the decoder. The present application does not make specific limitations.

[0073] S403. Utilizing the comment generation model, generating the human action comment according to the joint position corresponding to each single frame image.

[0074] In S402, specifically, the joint coordinate sequence can be constructed according to the joint position corresponding to each single frame image, input into the comment generation model, and the human action comment is obtained.

[0075] In this embodiment, the comment generation model based on the end-to-end Transformer architecture is used to generate the human action comment of the target object according to the joint position corresponding to the single frame image. Compared with the traditional method based on the full frame video, the above comment generation model only focuses on the human joint data, greatly reduces the computational overhead and improves the robustness, and can effectively improve the generation efficiency of the natural language comment of the human action.

[0076] In other embodiments of the present application, in order to further improve the coherence and professionalism of the generated comments, in other embodiments of the present application, the embedding layer is used to: (1) performing flattening processing on the joint position corresponding to each single frame image to obtain a flattened coordinate vector ; Specifically, Indicates the number of frames of a single frame image. A single frame image of a frame, assuming that it corresponds to Joint coordinates , flatten it and integrate it into a single vector to obtain the flattened coordinate vector ,in, , depending on the dimensions of the joint coordinates. If each joint coordinate is two-dimensional, then , if each joint coordinate is three-dimensional, then ,Through flattening processing, multi-dimensional data can be processed by machine learning models, and these coordinates represent the movement trajectory of the human body in the time series.

[0077] (2) According to the flattened coordinate vector Each joint The corresponding coordinate vector , with each joint The corresponding learnable embedding vector Perform element-by-element multiplication to obtain the first embedding vector ; Specifically, each joint The corresponding learnable embedding vector , can be continuously updated through the back-propagation algorithm during the training process of the comment generation model to capture the specific semantic and functional characteristics of the joint in the action sequence, and the flattened coordinate vector Each joint The corresponding coordinate vector and the corresponding learnable embedding vector , perform element-by-element multiplication to obtain the first embedding vector , to emphasize the importance of different joints, so that the comment generation model can capture the unique semantic characteristics of each joint in the action. This method is similar to the word embedding technology in natural language processing, which helps the comment generation model understand the functional differences of different joints.

[0078] (3) According to the flattened coordinate vector Each joint The corresponding coordinate vector , perform 1-dimensional convolution calculation to obtain the second embedding vector ; Specifically, through temporal context embedding, the flattened coordinate vector Each joint The corresponding coordinate vector , using 1D convolution (kernel width = 3, number of channels = d) to capture short-term motion dynamics, and obtain the second embedding vector By applying one-dimensional convolution in the time dimension, the comment generation model can extract the movement trend and change pattern of the joint in a short time, which helps to capture the dynamic characteristics of the action and improve the modeling ability of the comment generation model for time dependence.

[0079] (4) According to the first embedding vector and the second embedding vector , the gating coefficient is calculated, and the calculation formula is: ; In the formula, , represents the activation function, , represents the weight matrix, , and represents the bias term. According to the first embedding vector , the second embedding vector and the gating coefficient , the final embedding vector corresponding to each joint in each single-frame image is calculated, and the calculation formula is: ; Specifically, by calculating the gating coefficient and fusing it into the final embedding vector , the joint specificity and the time sequence context are naturally balanced. The gating mechanism learns a weight to perform weighted fusion between the above joint-specific embedding and the time sequence context embedding. In this way, the comment generation model can dynamically adjust the attention degree to static position and dynamic change according to specific circumstances, so as to more accurately model complex human actions.

[0080] In other embodiments of the present application, the encoder position encoder is used to: The rotation position encoding and the relative position encoding are used to perform position encoding on each final embedding vector to obtain the corresponding first position encoding.

[0081] Specifically, the rotary position encoding (Rotary Positional Encoding, RoPE) is a position encoding method for the Transformer model, which encodes position information by associating position encoding with each dimension of the input vector. This method can better capture the relative position relationship between elements in the sequence, thereby improving the sensitivity of the model to position information.

[0082] The rotary position encoding is applied to each final embedding vector , and is multiplied by a rotation matrix: ; wherein, , represents the latitude of the vector, and the rotation position encoding is obtained. In the Transformer model, the rotation position encoding is usually added to the input vector, and then input into the model, so as to effectively combine the absolute and relative position information. In this way, RoPE can capture the relative position information between elements in the sequence without increasing additional parameters, and enhance the modeling ability of the model for time series data.

[0083] Specifically, the relative position encoding (Relative Positional Encoding) is an improved position encoding method, which not only encodes the absolute position of the element, but also encodes the relative position between the elements. This method can better capture the relative order and distance of the elements in the sequence, thereby improving the understanding of the model for the position information. The relative position bias (Relative Position Bias) is an implementation of the relative position encoding, which directly encodes the relative position information by adding a bias term to the self-attention mechanism. First, the relative position between any two elements in the sequence is defined, and for each possible relative position, a corresponding relative position bias is generated. In the calculation of self-attention, the relative position bias is added to the attention score to reflect the relative position relationship between the elements, for example, according to the shortest path distance between the joints of and the shortest path distance add a learnable bias to the attention score to strengthen the spatial adjacency perception between the joints in the graph structure. This bias term is added to the attention score in the self-attention mechanism, so that the review generation model can better utilize the structural information of the skeleton and enhance the modeling of the relationship between the joints.

[0084] It can be understood that through the encoder position encoder, the rotation position encoding mechanism and the relative position encoding mechanism can be applied in the model at the same time.

[0085] In other embodiments of the present application, the encoder is configured to: process each joint corresponding final embedding vector and the corresponding first position encoding to obtain a first feature; Specifically, each joint corresponding final embedding vector and the corresponding first position encoding can be added and input into the encoder for calculation and processing to obtain the first feature.

[0086] The encoder consists of M encoder layers, where M is a positive integer that can be determined based on actual application requirements, for example, M = 8. The encoder is composed of multiple identical encoder layers stacked together, each of which contains two sublayers: an attention mechanism sublayer and a feedforward neural network sublayer. Residual connections and layer normalization can be used after each sublayer to avoid the vanishing gradient problem and stabilize the training process.

[0087] In this embodiment, among the M encoder layers of the encoder, the encoder layers from the 1st layer to the Lth layer adopt the local self-attention mechanism, and the encoder layers from the L+1th layer to the Mth layer adopt the global multi-head self-attention mechanism.

[0088] Among them, L is a positive integer and can be determined according to actual application requirements; layers 1 to L use a local self-attention mechanism, and each layer self-learns the maximum attention span , only calculate the distance between frames The attention of the model is to focus only on the adjacent areas of each joint in time and space, focusing on short-term spatiotemporal patterns. Through local self-attention, the comment generation model can efficiently capture local spatiotemporal dependencies and reduce computational complexity. Layers L+1 to M use standard multi-head global self-attention (8 heads, ), integrating long-range dependencies, and outputting the map back ,Global self-attention enhances the ability of the comment generation model to model long-distance dependencies, ,which helps to capture complex action patterns.

[0089] In other embodiments of the present application, the decoder position encoder is used to: Rotational position coding and relative position coding are used to perform position coding on each first feature to obtain a corresponding second position coding.

[0090] Specifically, for the implementation details of rotational position encoding and relative position encoding, please refer to the above-mentioned encoder position encoder part, which will not be repeated here; it can be understood that through the decoder position encoder, the rotational position encoding mechanism and the relative position encoding mechanism can be applied simultaneously in the model.

[0091] The decoder consists of N decoder layers, where N is a positive integer that can be determined based on actual application requirements, for example, N = 6. The decoder is composed of multiple identical decoder layers stacked together. Each decoder layer consists of three sublayers: a masked self-attention layer, a cross-attention layer (also known as the encoder-decoder attention layer), and a feedforward neural network.

[0092] Among them, the decoder is used to: (1) Perform global average pooling on the first feature and the corresponding second position encoding to obtain the action intention vector; Specifically, the first feature of the encoder's final output and the corresponding second position encoding are subjected to global average pooling (GAP) to obtain a fixed-length action intention vector. , this vector summarizes the macroscopic semantic features of the entire action and can be converted into the initial hidden state of the first layer of the decoder through a layer of linear mapping, so that the generation has the ability to perceive the overall structure at the beginning.

[0093] (2) Generate a skeleton review outline based on the action intention vector and the decoder’s historical output using the masked self-attention layer and feedforward neural network in the first K layers of the decoder; Specifically, K is a positive integer and can be determined according to actual application requirements, for example, K=2. In this embodiment, the decoder adopts a two-stage generation, stage 1 (Coarse): the first two layers of the decoder only use mask self-attention, relying on the action intention vector The historical output of the decoder (such as the generated historical vocabulary) is used to quickly generate the comment outline skeleton through autoregression, outlining the key phrases and the overall framework. In this process, future moments are strictly shielded to ensure the sequentiality and consistency of the autoregressive generation.

[0094] (3) Using the masked self-attention layer, cross-attention layer, and feedforward neural network in the decoder layer after the NK layer, the review outline skeleton and the first feature are fused through gated fusion to obtain the output vector of the decoder; Specifically, taking N=6 and K=2 as an example, in stage 2 (Fine), the last four layers of the decoder simultaneously perform cross-attention and masked self-attention, and combine the comment outline skeleton generated by the first two layers with the first feature output of the encoder through gated fusion, refining the comment content word by word to ensure that the action details and professionalism are accurately reflected while maintaining global coherence.

[0095] In other embodiments, the comment generation model uses beam search to generate candidate sets during the decoding process. Specifically, at each generation time step, all candidate words are sorted by cumulative logarithmic probability, and the five candidate sequences with the highest probability (beam size = 5) are retained. In the next step, these sequences are further expanded until all sequences are generated. <eos>The strategy balances the search space and the generation quality, effectively improving the decoding efficiency and the text readability.

[0096] An output layer for mapping the output vector of the decoder to human motion comments.

[0097] Specifically, the output layer is usually a linear transformation layer that maps the output vector of the decoder to the dimension of the vocabulary size, and then uses the softmax function to calculate the probability distribution of each word or token. For example, the top layer output of the decoder is mapped to the vocabulary size V, and then the word probability distribution is obtained through Softmax. When continuous generation <eos>or end decoding when a maximum length (e.g., 50 words) is reached, obtaining a final human motion comment.

[0098] In the embodiment, a comment generation model based on an end-to-end Transformer architecture is adopted, taking joint positions corresponding to a single frame of image as input, generating a human motion comment of a target object through adaptive embedding, mixed position coding, a hierarchical local-global encoder, and a two-stage decoder. The comment generation model can greatly reduce computational overhead and improve robustness, while jointly optimizing temporal-semantic mapping within a self-attention framework, effectively ensuring the coherence and professionalism of the generated comment text.

[0099] In other embodiments of the present application, for the dance training scene, in order to evaluate the fluency and rhythm of dance movements, the above-mentioned human motion visual capture and evaluation method further comprises: S401. Obtain a continuity detection result according to the human motion video and a preset fluency detection model; S402. Obtain a rhythm matching result according to the human motion video and a preset rhythm matching algorithm; S403. Obtain a fluency score according to the continuity detection result and the rhythm matching result.

[0100] Specifically, the human motion video is specifically a dance video of a target object, i.e., a dancer, the fluency detection model is used to detect the continuity between dance movements in the dance video, such as the speed, direction, and consistency of joint movement, to obtain a continuity detection result; wherein, by analyzing each frame of movement in the dance video, combining the changes in joint speed and direction, a time series-based model is constructed as the fluency detection model, for example, using an optical flow method or a convolutional neural network to extract the motion trajectory of the joint, detect the speed change and direction consistency of the joint between adjacent frames, and thus evaluate the continuity between dance movements, to obtain a continuity detection result.

[0101] The rhythm matching algorithm is used to analyze the matching degree of dance movements in the dance video and a set of standard movements in terms of movement rhythm, to obtain a rhythm matching result. Specifically, the rhythm matching algorithm can compare the time sequence features of the movements in the dance video and the set of standard movements (such as dance templates), calculate the rhythm difference of the movements, use dynamic time warping (DTW) or similarity matching algorithm to analyze the matching degree in terms of rhythm, judge the speed of the dance movements and the degree of agreement with the standard movements, and obtain a rhythm matching result.

[0102] According to the continuity detection result and the rhythm matching result, a fluency score is obtained according to a set fluency scoring rule. For example, a weight can be set for each index, and a weighted average method is used to obtain the final score. The score range can be 0 to 1, and the closer to 1 indicates that the dance movement is more fluent and the rhythm matching degree is high. In some embodiments, improvement suggestions can also be generated according to the fluency score, such as appropriately reducing the action pause time or adjusting the action speed, to help the dancer improve the continuity and rhythm of the dance.

[0103] In other embodiments of the present application, for the dance training scene, in order to evaluate the emotional expression and artistic appeal of the dance, the above-mentioned visual capture and evaluation method of human body movement further includes: S501. According to the human body movement video, the facial expression and emotion of the target object are identified; S502. According to the facial expression and emotion, an emotional expression score is obtained.

[0104] Specifically, the human body movement video is specifically a dance video of the target object, i.e. the dancer. According to the dance video, the facial expression and emotion of the dancer are identified and compared with the set standard facial expression and emotion, and an emotional expression score is obtained according to the comparison result and the set emotional expression scoring rule. The facial expression and emotion recognition can be realized by computer vision technology, especially a facial expression recognition model based on deep learning, such as OpenCV combined with a convolutional neural network or a pre-trained facial expression recognition model (such as a model on the FER-2013 dataset), to analyze each frame in the dance video, extract facial features and identify the emotional state (such as happy, sad, surprised, etc.) of the dancer, and obtain the facial expression and emotion of the dancer. Then the facial expression and emotion of the dancer are compared with the standard facial expression and emotion, and a similarity calculation method (such as cosine similarity or Euclidean distance) can be used to measure the difference between the two, to obtain the comparison result. The emotional expression score can be calculated according to the comparison result and the preset scoring rule, for example, if the facial expression matches the standard emotion well, the score is close to 1, and if the matching degree is low, the score is reduced.

[0105] In some embodiments, according to the emotional expression score, prompt information can also be given to help the dancer perform the dance more vividly. For example, according to the matching degree between the facial expression and emotion of the dancer and the standard facial expression and emotion, specific prompt information can be provided, such as suggesting the dancer to enhance certain emotional expression to improve the expressiveness and artistic appeal of the dance.

[0106] In other embodiments of the present application, for the dance training scene, in order to realize systematic comments on the performance of the dancer and the generation of personalized training plans, the above-mentioned visual capture and evaluation method of human body movement further includes: S601. Generating a multi-dimensional score and an evaluation report according to a human action video and a preset visual generative pre-training model.

[0107] S602. Obtaining historical training data of a target object, and generating a training plan of the target object according to the historical training data and the visual generative pre-training model.

[0108] The human action video is specifically a dance video of the target object, the historical training data is specifically historical dance training data of the dancer, the training plan is specifically a dance training plan of the dancer, and the visual generative pre-training model can be a large model such as GPT-4-vision. Through the multi-modal analysis capability of the GPT-4-vision model, detailed comments are made from multiple dimensions such as accuracy, fluency, and rhythm of the action, a multi-dimensional score and an evaluation report are obtained, and based on the historical dance training data of the dancer, personalized dance training calculations are provided for the dancer to help the dancer improve dance performance in all directions.

[0109] Specifically, the GPT-4-vision model extracts the dancer's limb posture, action continuity, rhythm change and other features from the dance video segment through a visual processing module, analyzes the dancer's performance in accuracy, fluency, rhythm and emotional expression, and compares and analyzes the dancer's action features with the set standard action. The GPT-4-vision model can identify the dancer's action errors and score by dimension, give multi-dimensional scores such as "action accuracy", "fluency", "rhythm mastery", and generate a multi-dimensional evaluation report according to the score results, detailing the dancer's strengths and weaknesses in each action. For example, "arm movements are in place, footwork needs further adjustment", "rhythm mastery is good, but emotional expression is weak". The dancer can view the evaluation report after each action and make action adjustments under the guidance of the GPT-4-vision model to help the dancer steadily improve in actual practice.

[0110] Further, the historical dance training data of the dancer is collected and analyzed by the GPT-4-vision model to generate a targeted dance training plan. The historical dance training data can be data generated by the dancer during the use of the dance training software, for example, using the GPT-4-vision model to generate stage-by-stage practice recommendations based on the dancer's historical dance performance, listing daily training tasks and segmented practice focuses. Each recommendation contains detailed action guidance such as "arm posture correction" and "center of gravity maintenance", and actual practice goals such as "practice footwork stability for 10 minutes". In subsequent practice, the GPT-4-vision model will dynamically adjust the dance training plan according to the dancer's progress.

[0111] In specific embodiments, taking the dance training scenario as an example, the above-mentioned visual capture and evaluation method of human body action can be applied to smart terminals such as mobile phones, to record and split-screen compare based on smart terminals, and perform offline analysis.

[0112] Specifically, as shown in Figure 2 After the dancer finishes dancing, the dancer uses a mobile phone to record a video. The mobile phone camera captures the dancer's movements and stores the complete dance video. The standard action video of the dance teacher is pre-stored on the mobile phone for subsequent comparison and analysis. The evaluation method of dance action pre-installed in the mobile phone is used to perform offline 3D reconstruction and human body posture detection on the recorded dance video. The process includes the following steps: detecting 2D human joint positions from each frame of the dance video through a first Visual Transformer model, and generating a 2D human skeleton; using a Transformer decoder to predict 3D human posture data and construct a 3D human model, comparing the 3D human model with the set standard action, obtaining the comparison result, and obtaining the dance action score according to the comparison result.

[0113] In addition, the videos of the dancer and the dance teacher can also be split-screen presented on the mobile phone screen, and compared left and right to enhance visibility. In split-screen comparison, color markers are used to mark the action differences between the dancer and the teacher. Red markers can be set to represent large deviations, yellow markers to represent small deviations, and green markers to represent actions that meet the standard. During playback, the score and error points will be displayed frame by frame, allowing the dancer to observe the improvement space of the action at a glance. Finally, an evaluation report of each dance action is automatically generated based on the action comparison result, including action consistency, gesture accuracy, rhythm, etc. The evaluation report highlights the score, deviation points, and improvement suggestions for each action. The dancer can read the evaluation report in detail after the practice to focus on the improvement items in the next practice. The above-mentioned embodiments are suitable for use in scenarios where the practitioner wants to review the action performance after dancing and gradually optimize it. The mobile phone can be used directly for shooting without connecting devices, which is convenient for playback and frame-by-frame analysis of action differences, and is suitable for high-precision offline analysis.

[0114] In other embodiments, the above-mentioned visual capture and evaluation method of human body action can also be applied to smart wearable devices such as XR glasses, to perform real-time action analysis based on smart wear.

[0115] Specifically, as shown in Figure 3 As shown, the dancer wears XR glasses to enter the dancing environment. The camera on the XR glasses captures the dancer's dancing movements in real time, providing a full-body view of the dancer. On the built-in computing unit in the XR glasses, the evaluation method of the above dancing movements is run to realize real-time reconstruction of 3D posture and dynamic capture of joint position and rotation angle. The standard posture of the dance teacher can be projected in the XR view in real time as a transparent virtual guide image (ghost figure), which is displayed synchronously with the dancer's posture. The dancer can clearly see the comparison between his / her own movements and the dance teacher's movements in the XR view. The XR glasses compare the joint rotation angles and positions of the dance teacher and the dancer, and display real-time prompts (such as "Please raise your arms slightly" or "Keep your foot position") in the view when the dancer's movements deviate, helping the dancer to adjust immediately.

[0116] In addition, the error value of the joint position can also be analyzed, and the parts exceeding the certain deviation threshold will be color-labeled in real time (for example, the joint parts are displayed in red to indicate large errors). Dynamic prompts are given in the XR view to enable the dancer to immediately perceive the deficiencies in the movements. Feedback can be automatically generated at the end of each movement, such as "The arm is not raised enough, try to raise the angle again" or "The movement is in place". The overall score and improvement suggestions are displayed at the end, such as the summary suggestions for the overall performance of the movement, as well as the comprehensive evaluation of multiple dimensions such as fluency, rhythm, and emotional expression. The dancer can choose to continue practicing in the XR environment or review after exiting the XR environment. The above embodiments are suitable for situations that require real-time correction of movements and rapid improvement of movement standards. They are particularly suitable for beginners and dancers who need to practice frequently. Through real-time feedback, the efficiency of dancing can be greatly improved, and the dancer can immediately correct the movements, which helps to quickly optimize and stabilize the movements.

[0117] In other embodiments, the above visual capture and evaluation method of human body movements can also be applied to offline analysis and personalized report generation of multiple dancers based on the dance studio camera.

[0118] Specifically, as shown in Figure 4 using a single camera device in the dance studio, combined with a multi-target tracking algorithm, the dancing movements of multiple dancers in the same space are captured, analyzed offline, and fed back individually. Through posture detection, 3D reconstruction, and multi-target tracking technology, the movements of multiple dancers can be distinguished and tracked, and personalized feedback reports can be generated. After completing the practice, the dancers can view the detailed reports on the intelligent terminal of the dance studio, such as a tablet device, to help each dancer improve their movements in a targeted manner.

[0119] Among them, a single camera can be installed in a high position in front of the dance hall, covering the entire dance hall space, ensuring that the dance movements of every dancer in the field of view can be clearly captured. The standard movement video of the dance teacher is pre-stored on the tablet device as a reference standard for comparison by multiple dancers, providing a benchmark for subsequent frame-by-frame analysis. After video acquisition, the first VisualTransformer model is used to detect the 2D human joint positions of each dancer frame by frame, generating an independent 2D human skeleton for each dancer. This process ensures that different individuals' poses are distinguished in the same frame. Based on the 2D human skeleton of each dancer, a Transformer decoder is used to generate 3D human pose data for each dancer, including joint positions and rotation angles, and perform 3D human reconstruction to obtain the corresponding 3D human model for each dancer. Using the 4D human tracking algorithm in the paper, the pose information of multiple dancers is associated frame by frame, and the 3D position, motion features, and appearance information of the dancers are used to achieve multi-target tracking. In the case of multiple people occluding each other or their positions overlapping, the algorithm can still stably track the poses of each dancer. The 4D human tracking algorithm associates data from previous and subsequent frames, and can predict and fill in missing poses when multiple people overlap or perform rapid movements, ensuring the continuity of tracking and the integrity of the poses.

[0120] After 3D pose reconstruction and tracking are completed, the 3D human model of each dancer is compared frame by frame with the standard movements of the dance teacher, and the key joint positions, angles, and pose deviations of each dancer are calculated. According to the comparison results of each frame of movement, a dance movement score is generated for each dancer independently. The scoring dimensions include movement consistency, gesture accuracy, rhythm, and fluidity. By calculating the movement deviation and pose stability frame by frame, independent scoring data for multiple dancers is generated.

[0121] In addition, for each dancer, the parts with larger movement deviations will be recorded, and the error points of joint positions and poses will be marked, such as arm angle deviation, knee position error, etc. Based on the movement scores and deviation analysis of each dancer, an individualized evaluation report is automatically generated. Each evaluation report includes but is not limited to the following content: Dance movement score: Overall score of each dancer's movement consistency, fluidity, rhythm, etc.; Deviation point analysis: Detailed marking of each dancer's deviation points and shortcomings in specific movements; Improvement suggestions: Automatically generate individualized improvement suggestions for each dancer's specific errors, pointing out the parts and movements that need to be adjusted, such as "left hand needs to be raised", "squatting amplitude is insufficient", etc.

[0122] After the dancer's practice is finished, the evaluation report is automatically uploaded to the tablet device of the dance studio, so that each dancer can individually view his / her own performance. Each dancer can select his / her own report on the tablet device, and a simple interface is provided to display the scores, deviation analysis and improvement suggestions of each dancer, so that each dancer can quickly check the content of his / her own report. According to the generated evaluation report, the dancer can gradually improve specific movements and systematically improve the accuracy and consistency of the posture.

[0123] As shown in Figure 5 In another embodiment of the present application, a visual capture and evaluation system for human action is also provided, comprising: A first acquisition unit 10 is configured to acquire a human action video to be evaluated, and perform video frame decomposition on the human action video to obtain a plurality of single-frame images. A second acquisition unit 11 is configured to acquire a first Visual Transformer model constructed in advance, wherein the first Visual Transformer model is configured to detect 2D human joint positions from single-frame images. A detection unit 12 is configured to detect each single-frame image using the first Visual Transformer model to obtain the 2D human joint positions of each single-frame image. A generation unit 13 is configured to generate a 2D human skeleton corresponding to each single-frame image according to the 2D human joint positions of each single-frame image. A third acquisition unit 14 is configured to acquire a Transformer decoder constructed in advance, wherein the Transformer decoder is configured to take the 2D human skeleton as input to predict 3D human pose data. A processing unit 15 is configured to process the 2D human skeleton corresponding to each single-frame image using the Transformer decoder to obtain the 3D human pose data corresponding to each single-frame image. A reconstruction unit 16 is configured to perform 3D human reconstruction according to the 3D human pose data corresponding to each single-frame image to obtain a 3D human model corresponding to each single-frame image. An evaluation unit 17 is configured to compare the 3D human model corresponding to each single-frame image with a set of standard actions to obtain a comparison result, and obtain a human action score according to the comparison result.

[0124] In other embodiments of the present application, the 3D human pose data in the above-mentioned visual capture and evaluation system for human action includes 3D human joint positions, human joint rotation angles, movement amplitudes, speed characteristics and overall body postures.

[0125] In other embodiments of the present application, in the above-mentioned visual capture and evaluation system of human body actions, the reconstruction unit 16 is specifically configured to: reconstruct a 3D human body according to the 3D human body pose data corresponding to each single-frame image to obtain a 3D human body model corresponding to each single-frame image.

[0126] In other embodiments of the present application, in the above-mentioned visual capture and evaluation system of human body actions, the system further comprises a tracking unit configured to:

[0127] In other embodiments of the present application, in the above-mentioned visual capture and evaluation system of human body actions, the system further comprises: a fourth acquisition unit configured to acquire a pre-constructed MediaPipe Hands model, wherein the MediaPipe Hands model is configured to detect 2D hand joint positions from single-frame images; a hand detection unit configured to detect each single-frame image using the MediaPipe Hands model to obtain 2D hand joint positions of each single-frame image; a fifth acquisition unit configured to acquire a pre-constructed second Visual Transformer model, wherein the second Visual Transformer model is configured to take the 2D hand joint positions as input to predict hand rotation angles; a hand processing unit configured to process the 2D hand joint positions corresponding to each single-frame image using the second Visual Transformer model to obtain hand rotation angles corresponding to each single-frame image; a gesture evaluation unit configured to compare the 2D hand joint positions and the hand rotation angles corresponding to each single-frame image with a set of standard gesture actions to obtain a gesture comparison result, and obtain a gesture action score according to the gesture comparison result.

[0128] In other embodiments of the present application, in the above-mentioned visual capture and evaluation system of human body actions, the system further comprises: a sixth acquisition unit configured to acquire joint positions corresponding to each single-frame image, wherein the joint positions are 2D human body joint positions or 3D human body joint positions; a seventh acquisition unit, configured to acquire a pre-built comment generation model, wherein the comment generation model is configured to take joint positions as input and output human action comments, and the comment generation model adopts an end-to-end Transformer architecture, including: an embedding layer, an encoder position encoder, an encoder, a decoder position encoder, a decoder, and an output layer; The comment generation unit is used to generate a human action comment based on the joint position corresponding to each single frame image by using the comment generation model.

[0129] In other embodiments of the present application, in the above-mentioned human motion visual capture and evaluation system, the embedding layer is used to: Flatten the joint position corresponding to each single frame image to obtain the flattened coordinate vector ,in, Indicates the frame number of a single frame image; According to the flattened coordinate vector Each joint The corresponding coordinate vector , with each joint The corresponding learnable embedding vector Perform element-by-element multiplication to obtain the first embedding vector ; According to the flattened coordinate vector Each joint The corresponding coordinate vector , perform 1-dimensional convolution calculation to obtain the second embedding vector ; According to the first embedding vector and the second embedding vector , calculate the gating coefficient , the calculation formula is: ; Where, represents the activation function, represents the weight matrix, represents the bias term; According to the first embedding vector , the second embedding vector and gating coefficient , calculate each joint in each single frame image The corresponding final embedding vector , the calculation formula is: .

[0130] In other embodiments of the present application, in the above-mentioned human motion visual capture and evaluation system, the encoder position encoder is used to: The rotation position encoding and the relative position encoding are used to perform position encoding on each final embedding vector to obtain a corresponding first position encoding. The encoder is configured to: perform processing on each joint in each single-frame image a corresponding final embedding vector and a corresponding first position encoding, to obtain a first feature. The encoder comprises M encoder layers, wherein the first layer to the Lth layer of the encoder layers adopt a local self-attention mechanism, and the L+1th layer to the Mth layer of the encoder layers adopt a global multi-head self-attention mechanism.

[0131] In other embodiments of the present application, the visual capture and evaluation system of human action is as follows: The rotation position encoding and the relative position encoding are used to perform position encoding on each first feature to obtain a corresponding second position encoding. The decoder comprises N decoder layers, wherein each decoder layer comprises a masked self-attention layer, a cross-attention layer and a feedforward neural network, and the decoder is configured to: perform global average pooling on the first feature and the corresponding second position encoding to obtain an action intention vector; generate an outline skeleton of the comment using the masked self-attention layer and the feedforward neural network in the first K decoder layers according to the action intention vector and the historical output of the decoder; fuse the outline skeleton of the comment and the first feature through a gating fusion using the masked self-attention layer, the cross-attention layer and the feedforward neural network in the last N-K decoder layers to obtain an output vector of the decoder.

[0132] In other embodiments of the present application, the visual capture and evaluation system of human action is as follows:

[0133] In other embodiments of the present application, the visual capture and evaluation system of human action further comprises a fluency evaluation unit configured to: obtain a continuity detection result according to the human action video and a preset fluency detection model; obtain a rhythm matching result according to the human action video and a preset rhythm matching algorithm; obtain a fluency score according to the continuity detection result and the rhythm matching result.

[0134] In other embodiments of the present application, the visual capture and evaluation system of human action further comprises an emotional expression evaluation unit configured to: According to the human action video, the facial expression and emotion of the target object are recognized; According to the facial expression and emotion, a sentiment expression score is obtained.

[0135] In other embodiments of the present application, the visual capture and evaluation system of human action further comprises a comprehensive evaluation unit, which is configured to: According to the human action video and the preset visual generative pre-training model, a multi-dimensional score and an evaluation report are obtained.

[0136] In other embodiments of the present application, the visual capture and evaluation system of human action, the comprehensive evaluation unit is further configured to: Obtain historical training data of the target object; According to the historical training data and the visual generative pre-training model, a training plan of the target object is generated.

[0137] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / eos> < / eos>

Claims

1. A method of visual capturing and evaluating human motion, characterized by, The method comprises: acquiring a human action video to be evaluated, and performing video frame decomposition on the human action video to obtain a plurality of single-frame images; acquiring a first Visual Transformer model constructed in advance, wherein the first Visual Transformer model is configured to detect 2D human joint positions from the single-frame images; detecting each single-frame image using the first Visual Transformer model to obtain the 2D human joint positions of each single-frame image; generating a 2D human skeleton corresponding to each single-frame image according to the 2D human joint positions of each single-frame image; acquiring a Transformer decoder constructed in advance, wherein the Transformer decoder is configured to take the 2D human skeleton as input to predict 3D human pose data; processing the 2D human skeleton corresponding to each single-frame image using the Transformer decoder to obtain 3D human pose data corresponding to each single-frame image; performing 3D human reconstruction according to the 3D human pose data corresponding to each single-frame image to obtain a 3D human model corresponding to each single-frame image; comparing the 3D human model corresponding to each single-frame image with a set of standard actions to obtain a comparison result, and obtaining a human action score according to the comparison result.

2. The method as claimed in claim 1, characterized in that, The 3D human pose data includes 3D human joint positions, human joint rotation angles, action amplitudes, speed characteristics, and overall body postures.

3. The method as claimed in claim 1, wherein, The 3D human pose data includes 3D human joint positions, human joint rotation angles, action amplitudes, speed characteristics, and overall body postures. The 3D human pose data includes 3D human joint positions, human joint rotation angles, action amplitudes, speed characteristics, and overall body postures.

4. The method as claimed in claim 1, wherein, The method further comprises: acquiring a MediaPipe Hands model constructed in advance, wherein the MediaPipe Hands model is configured to detect 2D hand joint positions from the single-frame images; detecting each single-frame image using the MediaPipe Hands model to obtain the 2D hand joint positions of each single-frame image; acquiring a second Visual Transformer model constructed in advance, wherein the second Visual Transformer model is configured to take the 2D hand joint positions as input to predict hand rotation angles; processing the 2D hand joint positions corresponding to each single-frame image using the second Visual Transformer model to obtain hand rotation angles corresponding to each single-frame image; The 2D hand joint positions and the hand rotation angle corresponding to each of the single-frame images are compared with a set of standard gesture actions to obtain a gesture comparison result, and a gesture action score is obtained according to the gesture comparison result.

5. The method as claimed in claim 2, wherein, Also comprising: obtaining the joint positions corresponding to each of the single-frame images, wherein the joint positions are the 2D human body joint positions or the 3D human body joint positions; obtaining a pre-constructed comment generation model, wherein the comment generation model is configured to take the joint positions as input and output human action comments, and the comment generation model adopts an end-to-end Transformer architecture, including an embedding layer, an encoder position encoder, an encoder, a decoder position encoder, a decoder, and an output layer; using the comment generation model to generate human action comments according to the joint positions corresponding to each of the single-frame images.

6. The method as claimed in claim 5, characterized in that, The embedding layer is used to: The joint positions corresponding to each of the single-frame images are flattened to obtain flattened coordinate vectors wherein, denotes the frame number of the single-frame images. According to the flattened coordinate vector Each joint The corresponding coordinate vector Each joint The corresponding learnable embedding vector Element-wise multiplication is performed to obtain a first embedding vector ; According to the flattened coordinate vector Each joint in the middle The corresponding coordinate vector , a 1-dimensional convolution calculation is performed to obtain a second embedding vector ; According to the first embedding vector and the second embedding vector , a gating coefficient is calculated, and the calculation formula is: ； wherein denotes an activation function, denotes a weight matrix, denotes a bias term; According to the first embedding vector , the second embedding vector and the gating coefficient , calculate each joint in each single frame image The corresponding final embedding vector , the calculation formula is: 。 7. The method as claimed in claim 6, characterized in that The encoder position encoder is used to: performing position encoding on each of the final embedding vectors to obtain a corresponding first position encoding; and performing position encoding on each of the final embedding vectors to obtain a corresponding first position encoding; and The encoder is used to: for each joint in each of the single-frame images corresponding final embedding vectors and the corresponding first positional encodings, to obtain first features; The encoder includes M encoder layers, wherein the first L layers of the encoder layers adopt a local self-attention mechanism, and the L+1 to M layers of the encoder layers adopt a global multi-head self-attention mechanism.

8. The method as claimed in claim 7, characterized in that, The decoder position encoder is used to: The first feature is position encoded using rotation position encoding and relative position encoding to obtain a corresponding second position encoding. The decoder includes N decoder layers, wherein each of the decoder layers includes a masked self-attention layer, a cross-attention layer, and a feedforward neural network, and the decoder is used to: perform global average pooling on the first feature and the corresponding second position encoding to obtain an action intent vector; use the masked self-attention layer and the feedforward neural network in the first K decoder layers to generate a comment outline skeleton according to the action intent vector and the historical output of the decoder; fuse the comment outline skeleton and the first feature through gated fusion using the masked self-attention layer, the cross-attention layer, and the feedforward neural network in the last N-K decoder layers to obtain an output vector of the decoder.

9. The method of claim 8, wherein the output layer is used to map the output vector of the decoder to a human action comment.

10. A visual capturing and evaluation system of human motion, characterized by, comprising: a first obtaining unit configured to obtain a human action video to be evaluated, and perform video frame decomposition on the human action video to obtain a plurality of single-frame images; a second obtaining unit configured to obtain a pre-constructed first Visual Transformer model, wherein the first Visual Transformer model is configured to detect 2D human body joint positions from the single-frame images; a detection unit configured to use the first Visual Transformer model to detect each of the single-frame images to obtain 2D human body joint positions of each of the single-frame images; a generation unit configured to generate a 2D human body skeleton corresponding to each of the single-frame images according to the 2D human body joint positions of each of the single-frame images; and a comment generation unit configured to use a comment generation model to generate human action comments according to the 2D human body joint positions corresponding to each of the single-frame images. a third obtaining unit, configured to obtain a pre-constructed Transformer decoder, wherein the Transformer decoder is configured to take the 2D human skeleton as input to predict 3D human pose data; a processing unit, configured to process the 2D human skeleton corresponding to each single-frame image by using the Transformer decoder to obtain 3D human pose data corresponding to each single-frame image; a reconstruction unit, configured to perform 3D human reconstruction according to the 3D human pose data corresponding to each single-frame image to obtain a 3D human model corresponding to each single-frame image; an evaluation unit, configured to compare the 3D human model corresponding to each single-frame image with a set standard action to obtain a comparison result, and obtain a human action score according to the comparison result.

Images