Method for constructing data set for action quality assessment

Abstract

The invention discloses a method for constructing a data set for action quality assessment. The method may comprise: providing a plurality of videos collected from a plurality of perspectives for a scene to be subjected to motion quality assessment, the plurality of videos being subjected to synchronization time calibration; estimating camera parameters of a plurality of cameras for acquiring the plurality of videos; determining a two-dimensional pose of the athlete from a plurality of video frames of the plurality of videos; determining a three-dimensional posture of the athlete based on the two-dimensional posture and the camera parameters; and combining the plurality of video frames and the three-dimensional pose to form a data set.

Description

Technical Field

[0001] The present invention relates generally to the field of artificial intelligence, and in particular to motion quality assessment. Background Art

[0002] Action quality assessment has important applications in many fields such as sports, fitness, and entertainment. Unlike action recognition tasks that only need to identify the category of an action, action quality assessment tasks require in-depth analysis of the subtle differences in action execution, including the speed, intensity, and timing of the action, which makes the task more complex and challenging.

[0003] Existing datasets for action quality assessment mainly cover diving (see [1] Paritosh Parmar and Brendan Morris. Action Quality Assessment Across Multiple Actions. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1468–1476, 2019. doi: 10.1109 / WACV.2019.00161; [2] Paritosh Parmar and Brendan Tran Morris. Learning to Score Olympic Events. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017; [3] Paritosh Parmar and Brendan Tran Morris. What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pages 304–313, 2019; [4] Jinglin Xu, Yongming Rao, Xumin Yu, Guangyi Chen, Jie Zhou, and Jiwen Lu. FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment. In 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2939–2948. IEEE, 2022), figure skating (see [5] Shunli Wang, Dingkang Yang, Peng Zhai, Chixiao Chen, and Lihua Zhang.TSA-Net: Tube Self-Attention Network for Action Quality Assessment. In Proceedings of the 29thACM International Conference on Multimedia, pages 4902–4910, 2021; [6] JingfeiXia, Mingchen Zhuge, Tiantian Geng, Shun Fan, Yuantai Wei, Zhenyu He, and FengZheng. Skating-mixer: Long-term sport audio-visual modeling with MLPs.InProceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of ArtificialIntelligence and Thirteenth Symposium on Educational Advances in ArtificialIntelligence,volume37of AAAI'23 / IAAI'23 / EAAI'23,pages 2901–2909.AAAIPress, 2023) and other fields. Typically, these datasets only provide single-view competition video recordings and their corresponding scores, but lack the support of detailed scoring process and multimodal data, resulting in significant limitations in the generalization and adaptability of the trained models. In addition, although the motion quality assessment task is essentially closely related to human motion, none of the existing datasets contain posture data.

[0004] Computers can accurately identify and track objects by analyzing images and videos, and have been widely used in various fields. In sports analysis, computer vision technology can provide an efficient and objective means to support the monitoring and analysis of athletes' performance. Specifically, the key technology of human posture detection and tracking based on computer vision makes it possible to extract athletes' human posture from videos for motion quality assessment. Some studies consider extracting two-dimensional (2D) posture features from single-view videos, but these features are often limited by the movement of the camera's viewpoint and the lack of the camera's true position, resulting in the ambiguity of the estimated three-dimensional (3D) posture attributes, which not only affects the posture estimation, but also affects the accuracy of the results of motion quality assessment.

[0005] There is a need in the art for a method to provide a dataset that is more helpful for training a model to better perform motion quality assessment tasks. Summary of the invention

[0006] The present invention is provided in order to provide a dataset that is more helpful for training a model to better perform motion quality assessment tasks.

[0007] According to one aspect of the present invention, a method for constructing a data set for evaluating the motion quality of an athlete's movements is provided, comprising: providing a plurality of videos captured from a plurality of perspectives for a scene for which motion quality evaluation is to be performed, wherein the plurality of videos are synchronized and time-calibrated; estimating camera parameters of a plurality of cameras used to capture the plurality of videos; determining a two-dimensional posture of the athlete from a plurality of video frames of the plurality of videos; determining a three-dimensional posture of the athlete based on the two-dimensional posture and the camera parameters; and combining the plurality of video frames and the three-dimensional posture to form the data set.

[0008] In the method as described above, providing multiple videos captured from multiple perspectives for a scene to be evaluated for motion quality comprises: capturing multiple videos for a scene to be evaluated for motion quality from the multiple perspectives; and aligning the multiple videos in the time domain.

[0009] A method as described in any of the above items, wherein aligning the multiple videos in the time domain includes: playing music when the multiple cameras capture the multiple videos; determining the offsets of the multiple audios respectively captured by the multiple cameras; aligning the multiple audios in the time domain based on the offsets; and aligning the multiple videos in the time domain by aligning the multiple audios.

[0010] A method as described in any of the above items, wherein determining the offsets of multiple audios respectively captured by the multiple cameras includes: determining the offset of the first audio in the second audio by maximizing the cross-correlation of the standardized Mel-frequency cepstral coefficients of the first audio and the second audio in the multiple audios.

[0011] A method as described in any of the above items, wherein the camera parameters include camera intrinsic parameters, and estimating the camera parameters of multiple cameras used to capture the multiple videos includes: calibrating the camera intrinsic parameters of the multiple cameras by photographing a moving array-arranged chessboard.

[0012] A method as described in any of the above items, wherein the camera parameters include camera extrinsic parameters, and estimating the camera parameters of multiple cameras used to capture the multiple videos includes: marking marker points with known positions and distance relationships in an annotation system; and estimating the camera extrinsic parameters of the multiple cameras based on three-dimensional object points and two-dimensional image points using a PnP algorithm.

[0013] A method as described in any of the above items, wherein determining the two-dimensional posture information of an athlete from multiple video frames of the multiple videos includes: using a target detection algorithm to identify all detectable human targets in the video frames; generating a bounding box sequence of the athlete using a two-dimensional tracking algorithm for the detected human targets; and using two-dimensional posture estimation on the images within the bounding boxes within the bounding box sequence to determine the two-dimensional posture of the athlete.

[0014] A method as described in any of the above, wherein determining the three-dimensional pose of the player based on the two-dimensional pose and the camera parameters includes: performing triangulation to determine the three-dimensional pose of the player based on the two-dimensional pose of the player and the camera parameters.

[0015] A method as described in any of the above items, wherein determining the three-dimensional posture of the athlete based on the two-dimensional posture and the camera parameters also includes: estimating the position range of the athlete appearing in the multiple video frames at the next moment through back projection technology based on the determined three-dimensional posture of the athlete at the current moment and the camera parameters; skipping target detection within the position range and directly estimating the two-dimensional posture of the athlete at the next moment; and skipping the matching task and directly performing triangulation on the athlete for the estimated two-dimensional posture of the athlete at the next moment.

[0016] The method as described in any of the above items further includes: labeling at least a portion of the data set to form a hierarchical data set including multiple levels.

[0017] A method as described in any of the above items, wherein labeling at least a portion of the data set includes: labeling at least some video frames with course information to form a data set including a course hierarchy, wherein the course information indicates a difficulty level of an action to be evaluated.

[0018] A method as described in any of the above, wherein the dataset is organized in hierarchical order according to difficulty level for training on performing an action quality assessment task.

[0019] A method as described in any of the above, wherein the course information is recorded when recording the at least some video frames.

[0020] A method as described in any of the above items, wherein labeling at least a portion of the data set further comprises: labeling action information for at least some video frames to form a data set including an action hierarchy, wherein each course includes multiple actions.

[0021] A method as described in any of the above items, wherein labeling action information for at least some video frames comprises: dividing the at least some video frames into multiple video segments, wherein each video segment corresponds to an action; and assigning an action category to each video segment based on course information.

[0022] A method as described in any of the above items, wherein labeling action information for at least some video frames also includes: for each action in the video frame, scoring the action by comparing the athlete's action in the video frame with a demonstration action, wherein the demonstration action is at least partially based on a scoring standard for the sport.

[0023] A method as described in any of the above items, wherein scoring the action includes: determining a deduction score based on the execution quality reflected by the comparison result of the athlete's action in the video and the demonstration action.

[0024] A method as described in any of the above items, wherein each action is assigned multiple penalty items, wherein each penalty item indicates a corresponding execution error, and each penalty item is assigned multiple error severity scores, and determining the penalty scores includes: for each action, based on the comparison result between the athlete's action in the video and the demonstration action, determining the error severity score of each penalty item; and accumulating the error severity scores of each penalty item to determine the penalty score.

[0025] A method as described in any of the above items, wherein labeling at least a portion of the data set further includes: labeling at least some video frames with motion event information to form a data set including a motion event hierarchy, wherein the motion events are associated with the sports event to which the action to be evaluated belongs, wherein each motion event includes multiple courses.

[0026] A method as described in any above, wherein the motion event information is recorded when recording the at least some video frames.

[0027] According to another aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the above methods.

[0028] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0029] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0030] The data set obtained according to the embodiment of the present invention can enable the model to better understand the action when training the model to perform action quality assessment, thereby achieving better task execution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Embodiments of the present invention are described with reference to the accompanying drawings, in which:

[0032] Figure 1 A flowchart showing a method for constructing a dataset for motion quality assessment according to some embodiments of the present invention.

[0033] Figure 2 A flow chart illustrating a process of providing multiple videos of a scene to be evaluated for motion quality according to some embodiments of the invention.

[0034] Figure 3 Shown is an example scene and camera layout for capturing multiple videos for a floor exercise sport.

[0035] Figure 4 A flow chart illustrating a process of estimating camera parameters according to some embodiments of the invention.

[0036] Figure 5 A flow chart illustrating a process of determining a player's 2D pose according to some embodiments of the invention.

[0037] Figure 6 A flow chart illustrating a process of determining a player's 3D pose according to some embodiments of the invention.

[0038] Figure 7 is a schematic diagram of estimating a 2D pose sequence and reconstructing a 3D pose sequence from multiple videos of multiple perspectives according to some embodiments of the present invention.

[0039] Figure 8 A flow chart illustrating a process of annotating at least a portion of a data set according to some embodiments of the present invention.

[0040] Fig. 9 A flow chart illustrating a process for scoring actions according to some embodiments of the invention.

[0041] Fig.10 is a schematic diagram of a labeled hierarchical dataset according to some embodiments of the present invention.

[0042] Fig.11 Shown is a layout of an annotation pipeline and system for annotating a data set according to some embodiments of the present invention.

[0043] Fig.12 The interactive interface layout of a labeling tool evaluation system for scoring actions according to some embodiments of the present invention is shown.

[0044] Fig.13 A comparative regression workflow with an additional score reduction head is shown according to some embodiments of the present invention.

[0045] Fig.14 A block diagram showing a computer device according to some embodiments of the invention is shown.

[0046] Fig.15 A block diagram illustrating a computer-readable storage medium according to some embodiments of the invention.

[0047] Fig.16 A block diagram illustrating a computer program product according to some embodiments of the invention. DETAILED DESCRIPTION

[0048] The present invention is generally applicable to sports that can score the quality of execution of movements (e.g., based on the scoring criteria of the sport), including but not limited to gymnastics, diving, figure skating, etc.

[0049] In the implementation process of the embodiment of the present invention, the athletes are informed that the entire process of their actions will be filmed before they perform the action, and the filming process is subject to the consent of the athletes. In the implementation process of the embodiment of the present invention, the faces of the athletes in the collected videos are blurred as necessary, so that the faces cannot be recognized in the processed videos, and the collected videos containing faces are not further disseminated.

[0050] According to one aspect of the present invention, a method for constructing a data set for evaluating the movement quality of an athlete's movements is provided.

[0051] Figure 1A flow chart of a method 100 for constructing a dataset for motion quality assessment according to some embodiments of the present invention is shown.

[0052] The method 100 may include: at block 110, providing a plurality of videos captured from a plurality of viewing angles for a scene to be subjected to motion quality assessment, wherein the plurality of captured videos are synchronized and time-calibrated. Figure 2 , Figure 3 The specific implementation of the step at box 110 is described. By providing multiple synchronized time-calibrated videos of the same scene, the 3D posture of the athlete in a complex sports scene can be captured more accurately, and the influence of potential occlusions and interferences on the recognition of athletes in the single-view video method can be overcome.

[0053] The method 100 may include, at block 120, estimating camera parameters of a plurality of cameras used to capture a plurality of videos. Figure 4 A specific implementation of the step at block 120 is described.

[0054] Method 100 may include, at block 130, determining a 2D pose of a player from a plurality of video frames of a plurality of videos. Figure 5 A specific implementation of the steps at block 130 is described.

[0055] Method 100 may include, at block 140, determining a 3D pose of the player based on the 2D pose and camera parameters. Figure 6 A specific implementation of the steps at block 140 is described.

[0056] The method 100 may include: at block 150, combining multiple video frames and 3D poses to form a data set, wherein the data set can be used to perform motion quality assessment. By combining multiple videos (e.g., RGB videos) and 3D pose data from multiple perspectives, a richer data set is provided for the motion quality assessment task, thereby improving the effect of the model performing motion quality assessment.

[0057] Optionally, the method 100 may include: at block 160, annotating at least a portion of the data set to form a hierarchical data set including a plurality of levels. Figure 8-Figure 12 A specific implementation of the steps at block 160 is described.

[0058] Figure 2 Flowchart showing a process 200 for providing multiple videos of a scene for motion quality assessment according to some embodiments of the present invention. The process 200 may be Figure 1 The specific implementation of the step at box 110 of method 100 in FIG. 1 is described below, but the scope of the present invention is not limited thereto.

[0059] Process 200 may include, at block 210, collecting multiple videos of a scene for which motion quality assessment is to be performed from multiple perspectives. Figure 3 Describes the example scene and camera layout.

[0060] In some high-intensity sports, athletes perform multiple postures, such as lying, crouching, rolling, and fast jumping. These actions involve a lot of self-occlusion and fast movement. The resulting complex scenes bring huge adjustments to the accurate inference of 3D postures from conventional single-view RGB sensors or depth sensors, which seriously affects the performance of motion quality assessment.

[0061] In some embodiments, a multi-view visual motion capture system may be used to perform the steps at block 210 to capture high-quality multi-view video. As an example, the system may use a DJI Osmo Action 3 camera. As an example, the camera may operate at a 4096×4096 (4K) resolution and 60 frames per second (fps) mode.

[0062] In order to reconstruct the 3D point position based on the correspondence between the 2D projection points of the human body in videos of multiple perspectives at the same moment, it is necessary to ensure the time synchronization between the multiple videos.

[0063] Process 200 may include, at block 220 , aligning a plurality of videos in the time domain.

[0064] In some embodiments, the step at block 220 may include: at block 221, playing music when the multiple cameras capture the multiple videos. As an example, the music may be music that needs to be played when a sport (e.g., rhythmic gymnastics, figure skating, etc.) performs an action. Alternatively, music may be played specifically to perform this step.

[0065] In some embodiments, the step at box 220 may include: determining the offset of multiple audios respectively captured by multiple cameras at box 222. In some embodiments, the step at box 222 may include: determining the offset of the first audio in the second audio by maximizing the cross-correlation of the standardized Mel Frequency Ceptral Coefficients (MFCC) of the first audio and the second audio in the multiple audios.

[0066] In some embodiments, the step at block 220 may include, at block 223 , aligning the plurality of audios in the time domain based on the offset.

[0067] In some embodiments, the step at block 220 may include, at block 224 , aligning the plurality of videos in time domain by aligning the plurality of audios.

[0068] In some embodiments, the step at block 220 may be performed serially with the step at block 210. For example, the step at block 210 may be performed first, and then the step at block 220. Alternatively, the step at block 220 may be performed first, and then the step at block 210.

[0069] In some embodiments, the steps at block 220 may be performed at least partially in parallel with the steps at block 210. For example, the steps at block 210 may be performed first, the music is played while the athlete does not perform an action; then, the steps at block 220 are performed, so that the videos captured by the multiple cameras are synchronized in the time domain; thereafter, the athlete starts to perform an action, thereby ensuring that the multiple videos captured from the time the athlete starts to perform the action are synchronized in time.

[0070] In some embodiments, a wired connection may be used to connect multiple cameras instead, thereby ensuring that the videos captured by each camera are synchronized in the time domain through a wired connection protocol without executing the steps in blocks 221 to 224 .

[0071] Figure 3 An example scene 300 and camera layout for capturing multiple videos of floor exercise is shown. The example scene 300 may be applied to the above-mentioned Figure 2 The steps at block 210 of process 200 are described, but the scope of the present invention is not limited in this regard.

[0072] In the example scene 300, an athlete 310 performs a floor exercise on a field 320. A plurality of cameras are arranged at positions P01, P02, P03, P04, P05, P06, P07, and P08 around the perimeter of the field 320. The video frames of the videos captured by each camera are shown in the figure.

[0073] Figure 3 The number and position of the cameras shown in the figure are exemplary only. In other embodiments, a greater or lesser number of cameras may be used, and the cameras may be arranged in different positions.

[0074] Figure 4 400 according to some embodiments of the present invention. Figure 1 Specific implementations of the steps at block 120 of method 100 are described, but the scope of the present invention is not limited thereto.

[0075] In some embodiments, the camera parameters may include camera intrinsic parameters, which can be used to eliminate image distortion.

[0076] In some embodiments, process 400 may include: estimating camera intrinsic parameters at block 410. In some embodiments, the step at block 410 may include: calibrating camera intrinsic parameters of multiple cameras by shooting a moving array-arranged chessboard at block 415. As an example, the EasyMocap tool may be used to calibrate the intrinsic parameters of each camera by shooting a video of a moving 7x9 chessboard.

[0077] In some embodiments, the camera parameters may include camera extrinsics, which may be used for spatial alignment between multiple cameras.

[0078] In some embodiments, process 400 may include: estimating camera extrinsics at block 420. In some embodiments, the step at block 420 may include: annotating marker points with known positions and distance relationships in an annotation system at block 421. The step at block 420 may also include: estimating camera extrinsics of multiple cameras using a PnP algorithm based on 3D object points and 2D image points at block 423, thereby establishing a 2D to 3D mapping.

[0079] Figure 5 A flowchart showing a process 500 for determining a 2D pose of a player according to some embodiments of the present invention. The process 500 may be a process of combining the above Figure 1 Specific implementations of the steps at block 130 of method 100 are described, but the scope of the present invention is not limited thereto.

[0080] Process 500 may include, at block 510 , identifying all detectable human objects in a video frame using an object detection algorithm.

[0081] Process 500 may include generating a sequence of bounding boxes of the player using a 2D tracking algorithm for the detected human target at block 530. Optionally, in some embodiments, the image within the bounding box may be cropped and enlarged.

[0082] Process 500 may include, at block 550 , using 2D pose estimation on images within bounding boxes within the sequence of bounding boxes to determine a 2D pose of the player.

[0083] Figure 6 A flowchart showing a process 600 for determining a 3D pose of a player according to some embodiments of the present invention. The process 600 may be a combination of the above Figure 1 Specific implementations of the steps at block 140 of method 100 are described, but the scope of the present invention is not limited thereto.

[0084] Process 600 may include, at block 610 , performing triangulation to determine a 3D pose of the player based on the 2D pose of the player and camera parameters.

[0085] In some embodiments, after the step at block 610 is performed to determine the 3D pose of the player at the current moment, the steps at blocks 620 , 630 , and 640 described below may be performed.

[0086] In some embodiments, process 600 may include: at block 620, estimating a position range where the player appears in a plurality of video frames at a next moment based on the player's 3D pose and camera parameters by back-projection technology.

[0087] In some embodiments, process 600 may include: at block 630, skipping target detection within the estimated position range and directly estimating the 2D pose of the player at the next moment. By utilizing the continuity of the player's movements, target detection at the next moment can be skipped, thereby shortening the time required for 2D pose estimation and even the entire 3D pose estimation.

[0088] In some embodiments, process 600 may include: at block 640, skipping the matching task and directly performing triangulation on the player for the estimated 2D pose of the player at the next moment. By directly performing triangulation on the player for the newly generated single-person 2D pose estimation result based on the consistency of the projection area of ​​the player at different viewing angles in the triangulation link, it is avoided to enumerate the matching between multiple detection results at different viewing angles, thereby further improving the matching accuracy, significantly shortening the matching time, and further shortening the time required for 3D pose estimation.

[0089] In contrast to the traditional top-down "target detection-target tracking-pose recognition" process with high false positive rate, large consumption of computing resources, slow computational reasoning speed, long matching time and high matching error rate in the multi-view triangulation process, the process 600 (especially the steps at blocks 620, 630 and 640) according to the embodiment can effectively improve the accuracy of 3D pose estimation and significantly shorten the time required for 3D pose estimation.

[0090] Figure 7 is a schematic diagram of estimating a 2D pose sequence and reconstructing a 3D pose sequence from multiple videos of multiple perspectives according to some embodiments of the present invention.

[0091] Specifically, Figure 7 Video frames 710, 720, 730, and 740 are shown for a scene of performing a back-roll straight-leg start maneuver captured from multiple perspectives. A 2D pose of the athlete can be estimated based on video frames 710, 720, 730, and 740, and a 3D pose 750 of the athlete can be determined eventually.

[0092] Figure 8FIG. 8 is a flowchart showing a process 800 for annotating at least a portion of a data set according to some embodiments of the present invention. Figure 1 Specific implementations of the steps at block 160 of method 100 are described, but the scope of the present invention is not limited thereto.

[0093] In some embodiments, the resulting data set (e.g., by executing Figure 1 The method 100 of the method 100 may be used to annotate the data set so that the model can be well trained with the annotated data set to better perform the action quality assessment task. In some embodiments, at least a portion of the data set may be annotated to form a hierarchical data set including multiple levels. Finer-grained hierarchical data annotations may be better used to train the model to perform the action quality assessment task.

[0094] In some embodiments, process 800 may include, at block 810 , labeling at least some video frames with motion event information to form a data set including a motion event hierarchy, wherein the motion event is associated with a sport to which the action to be evaluated belongs.

[0095] As an example, the sports event information may include specific gymnastics events, such as men's events: floor exercise, pommel horse, rings, vaulting horse, parallel bars, horizontal bar; women's events: floor exercise, vaulting horse, uneven bars, balance beam. As an example, the sports event information may also include diving events, figure skating events, etc.

[0096] In some embodiments, the sports event information may be recorded when recording at least some video frames. For example, when capturing a sports video of an athlete, it is known which sport the athlete is going to perform, and the sports event information may be recorded by the annotator when recording the video frames.

[0097] In some embodiments, process 800 may include: at block 830, labeling at least some video frames with course information to form a dataset including a course hierarchy, wherein the course information indicates the difficulty level of the action to be evaluated. The inventors have found through experiments that a model trained with a dataset including a course hierarchy performs better than a model trained with a general dataset in performing action quality assessment tasks, which will be discussed in conjunction with the following. Fig.13 The experimental results described are described.

[0098] In some embodiments of the data set including the data set of the sports event level, a sports event may include multiple courses. As an example, this may correspond to a sports event may include multiple difficulties. In other embodiments, sports event information may not be included and only course information may be included (for example, this may correspond to the data set itself corresponding to a sports event).

[0099] In some embodiments, the data sets may be organized according to the order of difficulty levels for use in training models to perform the action quality assessment task. For example, the data sets may be organized according to the order of difficulty levels in ascending order. The inventors have found through experiments that using data sets corresponding to progressive difficulty levels to train models to perform the action quality assessment task can achieve better performance, which will be discussed in conjunction with the following. Fig.13 The experimental results described are described.

[0100] In some embodiments, the course information may be recorded when recording at least some video frames. For example, when capturing a sports video of an athlete, it is known what difficulty the athlete is going to perform, and the course information may be recorded by the annotator when recording the video frames.

[0101] In some embodiments, process 800 may include: at block 850, annotating at least some video frames with action information to form a data set including an action hierarchy, wherein each course may include multiple actions. This may correspond to a certain difficulty level of a certain sport may include multiple actions.

[0102] In some embodiments, the step at block 850 may include: at block 851, segmenting at least some of the video frames into a plurality of video segments, wherein each video segment may correspond to an action. As an example, the step at block 821 may segment the plurality of video frames into a plurality of video segments corresponding to a single action, respectively, by the annotator comparing the plurality of actions described in the scoring criteria for the sport.

[0103] In some embodiments, the step at block 850 may include assigning an action category to each video clip based on the course information at block 853. As an example, the action category may identify which action is at the difficulty level indicated by the corresponding course information. As an example, the action category may be annotated by an annotator against a plurality of actions described in a scoring standard for a sport.

[0104] In some embodiments, the steps at box 850 may include: at box 855, for each action in the video frame, scoring the action by comparing the athlete's action in the video frame to a demonstration action, wherein the demonstration action may be based at least in part on the scoring criteria of the sport.

[0105] By integrating the action scoring process into the standard process of the dataset, it provides process-aware scores for training models to perform action quality assessment tasks.

[0106] Sports such as gymnastics, diving, and figure skating usually set a full score for each difficulty level, and determine the final score by deducting points accordingly (sometimes combined with bonus points) based on the quality of execution during the action.

[0107] To be consistent with the above-described scoring process, in some embodiments, the step at box 855 may include: at box 856, determining a deduction score based on the quality of execution reflected by the comparison of the athlete's action in the video with the demonstration action. Thus, a deduction process-aware score consistent with the actual scoring process of the sport is provided for training the model to perform quality assessment tasks. In addition, by adopting deduction-based annotations, samples without deductions can be obtained for each action category, thereby ensuring the availability of perfect demonstrators. In previous datasets, it is often challenging to obtain perfect demonstrators because that requires athletes to present perfect action execution, which is often difficult to achieve. The following will be combined with Fig. 9 The steps at block 856 are further described.

[0108] In some embodiments, process 800 can be performed independently as a separate embodiment without relying on previous steps (e.g., Figure 1 Execution of steps 110-150) of method 100.

[0109] Fig. 9 A flowchart of a process 900 for scoring an action according to some embodiments of the present invention is shown. The process 900 may be a process of combining the above Figure 8 Specific implementations of the steps at block 856 of process 800 are described, but the scope of the present invention is not limited in this regard.

[0110] In some embodiments, each action may be assigned multiple penalty items, wherein each penalty item may indicate a corresponding execution error, and each penalty item may be assigned multiple error severity scores, which will be described in conjunction with the following. Fig.12 For each action, the steps at block 910 and block 930 may be performed.

[0111] Process 900 may include, at block 910 , determining an error severity score for each penalty item based on a comparison of the athlete's movements in the video with the demonstration movements.

[0112] Process 900 may include, at block 930 , accumulating severity scores of each penalty item to determine a penalty score.

[0113] The inventors found through experiments that more fine-grained penalty labels help promote the model to better understand the quality of action when using a dataset to train the model to perform action quality assessment tasks.

[0114] Fig.10 FIG. 1 is a schematic diagram of a labeled hierarchical data set 1000 according to some embodiments of the present invention. The data set 1000 may be obtained by executing the above Figure 8 Method 800, Fig. 9 The data set is obtained by process 900 in the embodiment of the present invention, but the scope of the present invention is not limited to this.

[0115] The data set 1000 may be a hierarchical data set including multiple levels. In some embodiments, the data set 1000 may include a sports event level 1010 , a course level 1020 , and an action level 1030 .

[0116] The sports event hierarchy 1010 can be associated with a sports event. As an example, the sports event hierarchy is shown to include sports events. ABC , sports events DEF , ... As an example, the sports event ABC may be a men's floor exercise event, and the sports event DEF may be a men's rings event.

[0117] Each sporting event may include multiple courses. As an example, sporting event ABC includes courses ABC. 5 , Course ABC 4 , Course ABC 3 , Course ABC 2 As an example, it is also shown that the sports event DEF includes a course DEF 6 、DEF 5 、DEF 4 、DEF 3 Each course may indicate the difficulty level of the movements to be assessed. As an example, each course code may correspond to a corresponding difficulty code of a sport.

[0118] Each course may include multiple actions. As an example, it is shown that course ABC5 includes action ABC5 X1 ,ABC5 X2 ,ABC5 X3 ,ABC5 X4 ,ABC5 X5 As an example, it is shown that course ABC4 includes action ABC4 X1 ,ABC4 X2 ,ABC4 X3 ,ABC4 X4 As an example, it is shown that the course DEF6 includes the action DEF6 X1 、DEF6 X2 、DEF6 X3 、DEF6 X4 As an example, it is shown that the course DEF5 includes the action DEF5 X1 、DEF5 X2 、DEF5 X3 、DEF5 X4 .

[0119] In some embodiments, the data set 1000 may not include the sports event level 1010 , but only include the course level 1020 and the action level 1030 .

[0120] Fig.11 Shown is a layout of an annotation pipeline and system 1100 for annotating a data set according to some embodiments of the present invention. Fig.11 The annotation pipeline and system in can be used to perform the above combined Figure 8 The method described in 800, combined with Fig. 9 The process 900 described above can be combined with Fig.10 Data set 1000 is described, but the scope of the present invention is not limited in this regard.

[0121] The annotation pipeline and system 1100 are mainly performed by annotators 1111. The annotators 1111 may be provided with necessary training so that they fully understand and master the key points of scoring for action quality assessment. For example, the scoring key points may be taught to them by coaches / athletes 1112. For example, the annotators 1111 may learn the scoring standards 1113 for sports events to master the scoring key points.

[0122] As shown at 1120, the coach / athlete 1112 may perform the corresponding actions and collect videos from multiple perspectives to obtain an original video 1130. The original video 1130 may be annotated by annotator 1111 to form a label sequence 1140. At this stage, the label sequence 1140 may include sports events 1141 and courses 1142. This stage may be combined with the above Figure 8 The steps at blocks 810 and 830 described above correspond, but the scope of the present invention is not limited in this regard.

[0123] Subsequently, as shown at 1150, the annotator 1111 may perform video parsing on the original video 1130 to obtain a plurality of video clips, wherein each video clip may correspond to an action. At this stage, the annotator 1111 may annotate each video to form a label sequence 1140, which may include a sports event 1141 and a course 1142. This stage may be combined with the above Figure 8 The steps at block 851 described correspond, but the scope of the present invention is not limited in this regard.

[0124] Subsequently, as shown at 1160, the annotator 1111 may classify each video clip into action categories to assign each video clip a corresponding action category, as shown in different legends in the figure. At this stage, the annotator 1111 may annotate the action for each video clip to form a label sequence 1140, which may include sports events 1141, courses 1142, and actions 1143. Thus, a data set including three levels has been formed. This stage can be combined with the above Figure 8The steps at block 853 described correspond, but the scope of the present invention is not limited in this regard.

[0125] Subsequently, as shown at 1170, the annotator 1111 may perform action quality assessment on each video clip to further annotate the corresponding action with a deduction score and a final score. At this stage, the annotator 1111 may annotate branches for each action to form a label sequence 1140, which may include a sports event 1141, a course 1142, an action 1143, a deduction score 1144, and a score 1145. This stage may be combined with the above Figure 8 The steps at block 855 and block 526 described above correspond, but the scope of the present invention is not limited in this regard.

[0126] Fig.12 The interactive interface layout of a labeling tool evaluation system for scoring actions according to some embodiments of the present invention is shown. Fig.12 The standard tool evaluation system shown in FIG. 1 can be used to perform the above combined Fig. 9 Process 900 is described, but the scope of the present invention is not limited in this regard.

[0127] Fig.12 The interface layout shown in the figure may include a segment id. The interface layout may also include video frames of the target under multiple perspectives and video frames of a perfect demonstrator for action quality assessment. The interface layout may also include an action category and an action description corresponding to the video segment. The interface layout may also include several deduction items and deduction scores for the action category. When annotating, the annotator may compare the video frames of the target and the perfect demonstrator to determine the deduction score for each deduction item, thereby determining the deduction value for the action category by accumulating the deduction scores of each deduction item.

[0128] In order to verify the effect of using the dataset generated according to an embodiment of the present invention to train a model to perform motion quality assessment tasks, the inventors adopted a pair-wise contrastive regression method as the baseline architecture.

[0129] Fig.13 A comparative regression workflow 1300 with an additional score reduction head is shown according to some embodiments of the present invention.

[0130] Workflow 1300 performs regression calculations for target X 1310 and demonstrator Z 1320. Target X 1310 may include video frames 1312 and 3D poses 1314. Video frames 1312 may include multiple video frames of the same scene acquired from multiple vision systems. Demonstrator Z 1320 may include video frames 1322 and 3D poses 1324.

[0131] The backbone network can encode the corresponding input signal into deep network features. Specifically, the backbone video 1332 may encode the video frame 1312 of target X 1310 into deep network features, and perform video regression calculations on these deep network features at 1342. Backbone pose 1334 may encode the 3D pose 1314 of target X 1310 into deep network features, and perform pose regression calculations on these deep network features at 1344. Backbone Video 1336 may encode the video frame 1322 of demonstrator Z 1320 into deep network features, and perform video regression calculations on these deep network features at 1346. Backbone pose 1338 the 3D pose 1324 of demonstrator Z 1320 may be encoded as deep network features, for which pose regression calculations are performed at 1348 .

[0132] Action Decoder 1352 extracts key motion features across the time domain dimension for the result of the video regression calculation at 1342 to perform regression calculation for the target X 1310 at 1362. Motion Decoder 1354 extracts key motion features across the temporal dimension for the result of the video regression calculation at 1346 to perform regression calculation for the demonstrator Z 1320 at 1366

[0133] For encoder 1370 receives the results of the regression calculations at 1362 and 1364 to facilitate interaction between target X 1310 and demonstrator Z 1320 .

[0134] For encoder The output of 1370 is subjected to regression calculation at 1375. Score Regressor 1382 is used to map interaction features into relative scores 1391 .

[0135] Accordingly, given a pair of target X 1310 and demonstrator Z 1320, the contrastive regression problem can be expressed as follows:

[0136]

[0137] Among them, Θ represents the learnable parameter, is the predicted score of target X, y z is the groundtruth score of demonstrator Z.

[0138] The inventors used Spearman's rank correlation (ρ) and relative L2 distance (R-l2). ρ is used to evaluate the rank correlation between the predicted score and the true value score. R-l2 focuses on the numerical score difference between the predicted score and the true value score. The following Table 1 shows the experimental results.

[0139] Table 1 Experimental results of multimodal input

[0140]

[0141] As can be seen from Table 1, the ρ obtained by using posture 2d or posture 3d as input alone is less than the ρ obtained by using RGB (video) alone as input. The reduction of ρ may be due to the abstract nature of the key point data, which leads to the loss of important information (such as muscle deformation) for action evaluation. Compared with using RGB video alone, the dataset formed by combining RGB video with posture 3d according to an embodiment of the present invention increases ρ from 0.701 to 0.746, achieving a 6.4% improvement, and R-l2 is also reduced.

[0142] In order to explore the impact of course information (i.e., difficulty level) on action quality assessment, the inventors also used two training methods for training: 1) hybrid training, in which shuffled data sets are trained for all difficulty levels; 2) curriculum learning, in which the training data is organized in order of difficulty level, gradually introducing more difficult actions and more complex quality concepts. The experimental results show that both the hybrid training and curriculum learning methods including course-level data sets present higher ρ than those for data sets of a single difficulty (i.e., data sets without course levels). The experimental results also show that for the same amount of training data, the course learning method achieves better ρ than hybrid learning at all difficulty levels. This proves that the gradual progression of curriculum learning promotes the development of complex quality concepts on the basis of simpler quality concepts learned earlier.

[0143] In order to further evaluate the effect of introducing penalty score labels in the data set on the training model to perform action evaluation tasks, penalty head 1 1384, ..., penalty head N 1386 are also added to workflow 1300, which are used to determine the predicted penalty scores of corresponding penalty items 1, ..., penalty item N, respectively, so as to determine the penalty scores 1392 of penalty items 1-N.

[0144] Table 2 below shows the effect of the number of penalty items (penalty heads) on training the model to perform action evaluation tasks.

[0145] Table 2 The influence of the number of deduction heads on the training effect

[0146]

[0147] As can be seen from Table 2, the model with the deduction head has achieved significant improvements, with ρ increasing by 5.7% to 0.741 and R-l2 decreasing by 20% to 0.501. This proves that the fine-grained deduction labels enhance the model's understanding of action quality. In addition, the use of deduction-based annotations enables the collection of non-deduction samples for each action category, thus ensuring the availability of perfect demonstrators, which is difficult to obtain in previous datasets.

[0148] According to another aspect of the present invention, a computer device is provided.

[0149] Fig.14 A block diagram of a computer device 1400 is shown according to some embodiments of the present invention.

[0150] Computer device 1400 may include memory 1410 and processor 1420. Memory 1410 may include computer program 1415. Processor 1420 may execute computer program 1415 to implement the steps of any method or process described above.

[0151] According to another aspect of the present invention, a computer-readable storage medium is provided.

[0152] Fig.15 A block diagram of a computer-readable storage medium 1500 is shown according to some embodiments of the invention.

[0153] The computer readable storage medium 1500 may include a computer program 1550. The computer program 1550, when executed by a processor, may implement the steps of any of the methods or processes described above.

[0154] According to another aspect of the present invention, a computer program product is provided.

[0155] Fig.16 A block diagram of a computer program product 1600 is shown according to some embodiments of the invention.

[0156] The computer program product may include a computer program 1650. The computer program 1650, when executed, may implement the steps of any of the methods or processes described above.

[0157] Embodiments of the present invention have been described with reference to the accompanying drawings. The embodiments are illustrative rather than restrictive.

Claims

1. A method for constructing a data set for evaluating the quality of an athlete's movements, comprising: Providing a plurality of videos captured from a plurality of viewing angles for a scene to be subjected to motion quality assessment, wherein the plurality of videos are synchronized and time-aligned; estimating camera parameters of a plurality of cameras used to acquire the plurality of videos; determining a two-dimensional pose of the athlete from a plurality of video frames of the plurality of videos; determining a three-dimensional pose of the athlete based on the two-dimensional pose and the camera parameters; and The plurality of video frames and the three-dimensional poses are combined to form the dataset.

2. The method of claim 1, wherein: Provide multiple videos of the scene to be evaluated for motion quality, captured from multiple viewpoints, including: Collecting a plurality of videos for the scene to be evaluated for motion quality from the plurality of viewing angles; and The multiple videos are aligned in time domain.

3. The method of claim 2, wherein: Aligning the multiple videos in time domain includes: Playing music when the multiple cameras capture the multiple videos; Determine the offsets of multiple audios respectively collected by the multiple cameras; aligning the plurality of audio frequencies in a time domain based on the offset; and The multiple videos are aligned in the time domain by aligning the multiple audios.

4. The method of claim 3, wherein: Determining the offsets of the multiple audios respectively collected by the multiple cameras includes: An offset of a first audio in a second audio of the plurality of audios is determined by maximizing a cross-correlation of normalized Mel-frequency cepstral coefficients of the first audio and the second audio.

5. The method of claim 1, wherein: The camera parameters include camera intrinsic parameters, and Estimating camera parameters of a plurality of cameras used to acquire the plurality of videos includes: The camera intrinsic parameters of the plurality of cameras are calibrated by photographing a moving array-arranged chessboard.

6. The method of claim 1, wherein: The camera parameters include camera extrinsics, and Estimating camera parameters of a plurality of cameras used to acquire the plurality of videos includes: Mark points with known positions and distance relationships in the annotation system; The camera extrinsic parameters of the plurality of cameras are estimated using a PnP algorithm based on the three-dimensional object points and the two-dimensional image points.

7. The method of claim 1, wherein: Determining two-dimensional posture information of the athlete from a plurality of video frames of the plurality of videos includes: Use object detection algorithms to identify all detectable human objects in the video frame; For the detected human targets, a two-dimensional tracking algorithm is used to generate a sequence of bounding boxes of the athletes; and Two-dimensional pose estimation is used on images within bounding boxes within the sequence of bounding boxes to determine a two-dimensional pose of the player.

8. The method of claim 1, wherein: Determining the three-dimensional pose of the athlete based on the two-dimensional pose and the camera parameters includes: Based on the two-dimensional pose of the player and the camera parameters, triangulation is performed to determine a three-dimensional pose of the player.

9. The method of claim 8, wherein: Determining the three-dimensional pose of the athlete based on the two-dimensional pose and the camera parameters further includes: Based on the determined three-dimensional posture of the athlete at the current moment and the camera parameters, estimating the position range of the athlete in the multiple video frames at the next moment by back-projection technology; Skipping target detection within the position range and directly estimating the two-dimensional posture of the player at the next moment; and For the estimated two-dimensional posture of the player at the next moment, the matching task is skipped and triangulation of the player is directly performed.

10. The method of claim 1, further comprising: At least a portion of the data set is labeled to form a hierarchical data set including a plurality of levels.

11. The method of claim 10, wherein: Labeling at least a portion of the data set includes: At least some of the video frames are annotated with lesson information to form a data set including a lesson hierarchy, wherein the lesson information indicates a difficulty level of an action to be evaluated.

12. The method of claim 11, wherein: The dataset is organized in hierarchical order according to difficulty level for training on the performance quality assessment task.

13. The method of claim 11, wherein: The course information is recorded when recording the at least some of the video frames.

14. The method of claim 11, wherein: Labeling at least a portion of the data set further includes: Action information is annotated for at least some video frames to form a dataset including an action hierarchy, wherein each course includes a plurality of actions.

15. The method of claim 14, wherein: Labeling action information for at least some video frames includes: segmenting the at least some video frames into a plurality of video segments, wherein each video segment corresponds to an action; and Each video clip is assigned an action category based on the curriculum information.

16. The method of claim 15, wherein: Labeling action information for at least some video frames also includes: For each action in the video frame, the action is scored by comparing the athlete's action in the video frame to a demonstration action, wherein the demonstration action is based at least in part on a scoring criteria for the sport.

17. The method of claim 16, wherein: Scoring of actions includes: The deduction score is determined based on the quality of execution as reflected by the comparison of the athlete's movements in the video with the demonstration movements.

18. The method of claim 17, wherein: Each action is assigned a plurality of penalty items, wherein each penalty item indicates a corresponding execution error, and each penalty item is assigned a plurality of error severity scores, and Determination of deduction points includes: For each action, Determine the severity of the error for each penalty item based on a comparison of the athlete's movements in the video with the demonstration movements; and The error severity scores of the various deduction items are accumulated to determine the deduction score.

19. The method of claim 11, wherein: Labeling at least a portion of the data set further includes: At least some of the video frames are labeled with sports event information to form a data set including a sports event hierarchy, wherein the sports events are associated with the sports event to which the action to be evaluated belongs, wherein each sports event includes a plurality of lessons.

20. The method of claim 19, wherein: The motion event information is recorded when the at least some video frames are recorded.

21. A computer device comprising a memory, a processor and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1-20.

22. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 20.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 20.

Images