Method for improving markerless motion analysis

Abstract

Systems, methods, and computer-readable storage devices for improving label-free motion analysis are disclosed. One method includes: receiving positional data of joint centers of a moving body captured by at least one camera device; enhancing the positional data of the joint centers of the body with three-dimensional (3D) angular kinematic data using model equations, wherein the enhanced 3D angular kinematic data includes increased measurement accuracy of the positional data of the joint centers of the body; and providing the enhanced 3D angular kinematic data for display to evaluate motion performance.

Description

[0001] Cross-references to related applications

[0002] This application was filed on February 17, 2022 as a PCT international patent application and claims priority to U.S. Provisional Patent Application No. 63 / 150511, filed on February 17, 2021, for the Method for Improving Markerless Motion Analyses, the subject matter of which is incorporated herein by reference. Technical Field

[0003] This invention generally relates to methods, systems, and computer-readable media for providing physical movement training and instruction using markerless motion analysis. More specifically, this invention relates to a computer-implemented system for providing improved markerless motion analysis for exercise training and instruction. Background Technology

[0004] Many different techniques have been developed to teach the correct mechanics of a sport (e.g., swinging a golf club or bat). Currently, coaches (e.g., golf professionals) use imaging and / or video analytics systems to teach how to swing a golf club correctly. Using typical video or imaging analytics systems, the golf swing is captured by imaging devices (e.g., camera devices and / or video recording equipment). The coach plays back the recorded images and / or video information to illustrate the golf swing while providing feedback. This feedback can be comments about swing-related problems, praise for swing improvement, suggestions for correcting the swing, and / or any other verbal instructional comments related to the swing. Visualizing an individual golf swing in this way has been recognized as a valuable tool for identifying problems and correcting them to improve the overall golf swing.

[0005] While imaging and / or video analytics systems are widely used by sports professionals such as professional golfers and baseball players, these systems have particular drawbacks. One such drawback involves the fact that these systems require the identification of human posture and spatial landmarks. For example, professionals must subjectively analyze image and / or video information to identify human posture and spatial landmarks. However, given different camera angles, insufficient camera equipment, loose clothing, etc., typical images and videos may not capture enough information on their own. Therefore, professionals may be forced to guess information about human posture and spatial landmarks. Consequently, the human posture and spatial landmark information identified by professionals alone may be inaccurate because it is difficult to separate the mechanics and measurements of the swing from the image and / or video.

[0006] To overcome the drawbacks associated with typical imaging and / or video analytics systems, motion analysis systems may require users to wear markers and / or sensor elements, which transmit positional data for isolated body parts (e.g., hands, hips, shoulders, and head). These isolated points on the body are measured during the swing against an absolute reference system (e.g., a Cartesian coordinate system) whose center point is a fixed point in the room. By using motion analysis, precise measurements can be provided to more accurately pinpoint problems in the swing.

[0007] The drawback of such tag-based imaging and / or video systems is that they require users to wear tags and may require precise placement of camera devices and / or video equipment. Therefore, the development of tagless motion capture systems / methods has been driven by a wide range of sports and clinical applications.

[0008] However, limitations remain in using markerless motion capture to achieve three-dimensional / three-axis (3D) rotational motion (angular motion) due to the limited set of spatial coordinates provided by standard markerless motion capture methods. This application was made in response to these and other considerations. Summary of the Invention

[0009] According to certain embodiments, systems, methods, and computer-readable media for improving label-free motion analysis are disclosed.

[0010] According to some embodiments, a method for improving a computer-implemented label-free motion analysis is provided. One method includes: receiving position data of joint centers of a moving body captured by at least one camera device; enhancing the position data of the body's joint centers with three-dimensional (3D) angular kinematics data using model equations, wherein the enhanced 3D angular kinematics data includes increased measurement accuracy of the position data of the body's joint centers; and providing the enhanced 3D angular kinematics data for display to evaluate motion performance.

[0011] According to some embodiments, a system for improving label-free motion analysis is disclosed. One system includes: a data storage device storing instructions for improving label-free motion analysis; and a processor configured to execute the instructions to perform a method comprising: receiving position data of joint centers of a moving body captured by at least one camera device; enhancing the position data of the joint centers of the body with three-dimensional (3D) angular kinematic data using model equations, wherein the enhanced 3D angular kinematic data includes increased measurement accuracy of the position data of the joint centers of the body; and providing the enhanced 3D angular kinematic data for display to evaluate motion performance.

[0012] According to some embodiments, a non-transitory computer-readable medium is disclosed that stores instructions, when executed by a computer, to cause the computer to perform a method for improving label-free motion analysis. One method of the computer-readable medium includes: receiving position data of joint centers of a moving body captured by at least one imaging device; enhancing the position data of the joint centers of the body with three-dimensional (3D) angular kinematic data using model equations, wherein the enhanced 3D angular kinematic data includes increased measurement accuracy of the position data of the joint centers of the body; and providing the enhanced 3D angular kinematic data for display to evaluate motion performance.

[0013] Additional objects and advantages of the disclosed embodiments will be set forth in part in the description which follows, and will be apparent from that description or may be learned by practice of the disclosed embodiments. The objects and advantages of the disclosed embodiments will be achieved and attained by means of the elements and combinations particularly pointed out in the appended claims.

[0014] It should be understood that both the foregoing general description and the following detailed description are exemplary and illustrative only, and do not limit the disclosed implementations as claimed. Attached Figure Description

[0015] In the following detailed description, reference will be made to the accompanying drawings. The drawings illustrate different aspects of this disclosure, and where appropriate, reference numerals indicating similar structures, parts, materials, and / or elements in the different drawings are similarly labeled. It should be understood that various combinations of structures, parts, and / or elements are contemplated and are within the scope of this disclosure, in addition to those specifically shown.

[0016] Furthermore, this document describes and illustrates numerous embodiments of the present disclosure. The present disclosure is neither limited to any single aspect or embodiment thereof, nor to any combination and / or arrangement of such aspects and / or embodiments. Moreover, each aspect and / or embodiment of the present disclosure may be used alone or in combination with one or more other aspects and / or embodiments of the present disclosure. For the sake of brevity, certain arrangements and combinations are not discussed and / or described separately herein.

[0017] Figure 1 A golf swing performance according to an embodiment of this disclosure is shown as an example of the embodiment and the optimal implementation mode of the proposed method for performing markerless motion analysis.

[0018] Figure 2 illustrates a method for calculating 3D angular kinematic data by deriving a body segment coordinate system from a finite set of coordinates with the body as a reference, according to an embodiment of the present disclosure.

[0019] Figure 3 The present disclosure illustrates a process for providing accurate 3D angle measurements using markerless motion capture, according to an embodiment of the present disclosure.

[0020] Figure 4 A kinematic enhancement method for improving the accuracy of 3D angle measurements using markerless motion capture, according to an embodiment of the present disclosure, is shown.

[0021] Figure 5 A method for improving label-free motion analysis according to embodiments of the present disclosure is described.

[0022] Figure 6 A high-level description of exemplary computing devices that can be used with systems, methods, and computer-readable media disclosed herein, according to embodiments of the present disclosure, is provided.

[0023] Figure 7 This document provides a high-level description of exemplary computing systems that can be used with systems, methods, and computer-readable media disclosed herein, according to embodiments of the present disclosure.

[0024] Similarly, numerous embodiments are described and illustrated herein. This disclosure is neither limited to any single aspect or embodiment thereof, nor to any combination and / or arrangement of such aspects and / or embodiments. Each aspect and / or embodiment of this disclosure may be used alone or in combination with one or more of other aspects and / or embodiments of this disclosure. For the sake of brevity, many combinations and arrangements of these are not discussed separately herein. Detailed Implementation

[0025] Those skilled in the art will recognize that various implementations and methods of this disclosure can be practiced in accordance with the specification. All such implementations and methods are intended to be included within the scope of this disclosure.

[0026] As used herein, the terms “comprise,” “comprising,” “have,” “having,” “include,” “including,” or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements may include not only those elements but also other elements not expressly listed or inherent to those elements. The term “exemplary” is used in the sense of “example” rather than “ideal.” Additionally, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, the phrase “X employs A or B” is intended to mean any natural inclusion arrangement. For example, any of the following instances satisfy the phrase “X employs A or B”: X employs A; X employs B; or X employs both A and B. Furthermore, the articles “a” and “an” used in this application and the appended claims should generally be interpreted as “one or more” unless otherwise specified or clearly indicated from the context as a singular form.

[0027] For the sake of brevity, this document may not describe in detail the conventional techniques associated with the methods and other functional aspects of systems and servers (and their various operating components). Furthermore, the connecting lines shown in the various figures included herein are intended to illustrate exemplary functional relationships and / or physical connections between elements. It should be noted that many alternative and / or additional functional relationships or physical connections may exist in embodiments of this subject matter.

[0028] Exemplary embodiments of the present disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. Throughout the drawings, the same reference numerals will be used to refer to the same or similar parts whenever possible.

[0029] This disclosure generally relates to methods for estimating the three-dimensional spatial segment orientation of more than one segment when the available spatial reference point data does not adequately define one or more segments.

[0030] Now refer to the attached diagram, Figure 1 An environment 100 for performing enhanced motion analysis according to an embodiment of this disclosure is depicted. For example... Figure 1As shown, environment 100 includes an imaging and / or video analysis system 102 that uses one or more camera devices and / or video recording devices 104 to record information about physical motion processes captured by the camera devices and / or video recording devices 104. The imaging and / or video analysis system 102 can also capture and / or calculate position information. The imaging and / or video analysis system 102 processes the data to generate analytical or instructional information that can be used for golf swing analysis and training.

[0031] Although environment 100 is described below as a system and method for providing golf swing analysis, imaging and / or video analysis system 102 can be used to provide motion analysis in other sports (e.g., baseball, tennis, cricket, polo, or any other sport in which motion is a measure of the elements of motion). Furthermore, the analysis can be similarly used to provide almost any form of physical motion analysis. Additionally, while environment 100 depicts two camera devices 104, a single camera device or a single video recording device can be used.

[0032] In embodiments of this disclosure, measurement accuracy may be important, and accurate motion capture is improved through the development of image processing algorithms and body models that simulate anatomical constraints to enhance the detection, tracking, and spatial transformation of skeletal segments and joint centers in sequential digital images. The development of label-free motion capture systems has broad applications in sports and clinical settings, and employs optical systems to identify human posture and spatial landmarks.

[0033] However, limitations in achieving markless motion capture for 3D / triaxial rotational motion (angular motion) may persist due to the limited set of spatial coordinates provided by markless motion capture methods. Complex multibody motion (as demonstrated by human motion) can be achieved through two or more segments mechanically constrained at joints that allow 3D rotation between adjacent segments. According to Hilbert's fundamental axiom of geometric affinity, three non-collinear points allow for the definition of a plane in 3D space (Euclidean). This condition enables the realization of an analytically defined coordinate system that can create a body-fixed reference frame for segmental orientation measurements. When the spatial coordinates of the three body-fixed non-collinear points for each segment are unavailable, alternative methods can be used to approximate the 3D measurement.

[0034] For example, a partial set of 3D angular kinematic data can be provided during a slow-moving task (e.g., squatting or walking) performed in a specific 2D plane of motion. Despite these limitations in motion capture, large angular deviations can be observed compared to simultaneous measurements determined by a marker-based motion analysis system. As described above, marker-based motion analysis systems directly attach markers to bony prominences with established and validated reliability to minimize displacement of anatomical landmarks during dynamic activity.

[0035] However, identifying marker points (keypoints) in a markerless system may rely on probabilistic, nondeterministic feature detection that can be highly sensitive to environmental conditions (lighting, obstacles / clothing, pose), potentially introducing random positional errors where reported marker coordinates deviate randomly from the true marker location. When attempting to use these points to define a planar (2D) reference frame for kinematic measurements, the variance of random positional errors occurring at each keypoint can introduce compound errors across frames in an image sequence. Furthermore, keypoint positional errors can propagate through the analytically defined reference geometry in each frame, resulting in erroneous biases. In addition to factors directly affecting keypoint positional errors, reference frame errors derived from keypoints can also be influenced by segment geometry and the location of keypoints on segments, potentially affecting actors using the reference frame size of markerless keypoints and potentially compounding / amplifying them. Considering these potential errors, various methods discussed below can be used to mitigate the impact on positional errors.

[0036] 2D angular kinematics data can be measured by referencing the 3D coordinates of three non-adjacent joint centers (i.e., hip, shoulder, elbow, etc.), which produce motion patterns with relatively good consistency between label-free and label-based systems. Similar motion patterns are accompanied by persistent large angular deviations, prompting alternative measurements of rotational motion to use label-free motion capture to create effective performance measurements. Furthermore, combining musculoskeletal body models with applied anatomical constraints improves the accuracy of angular kinematics measurements.

[0037] However, despite these methods, lateral plane rotational motion, in which body segments twist away from the 2D view plane of the camera setup, may rarely be extracted and reported. While larger camera arrays can improve the likelihood of markerless motion capture systems adequately tracking 3D joint center positions during torsional motion tasks, lateral plane angular motion may be difficult to extract accurately and reliably.

[0038] One limitation of markerless motion capture techniques may be the number of body-referenced spatial coordinates available for measuring 3D angular kinematics data. As discussed in more detail below, measurements may require body segments to have three reference points to define a local 3D coordinate system. The 3D spatial orientation of one body segment relative to another is physically quantified as the relative rotational difference between the two 3D coordinate systems. Measuring the 3D spatial orientation of a body segment (rigid body) requires defining a 3D coordinate system relative to the global 3D coordinate system (reference frame) for the rigid body of interest. To define a fixed 3D body reference frame, the spatial coordinates of at least three independent, non-collinear points fixed within the reference frame must be known. This geometric requirement may not be met by markerless motion capture and may therefore prevent analytical methods from quantifying 3D spatial orientation when body segment coordinate data is insufficient or uncertain.

[0039] To improve markerless motion capture technology, embodiments of this disclosure implement a novel analytical method for measuring 3D angular kinematics data based on constrained physical systems. This method could be a useful tool for analyzing torsional motion performance (i.e., a golf swing) and injury risks (i.e., knee injuries). Increasing the number of cameras used for markerless motion capture can improve the accuracy of body-referenced coordinates, but increasing the number of cameras is unlikely to increase the computational requirements for providing effective and reliable 3D angular kinematic measurements. Therefore, methods to improve the practicality of markerless motion capture technology are discussed in detail below.

[0040] This disclosure relates to methods for providing and improving 3D motion analysis using markerless motion capture techniques with at least one camera device (e.g., a single camera device). Embodiments of this disclosure provide methods for estimating the 3D spatial orientation of one or more rigid bodies, where fewer than three body-fixed reference points are known for each segment, and / or where these points are not rigidly fixed to the body due to unexplained / random measurement variance.

[0041] In embodiments of this disclosure, a method is provided that enables the estimation of the 3D spatial orientation of one or more rigid bodies by: (1) using available spatial information, such as, but not limited to, two or more points on at least two directly or indirectly kinematically constrained segments, to establish a related segmental reference frame, which utilizes additional constraints supplemented by details provided by directly observable body fixation points; and (2) applying a probabilistic mapping of the geometric relationship between one or more body fixation points and one or more system kinematic constraints to the 3D body fixation reference frame. The process can utilize direct measurements from the kinematically constrained body fixation coordinate system to achieve an indirect representation of multi-body system kinematics. When the measurements track the related reference frame, the measurements reflect the net effect of the constrained multi-body system kinematics. In other words, it can be viewed as a kinematic constraint centroid used as a third body fixation point shared by two segments. Therefore, a representative sample of kinematically constrained measurements can be paired with corresponding verified reference measurements of any diagnostically meaningful motion to determine a weighted feature mapping associated with the two sets of measurements. Methods may include, but are not limited to, supervised learning, latent variable models, and features derived from constrained kinematics and / or keypoints.

[0042] According to the implementation method, as discussed in more detail below, the calculation method is applied when at least two body fixed points (key points) are known on at least two kinematically constrained rigid bodies (segments) and a global reference frame is defined. Figures 2A to 2D Methods for calculating 3D angular kinematic data by deriving a body segment coordinate system from a finite set of coordinates referenced to the body, according to various aspects of this disclosure, are described. In particular, Figures 2A to 2D A method is described that defines a body segment coordinate system to enable the calculation of three-dimensional (3D) angular kinematic data when at least two fixed points on at least two kinematically constrained rigid bodies are known. Figure 2A As shown, segments 202A and 202B each have at least two body fixation points (key points) 204A and 204B. Each key point 204A and 204B can be based on a reference coordinate system 206 fixed in an inertial frame (global reference frame). Figures 2A to 2D Each of these defines a reference axis. Figure 2B An axis 208 is shown between two body fixation points 204A and 204B defined on each body segment 202A and 202B. Body fixation points 204A and 204B may not include joint constraints shared by the two segments 202A and 202B. A temporary axis 210 may be defined by the midpoint of each of the axes 208 on adjacent segments 202A and 202B. Figure 2C Axis 212 is shown, which can be defined as an axis orthogonal to segment-specific axes 208 and 210. Figure 2D A final axis 214 is shown, which can be defined as an axis orthogonal to axes 208 and 212. This method provides a frame of reference at each segment such that one axis (axis 208) is body-fixed, and the two axes capture orientation-driven interactions representing the system's constrained joint movements between the two segments.

[0043] Measurements captured from a body-axis-fixed reference frame provide a systematic representation that includes the (constrained) relative spatial orientations between segments and each segment relative to the global reference frame 206. As discussed in more detail below, 3D angular kinematic data can then be directly computed based on conventions that can be applied according to the assumption of an independent, body-fixed coordinate system.

[0044] For example, the x-axis and z-axis can be defined as parallel to the floor / ground and perpendicular to each other, and the y-axis can be defined as perpendicular to the x-axis and z-axis and orthogonal to the floor / ground. Alternatively, the reference to the coordinate origin can be defined as unique to the system user. Using an axis system, measurements can be determined for angular rotation values. For example, Φ s It can represent the rotation angle of the shoulder about the x-axis, Θ s ζ can represent the rotation angle of the shoulder about the y-axis. s It can represent the rotation angle of the shoulder about the z-axis, Φ h It can represent the rotation angle of the hip about the x-axis, Θ h It can represent the rotation angle of the hip about the y-axis, and ζ h This can represent the rotation angle of the hip about the z-axis. Therefore, Φ relates to shoulder and hip flexion, Θ relates to shoulder and hip rotation, and ζ relates to shoulder and hip tilt. Positional elements related to the flexion, rotation, and tilt of both the shoulder and hip can be determined by measuring with reference to the coordinate system.

[0045] As described above, embodiments of this disclosure provide supervised learning methods and / or machine learning methods that can be used to augment kinematic data. The methods of embodiments of this disclosure can determine model equations for augmenting kinematic data based on training one or more machine learning methods. While machine learning has been discussed more generally, an example of machine learning can include neural networks, including but not limited to convolutional neural networks, deep neural networks, recurrent neural networks, etc.

[0046] As discussed in more detail below, for a specific motion task, the kinematic enhancement process can approximate the true, analytically defined, independent body-fixed reference frame orientation relative to a global reference frame. This process can use a probabilistic mapping determined using representative example measurements from kinematically constrained validation examples of independent segmental orientations, correlated with measurements calculated using an alternative body-axis-fixed interactive reference frame during representative samples of the relative orientations. The model equations from the mapping process can then be used to measure 3D angular kinematic data for new motion performance. The results can be displayed on dashboards, such as computer screens, smart devices, etc.

[0047] Figure 3 A method 300 for providing accurate 3D angle measurements using markerless motion capture, according to an embodiment of this disclosure, is shown. Figure 3 As shown, the athlete / user / performer can perform action 302, which includes body movement of the athlete / user / performer. At 304, at least one camera device, such as a single camera device, can capture at least two images of the moving body. At 306, image processing can be performed to output an image with a two-dimensional pixel array. Then, at 308, camera device calibration can be performed to generate two-dimensional coordinates of the moving body. Next, as at 310, a direct linear transformation can be performed on the image to generate three-dimensional coordinates of key points of the moving body. For example, the three-dimensional coordinates of the key points of the moving body could be a three-dimensional joint center position 312.

[0048] At position 314, the three-dimensional joint center position of the moving body can be received. For example... Figure 3 As shown at 316, the 3D angular kinematics data can then be directly calculated based on the assumed independent, body-fixed coordinate system. At 318, an approximate kinematic augmentation process can be performed for a specific motion task captured by at least one camera device, using an orientation of a real, analytically defined, independent body-fixed reference frame relative to a global reference frame. As discussed in more detail below, this process can use a probabilistic mapping determined using representative example measurements from kinematically constrained validation examples of independent segmental orientations, which are correlated with measurements calculated using an alternative body-axis-fixed interactive reference frame during representative samples of relative orientations. Then, at 320, the model equations from the mapping process can be used to measure the new motion performance's 3D angular kinematics data, and at 322, the model equations can be displayed on a computer screen or a smart device's dashboard.

[0049] Figure 4A kinematic enhancement method for improving the accuracy of 3D angle measurements using markerless motion capture, according to an embodiment of this disclosure, is illustrated. The process can begin at 402, where reference measurements can be received / acquired via data processing 403. A probability mapping can be used, and this probability mapping is determined using representative example measurements from kinematically constrained validation examples of independent segmental orientations, said representative example measurements being correlated with measurements calculated using an alternative body axis fixed interactive reference frame during representative samples of relative orientations.

[0050] For example, data processing 403 can process label-free motion capture data 404 used in the training of one or more machine learning methods. In a representative example measurement of label-free motion capture 404, an athlete / user / performer can perform actions including the athlete / user / performer's body movements. At 404A, at least one camera device (e.g., a single camera device) can capture at least two images of the moving body. At 404B, image processing can be performed to output an image with a two-dimensional pixel array. Then, at 404C, camera device calibration can be performed to generate two-dimensional coordinates of the moving body. Next, as at 404D, a direct linear transformation can be performed on the images to generate three-dimensional coordinates of key points of the moving body. Then, as shown at 404E, 3D angular kinematics data can be directly computed according to the conventions applied, assuming an independent, body-fixed coordinate system. The 3D angular kinematics data is applied to generate rotational data about the three-dimensional coordinates.

[0051] Furthermore, data processing 403 can process tag-based motion capture data 406 used in the training of one or more machine learning methods. For example, in tag-based motion capture, one or more infrared cameras 406A can capture tags placed on the user. By using camera calibration 406B, x-axis and y-axis coordinates can be extracted from the captured tags. Based on the extracted coordinates, a direct linear transformation 406C can be used to generate three-dimensional coordinates on the x, y, and z axes. Finally, three-dimensional angular kinematic data 406D can be applied to generate rotational data about the three-dimensional coordinates.

[0052] In embodiments of this disclosure, data processing 403 may employ a motion analysis method that extracts and transforms a set of body-referenced landmarks (keypoints) from continuous video images of motion performance. The set of keypoints may be finite and may provide some of the joint center locations of the body (e.g., midpoint locations between keypoint pairs). Keypoints may be received and / or input to generate three-dimensional angular kinematic data. A probabilistic mapping 408 may then be used. During the probabilistic mapping, representative example measurements from kinematically constrained validation examples of independent segmental orientations may be used, these representative example measurements being correlated with measurements calculated using an alternative body axis fixed interactive reference frame during representative samples of relative orientations. The probabilistic mapping transformation 408 may provide model equations 410 to enhance the computational accuracy of the 3D angular kinematic data.

[0053] The enhanced kinematic data can then be transferred to a dashboard where relevant task-specific metrics are extracted and displayed digitally and / or graphically on a computer screen or smart device. (See also...) Figure 3 (Refs. 320 and 322). Performance metrics can be used to evaluate motion performance and provide actionable insights. Probabilistic mapping can establish a relationship between coupled, constrained reference frames and analytically defined equivalent reference frames. As discussed above, this is achieved using paired, validated examples of independent segmental orientations, correlated with measurements calculated using an alternative body axis fixed interactive reference frame during representative samples of relative orientations. Such mapping provides improved quantitative accuracy for measuring 3D angular kinematic data based on a set of constrained / ambiguous body-fixed reference positions and advances label-free motion techniques by providing a motion analysis approach not limited by small sets of body reference markers to generate accurate 3D angular kinematic data.

[0054] Probability mappings can be trained on multiple example datasets to account for variability between the measured subject and / or different measurement settings. Furthermore, several repositories of example datasets for various motion and functional tasks can exist. These repositories may be previously generated and freely available. By using these multiple example datasets, enhanced 3D angular kinematics data can be generated using a finite set of keypoints captured by a single camera device during label-free motion capture.

[0055] By training on multiple example datasets, probabilistic mappings can learn common features among these datasets. When analytical solutions are unavailable, probabilistic mappings can parameterize relationships between related phenomena. Probabilistic mappings utilize pairwise data relating to the inputs and outputs of a mechanism of interest. Methods applying probabilistic mappings can include numerical approximations or other functional approximations that apply error measures based on probability-based constraints on the results.

[0056] Supervised learning is an application of probabilistic mapping. For example, supervised learning can use analytical solutions to determine location data based on multiple factors / data points from an example dataset. Using examples of paired input-output data representing parameters and locations, a function can be approximated that maps the location data to 3D angular kinematics data by minimizing the prediction error on the example data. As described above, embodiments of this disclosure can be used to generate model equations using supervised learning, machine learning, neural networks, etc.

[0057] More generally, this disclosure can be used to improve various aspects of label-free motion capture by using, for example, supervised learning or machine learning (e.g., neural networks). In an exemplary embodiment of this disclosure, reference measurements used by a trained neural network can generate model equations. Therefore, values ​​can be fed into the neural network. The neural network can then be trained to directly output model equations. To train the neural network, it can receive label-free motion capture data 404 and labeled motion capture data 406 as input data.

[0058] Figure 5 A method 500 for improving label-free motion analysis according to embodiments of this disclosure is described. Method 500 may begin at step 502, in which a neural network model may be constructed, a neural network may be received, and / or model equations may be received directly. The neural network model may include multiple neurons. The neural network model may be configured to output model equations. The multiple neurons may be arranged in multiple layers including at least one hidden layer and may be connected by connections. Each connection includes weights. The neural network model may include a convolutional neural network model, a deep neural network, or a recurrent neural network.

[0059] If a neural network is received / built or supervised learning is used to generate model equations, a training example dataset may be received at step 504. The training example dataset may include positional data of joint centers of a moving body. By training on the example dataset, the probability map can learn common features among the example datasets. When an analytical solution is unavailable, the probability map can parameterize the relationships between related phenomena. The probability map utilizes pairwise data associated with the inputs and outputs of a mechanism of interest. Furthermore, the received training dataset may include data previously captured by a label-free motion capture system and / or a label-based motion capture system.

[0060] At step 506, a training example dataset can be used to train a neural network model or generate model equations. Then, at step 508, the trained neural network model / model equations can be output. At step 510, a test dataset can be received. Alternatively and / or additionally, a test dataset can be created. Then, at step 512, the test dataset can be used to test the trained neural network or the output model equations for evaluation. Furthermore, once the evaluation passes a predetermined threshold, the trained neural network or the output model equations can be utilized. Additionally, in some embodiments of this disclosure, the steps of method 500 can be repeated to generate multiple model equations. The multiple model equations can then be compared with each other. Alternatively, steps 510 and 512 can be omitted.

[0061] At step 514, an output trained neural network model and / or model equation configured to output a model equation may be received. Then, at step 516, position data of the joint centers of a moving body captured by at least one camera device may be received. For example, the at least one camera device may be a single camera device that captures images using markerless motion capture. Alternatively, before receiving the position data, the single camera device may be used to capture a first image and a second image of the moving body at a first time and a second time different from the first time. The position data of the joint centers of the moving body may then be generated based on the images. For example, receiving the position data of the joint centers of the body may include receiving at least two keypoints of a first segment of the moving body and at least two keypoints of a second segment of the moving body for at least two separate time points. The keypoints correspond to the positions of parts of the moving body captured by at least one camera device. Based on the received position data, a first axis can be defined between at least two keypoints of each segment, a temporary axis midpoint can be defined for each of the first axes, a second axis orthogonal to the temporary axis and the corresponding first axis of each segment can be defined, and a third axis orthogonal to the first and second axes of each segment can be defined. Then, 3D angular kinematic data for the first and second segments can be generated based on at least two keypoints of the first segment from at least two individual time points, at least two keypoints of the second segment, and the first, second, and third axes based on the defined keypoints.

[0062] Then, at step 518, the 3D angular kinematics data of the body's joint center position data can be enhanced. This enhancement can be achieved using model equations. As described above, model equations can be generated based on probability mappings to enhance the 3D angular kinematics data. The enhanced 3D angular kinematics data includes increased measurement accuracy of the body's joint center position data. Finally, at 520, the enhanced 3D angular kinematics data can be provided for display to evaluate motor performance.

[0063] Figure 6High-level illustrations depict an exemplary computing device 600 that can be used with systems, methods, and computer-readable media disclosed herein, according to embodiments of the present disclosure. For example, the computing device 600 can be used as a system to perform methods according to embodiments of the present disclosure. The computing device 600 may include at least one processor 602 that executes instructions stored in memory 604. Instructions may be, for example, instructions for implementing functions described as being performed by one or more components discussed above, or instructions for implementing one or more methods described above. The processor 602 can access memory 604 via a system bus 606. In addition to storing executable instructions, memory 604 may also store data, images, information, event logs, etc.

[0064] The computing device 600 may also include a data storage device 608 accessible by the processor 602 via the system bus 606. The data storage device 608 may include executable instructions, data, images, information, event logs, etc. The computing device 600 may also include an input interface 610 that allows external devices to communicate with it. For example, the input interface 610 may be used to receive instructions from external computer devices, users, etc. The computing device 600 may also include an output interface 612 that interfaces the computing device 600 with one or more external devices. For example, the computing device 600 may display text, images, etc., through the output interface 612.

[0065] It is conceivable that external devices communicating with computing device 600 via input interface 610 and output interface 612 can be included in an environment that provides a user interface of virtually any type to which the user can interact. Examples of user interface types include graphical user interfaces, natural user interfaces, etc. For example, a graphical user interface can accept input from a user using input devices such as a keyboard, mouse, remote control, etc., and can provide output on an output device such as a display. Furthermore, a natural user interface allows the user to interact with computing device 600 in a manner unconstrained by input devices such as keyboards, mice, remote controls, etc. Instead, a natural user interface can rely on speech recognition, touch and stylus recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, etc.

[0066] Furthermore, although shown as a single system, it should be understood that computing device 600 can be a distributed system. Therefore, for example, several devices can communicate via a network connection and collaboratively perform tasks described as being performed by computing device 600.

[0067] Turning Figure 7 , Figure 7High-level illustrations depict an exemplary computing system 700 that can be used with the systems, methods, and computer-readable media disclosed herein, according to embodiments of the present disclosure. For example, computing system 700 may be or may include imaging and / or video analysis system 102. Additionally and / or alternatively, imaging and / or video analysis system 102 may be or may include computing system 700.

[0068] The computing system 700 may include multiple server computing devices, such as server computing device 702 and server computing device 704 (collectively referred to as server computing devices 702-704). Server computing device 702 may include at least one processor and memory; the at least one processor executes instructions stored in the memory. Instructions may be, for example, instructions for implementing functions described as being performed by one or more components discussed above, or instructions for implementing one or more methods described above. Similar to server computing device 702, at least one subset of server computing devices 702-704, excluding server computing device 702, may each each include at least one processor and memory. Furthermore, at least one subset of server computing devices 702-704 may include corresponding data storage devices.

[0069] One or more processors in server computing devices 702-704 may be, or may include, the processor of imaging and / or video analysis system 102. Furthermore, one or more memories (or multiple memories) in server computing devices 702-704 may be, or may include, the memory of imaging and / or video analysis system 702. Additionally, one or more data memories (or multiple data memories) in server computing devices 702-704 may be, or may include, the data storage device of imaging and / or video analysis system 102.

[0070] The computing system 700 may also include various network nodes 706 that transmit data between server computing devices 702-704. Furthermore, network nodes 706 can transmit data from server computing devices 702-704 to external nodes (e.g., outside the computing system 700) via network 708. Network nodes 702 can also transmit data from external nodes to server computing devices 702-704 via network 708. Network 708 may be, for example, the Internet, a cellular network, etc. Network nodes 706 may include switches, routers, load balancers, etc.

[0071] The architecture controller 710 of the computing system 700 can manage the hardware resources of the server computing devices 702-704 (e.g., the processors, memory, data storage devices, etc. of the server computing devices 702-704). The architecture controller 710 can also manage the network node 706. Furthermore, the architecture controller 710 can manage the creation, provisioning, deprovisioning, and monitoring of the managed runtime environment instantiated on the server computing devices 702-704.

[0072] As used herein, the terms "component" and "system" are intended to cover a computer-readable data storage device configured with computer-executable instructions that, when executed by a processor, cause certain functions to be performed. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be located on a single device or distributed across several devices.

[0073] The various functions described herein can be implemented in hardware, software, or any combination thereof. When implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and / or transferred as one or more instructions or code through a computer-readable medium. Computer-readable media may include computer-readable storage media. Computer-readable storage media can be any available storage medium accessible by a computer. By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. As used herein, disks and optical discs may include compact discs (“CDs”), laser discs, optical discs, digital versatile optical discs (“DVDs”), floppy disks, and Blu-ray discs (“BDs”), wherein disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Furthermore, transmitted signals are not included within the scope of computer-readable storage media. Computer-readable media may also include communication media, which include any medium that facilitates the transfer of computer programs from one place to another. A connection may be, for example, a communication medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of communication media. Combinations of the above may also be included within the scope of computer-readable media.

[0074] Alternatively and / or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field-programmable gate arrays (“FPGAs”), application-specific integrated circuits (“ASICs”), application-specific standard products (“ASSPs”), systems-on-chips (“SOCs”), complex programmable logic devices (“CPLDs”), and the like.

[0075] The above description includes examples of one or more embodiments. Of course, for the purposes of describing the foregoing aspects, it is impossible to describe every possible modification and alteration of the described apparatus or method; however, those skilled in the art will recognize that many further modifications and arrangements of the various aspects are possible. Therefore, the described aspects are intended to cover all such changes, modifications, and variations falling within the scope of the appended claims.

Claims

1. A computer-implemented method for improved markerless motion analysis, the method comprising: receiving three-dimensional position data of a reference marker affixed to a rigid body in a connected segment rigid body system captured by at least one camera device, wherein the three-dimensional position data is insufficient for constructing three-dimensional orientation information of at least one segment of the connected segment rigid body system, and wherein the connected segment rigid body system comprises two or more points on at least two segments that are directly or indirectly kinematically constrained; generating a model equation for the three-dimensional position data using probabilistic mapping; deriving three-dimensional kinematic data of one or more segments of the connected segment rigid body system using the model equation, wherein the deriving comprises analyzing the two or more points on the at least two segments that are directly or indirectly kinematically constrained to construct a relevant segment frame of reference that utilizes additional constraints that supplement details provided by directly observable body-fixed points, and wherein the three-dimensional kinematic data comprises three-dimensional orientation information of at least one segment of the connected segment rigid body system; and providing the three-dimensional kinematic data for display to assess motion performance.

2. The method of claim 1, wherein the at least one camera device is a single camera device, and wherein the method further comprises: capturing, using the single camera device, a first image and a second image of the connected segment rigid body system at a first time and a second time different from the first time.

3. The method of claim 2, wherein, the first image and the second image are captured using markerless motion capture.

4. The method of claim 1, wherein, receiving the three-dimensional position data of the reference marker comprises: receiving, for at least two separate points in time, at least two key points of a first segment of the connected segment rigid body system and at least two key points of a second segment of the connected segment rigid body system, the key points corresponding to locations of parts of the connected segment rigid body system captured by the at least one camera device.

5. The method of claim 4, further comprising: defining a first axis between the at least two key points of each segment; defining a temporary axis midpoint of each of the first axes, defining a second axis of each segment that is orthogonal to the temporary axis and the respective first axis of each segment; and defining a third axis of each segment that is orthogonal to the first axis and the second axis.

6. The method of claim 5, further comprising: generating three-dimensional kinematic data of the first segment and the second segment based on the at least two key points of the first segment, the at least two key points of the second segment from the at least two separate points in time, and the defined first, second, and third axes based on the key points.

7. The method of claim 1, wherein, generating the model equation comprises using a neural network model, and wherein the method further comprises: receiving a plurality of example data sets comprising a plurality of position data of joint centers of the connected segment rigid body system; and training the neural network model using the plurality of example data sets, the neural network model configured to output the model equation.

8. The method of claim 7, further comprising: constructing the neural network model comprising a plurality of neurons arranged in a plurality of layers comprising at least one hidden layer and connected by a plurality of connections, the neural network model configured to output the model equation.

9. The method of claim 1, wherein, deriving the three-dimensional kinematic data using the model equation comprises using a probabilistic mapping to augment the three-dimensional kinematic data.

10. A system for improved markerless motion analysis, the system comprising: a data storage device storing instructions for improved markerless motion analysis; and a processor configured to execute the instructions to perform a method, the method comprising: receiving three-dimensional position data of a reference marker affixed to a rigid body in a connected segment rigid body system captured by at least one camera device, wherein the three-dimensional position data is insufficient for constructing three-dimensional orientation information of at least one segment of the connected segment rigid body system, and wherein the connected segment rigid body system comprises two or more points on at least two segments that are directly or indirectly kinematically constrained; generating a model equation for the three-dimensional position data using a probabilistic mapping; deriving three-dimensional kinematic data of one or more segments of the connected segment rigid body system using the model equation, wherein the deriving comprises analyzing the two or more points on the at least two segments that are directly or indirectly kinematically constrained to construct a relevant segment frame of reference that utilizes additional constraints that supplement details provided by directly observable body-fixed points, and wherein the three-dimensional kinematic data comprises three-dimensional orientation information of at least one segment of the connected segment rigid body system; and providing the three-dimensional kinematic data for display to assess motion performance.

11. The system of claim 10, wherein the at least one camera device is a single camera device, and wherein the method further comprises: capturing, using the single camera device, a first image and a second image of the connected segment rigid body system at a first time and a second time different from the first time.

12. The system of claim 11, wherein, the first image and the second image are captured using markerless motion capture.

13. The system of claim 10, wherein, receiving the three-dimensional position data of the reference marker comprises: receiving, for at least two separate points in time, at least two key points of a first segment of the connected segment rigid body system and at least two key points of a second segment of the connected segment rigid body system, a key point corresponding to a location of a part of the connected segment rigid body system captured by the at least one camera device.

14. The system of claim 13, further comprising: defining a first axis between at least two key points of each segment; defining a provisional axis midpoint of each of the first axes, defining a second axis of each segment that is orthogonal to the provisional axis and the respective first axis of each segment; and defining a third axis of each segment that is orthogonal to the first axis and the second axis.

15. The system of claim 14, further comprising: generate three-dimensional kinematic data for the first segment and the second segment based on the at least two key points of the first segment, the at least two key points of the second segment, and the defined first axis, second axis, and third axis based on the key points.

16. The system of claim 10, wherein, generating the model equation includes using a neural network model, and wherein the method further comprises: receiving a plurality of example data sets comprising a plurality of position data of centers of joints of the connected segment rigid body system; and training a neural network model using the plurality of example data sets, the neural network model configured to output the model equation.

17. The system of claim 16, further comprising: constructing the neural network model comprising a plurality of neurons arranged in a plurality of layers comprising at least one hidden layer and connected by a plurality of connections, the neural network model configured to output the model equation.

18. The system of claim 10, wherein, deriving the three-dimensional kinematic data using the model equation includes using a probabilistic mapping to augment the three-dimensional kinematic data.

19. A non-transitory computer-readable storage device storing instructions that, when executed by a computer, cause the computer to perform a method for improving markerless motion analysis, the method comprising: receiving three-dimensional position data of a reference marker affixed to a rigid body in a connected segment rigid body system captured by at least one camera, wherein the three-dimensional position data is insufficient to construct three-dimensional orientation information for at least one segment of the connected segment rigid body system, and wherein the connected segment rigid body system comprises two or more points on at least two segments that are directly or indirectly kinematically constrained; generating a model equation for the three-dimensional position data using a probabilistic mapping; deriving three-dimensional kinematic data for one or more segments of the connected segment rigid body system using the model equation, wherein the deriving includes analyzing the two or more points on the at least two segments that are directly or indirectly kinematically constrained to construct a relevant segment frame of reference that utilizes additional constraints that supplement details provided by directly observable body-fixed points, and wherein the three-dimensional kinematic data comprises three-dimensional orientation information for at least one segment of the connected segment rigid body system; and providing the three-dimensional kinematic data for display to evaluate motion performance.

20. The computer-readable storage device of claim 19, wherein, the at least one camera is a single camera, wherein the method further comprises: capturing, using the single camera, a first image and a second image of the connected segment rigid body system at a first time and a second time different from the first time, and wherein the first image and the second image are captured using markerless motion capture.

Images