System and method for matching a test frame sequence to a reference frame sequence

Through the computer system, the 2D skeleton projection and neural network on the virtual sphere can match the user's exercise performance and reference frame sequence in real time, solving the problem that users find it difficult to correctly match the test frame sequence in the home environment, and achieving correct training in unsupervised training Guidance and health protection.

CN115035542BActive Publication Date: 2024-11-05LINDERA GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210192321.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-03-03
Filing Date
2022-03-01
Publication Date
2024-11-05
Estimated Expiration
2042-03-01

AI Technical Summary

Technical Problem

In a home environment, it is difficult for users to correctly match the test frame sequence and the reference frame sequence when performing physical exercise, especially when using a simple smartphone or tablet to capture the test frame sequence, and the existing hardware device is complex and cannot be flexible. position.

Method used

Through the computer system, the user's exercise performance is matched with the coach performance in the reference frame sequence using the 3D joint coordinate list in the posture data file. The 2D skeleton projection on the virtual sphere is used, independent of the camera's field angle and distance. Neural networks perform real-time matching and feedback.

Benefits of technology

It realizes real-time matching of user exercise performance and reference frame sequences without complex hardware devices, providing correctness feedback, ensuring that users can exercise correctly during unsupervised training and prevent health damage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure GDA0004975992360000101
    Figure GDA0004975992360000101
  • Figure GDA0004975992360000111
    Figure GDA0004975992360000111
  • Figure GDA0004975992360000112
    Figure GDA0004975992360000112
Patent Text Reader

Abstract

The present invention discloses a computer-implemented method, a computer program product, and a computer system for matching a test frame sequence with a reference frame sequence. For each frame of the reference frame sequence, a posture data file represents multiple two-dimensional skeleton projections on a virtual sphere. For a specific frame, each two-dimensional skeleton projection on the sphere corresponds to a two-dimensional reference posture image of a three-dimensional skeleton of a first person from a different field of view. The test frame sequence represents the movement of a second person while performing physical exercise. A real-time two-dimensional skeleton detector module detects the two-dimensional skeleton of the second person in each received test frame of the test frame sequence. A posture matching module selects a specific two-dimensional skeleton projection of the first person with the minimum mathematical distance to the two-dimensional skeleton of the second person in the current test frame to match the current posture of the second person in the current test frame with the corresponding reference posture image of the posture data file.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates generally to electronic data processing and, more particularly, to a computer-implemented method, computer program product, and system for matching a test frame sequence showing a user performing a physical exercise with a corresponding reference frame sequence. Background Art

[0002] In general, performing physical exercise promotes the health of the human musculoskeletal system and generally improves the health of an individual. Physical therapy measures that help patients regain their full mobility after accidents, surgeries, herniated discs, etc. often include specialized physical exercises that need to be performed correctly by the patient. If not performed correctly, such exercises may even harm the patient's health.

[0003] While during a medical session, an experienced physical therapist or trainer can observe the patient during the exercise and provide instructions on how to properly perform the exercise, this is not possible when the patient performs the exercise at home without the supervision of a physical therapist or trainer. It is often highly advantageous to repeat the exercise regularly between medical sessions to achieve sustainable improvement in the patient's medical status.

[0004] There are commercial applications that provide instruction videos to users, which show the correct exercise performance of the instructor. However, either the user does not receive feedback (such as in a medical treatment) if the exercise is performed correctly or not performed correctly, in which case the user's health may even deteriorate when exercising in an incorrect manner, or complex hardware devices are required, such as the VAHA mirror from etone Motion Analysis GmbH, 10587 Berlin, Germany, or the PIXFORMANCE mirror provided by Pixformance Sports GmbH, 14624 Dargo-Deberitz, Germany, which also includes a depth sensor. Such hardware mirror solutions include large devices that require complex setup procedures. After setting, they need to operate in a fixed position. Summary of the invention

[0005] Therefore, there is a need for an application that is able to provide instructions to a user in such a way that damage to the user's health is prevented by ensuring that physical exercises are performed correctly during unsupervised training without the need for specific complex hardware devices. This is achieved by providing an application that allows, according to the embodiments described in the independent claims, to identify deviations in the exercise performance of said user captured by a test frame sequence of a video from a reference frame sequence showing an instructor performing the correct sequence.

[0006] Therefore, the embodiment solves further technical problems. When trying to match a test frame sequence with a reference frame sequence, it is usually necessary to position the camera used to capture the test frame sequence in the same way as the camera used to capture the test frame sequence (i.e., under the same field of view). In a home environment, this may be challenging, especially when using a simple smartphone or tablet with an integrated camera to capture the test frame sequence. Typically, such devices cannot be easily installed in the corresponding position. Sometimes even the position where the user exercises does not allow such positioning. Therefore, it is necessary to provide an application that can match the test frame sequence with the reference frame sequence in such a way that the field of view of the camera toward the user is unknown when capturing the user's exercise performance.

[0007] The technical problem is solved by embodiments - computer-implemented methods, computer program products and computer systems according to the independent claims - to match a test frame sequence showing the user (while performing the physical exercise) with a corresponding reference frame sequence (demonstrated by a coach / instructor the correct performance of the physical exercise).

[0008] A computer system, which has a corresponding computer program product loaded into its memory and processes the loaded computer program using one or more processors, is configured to perform the computer-implemented method as described below.

[0009] The computer system has an interface suitable for receiving a posture data file. For example, the posture data file can be loaded from a data carrier such as a DVD or USB device, or can be retrieved from a remote storage device (for example, a server accessible via the Internet). The posture data file is based on a list of three-dimensional joint coordinates (one list for each feature frame of a reference video), and the three-dimensional joint coordinates represent the positions of multiple joints of the three-dimensional skeleton of the first person (coach / instructor) in the corresponding frame. Such 3D joint coordinates can be obtained from a reference video, for example, by using the technology described in the publication "VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera" (ACM Transactions on Graphics, Vol. 36, No. 4, July 2017, Article 44) by DUSHYANT MEHTA et al. This article describes in detail an implementation example of a CNN, which is based on the ResNet50 network architecture in "Deep Residual Learning for Image Recognition" by He et al. (IEEE Conference on Computer Vision and Pattern Recognition, 2016). European patent EP3656302 describes in detail how to extract three-dimensional gait information from an acquired video stream by using the above method or other methods similar to the SMPLify method (see "Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image" by Federica Bogo et al., available at http: / / files.is.tue.mpg.de / black / papers / BogoECCV2016.pdf).

[0010] For at least one frame subset of the reference frame sequence, a feature frame subset, based on the corresponding list of 3D joint coordinates, multiple two-dimensional skeleton projections on the virtual sphere are stored in the posture data file. The frame subset (i.e., the feature frame subset) contains one or more frames with characteristic postures of physical exercise (i.e., clearly allowing the posture of physical exercise to be characterized, which makes it possible to distinguish physical exercise from other exercises based on such postures). For example, a subset of characteristic postures of squatting may include a first feature frame and a second feature frame, wherein the first feature frame shows the posture of the first person in the squat, and the second feature frame shows the posture of the first person in the upright position. Therefore, a specific joint is placed at the geometric center of each sphere. Usually, the pelvis of the first person is used as the specific center joint. In other words, each two-dimensional skeleton projection of a specific frame on a specific virtual sphere corresponds to a two-dimensional reference posture image of the three-dimensional skeleton of the first person at a corresponding time point from different field of view angles. In a first embodiment, different concentric virtual spheres can be used, wherein different radii reflect the different (physical) distances of the 2D skeleton projection from the 3D center joint. In a second embodiment, a single virtual sphere with a standardized radius is used. In this embodiment, the (physical) distance between the camera and the user becomes irrelevant. Details about these two embodiments are given in the specific embodiments. For example, a 2D projection on a specific sphere (where the pelvis is on the equator of the sphere) corresponds to a camera field of view, where the camera will operate at a height above the ground corresponding to the height of the pelvis. The projection on the equator reflects any field of view of 0 to 360 degrees around the first person. For example, the 2D projection can be calculated with a resolution of one projection per 1 degree. Of course, other resolutions can be selected.

[0011] The 2D projection of the pelvis in the upper half of the sphere corresponds to a situation where the camera is located at a position higher than the pelvis of the first person. The 2D projection of the pelvis in the lower half of the sphere corresponds to a situation where the camera is located at a position lower than the pelvis of the first person. Therefore, the 2D projection corresponding to the camera position below the ground is of course not of practical relevance, because in real world scenarios, the camera cannot be positioned below the ground in the user's exercise environment.

[0012] In a first embodiment, for a given projection angle (i.e., field of view), projections on various concentric spheres with different radii reflect different distances between the first person and the camera. The projection on one of the outer spheres produces a smaller skeleton image than the projection on the inner sphere of the plurality of concentric spheres. The larger projection image corresponds to a smaller distance between the imaginary camera and the first person than the smaller projection image. Therefore, it is irrelevant which real-world distance between the user and the camera is associated with the corresponding projection image size. It is only relevant that the sum of all projection sizes in different spheres covers a certain distance range, which includes the distance between the camera and the user when the camera is used to capture the user's video while exercising. In other words, the actual distance range covered by the 2D projection is between 2 and 10 meters. The specific implementation describes in detail how to derive a 2D projection from a list of 3D joint coordinates to obtain a posture data file, which is then used as an input to a computer system.

[0013] In a second embodiment, only a single sphere with a standardized radius is used. That is, for each 2D skeleton projection, the joint coordinates are standardized so that the corresponding x, y coordinates are between 0 and 1. This embodiment takes advantage of the fact that when a particular pose is viewed from a certain angle, the proportions between the various body parts remain unchanged regardless of the distance from which they are viewed. That is, the standardized 2D projection on a single virtual sphere at a specific angle is constant, regardless of the actual distance between the camera and the user.

[0014] In addition, the computer system has an interface that is configured to receive a test frame sequence representing the motion of a second person (user) when imitating physical exercise. Imitating physical exercise means that the second person attempts to perform physical exercise in real time in the same manner as the first person demonstrates physical exercise in a reference frame sequence. Interface technology for receiving video streams in real time from a camera device is well known in the art. Therefore, the user has placed the camera at any field of view angle to capture the test frame sequence when the user is exercising. Usually, the user watches the reference video while exercising. However, this is not necessary. For example, the user may have memorized how to perform physical exercise. In this case, the user may not need to still watch the reference video while exercising. However, the following steps of the method implemented by the computer can still be performed.

[0015] The system also has a real-time 2D skeleton detector, which is configured to detect the two-dimensional skeleton of the second person in each received test frame of the test frame sequence. As used herein, real-time describes a near real-time system response, wherein the calculation in response to the received input only delays the time delay introduced between the occurrence of an event and the use of processed data (such as for display or feedback and control purposes) by automatic data processing or network transmission. For example, a near real-time display will deduct the event or situation existing at the processing time at the current time as the time close to the live event. Therefore, the 2D skeleton of the second person is a two-dimensional representation of the posture of the second person in the corresponding test frame. A real-time 2D skeleton detector that can calculate the 2D skeleton of the user in real time from the user's video is known in the art. For example, an embodiment is disclosed in European Patent EP3656302. Other embodiments are commercially available, such as the 2D tracker of Kaia Health Software Co., Ltd. (Kaia Health Software GmbH) from Munich, Germany 80803. As used herein, near real-time system response means that the calculation of the 2D skeleton only delays the time delay introduced (by automatic data processing or network transmission) between the occurrence of the event (e.g., the reception of the video frame) and the use of the processed data (e.g., the 2D data calculated by further applications or users). For example, a near real-time display depicts an event or situation existing at the current time minus the processing time as a time close to the live event. For example, a current test frame including a representation of the corresponding two-dimensional skeleton of the second person can be visualized to the second person in near real time. In the case where the camera used to capture the second person is a front camera of a smart phone or a tablet computer, the visualization of the detected 2D skeleton can be presented on the display of a smart phone or a desktop computer as an overlay to the corresponding test frame. In this way, only a single device is needed to capture the test frame and display any result information to the second user. In an alternative setting, the display device can be a separate device. For example, a computer or television screen can be used for display, and a webcam installed on the top of the screen can be used to capture the person exercising.

[0016] In the case of using the first embodiment with concentric spheres, the detected 2D skeleton is directly provided as input to the posture matching module. In the case of using the second embodiment with standardized spheres, the detected 2D skeleton is standardized before being provided as input to the posture matching module. That is, the detected or standardized 2D skeleton is then provided as input to the posture matching module of the computer system, which is configured to select the specific two-dimensional skeleton projection of the first person with the minimum mathematical distance to the two-dimensional skeleton of the second person in the current test frame to match the current posture of the second person in the current test frame with the corresponding reference posture image of the posture data file. Therefore, the specific two-dimensional skeleton projection represents the corresponding reference posture image at the field of view angle corresponding to the specific angle of the camera device. In other words, the posture matching module compares the identified 2D skeleton of the current test frame (associated with the second person) with the 2D skeleton projection of the reference video (associated with the first person). Therefore, the posture matching module estimates the mathematical distance between the identified 2D skeleton and the 2D skeleton projection on the concentric sphere, or in the case of the second embodiment, estimates the mathematical distance between the standardized sphere (e.g., by calculating the Euclidean distance between the corresponding joints or by using the corresponding trained neural network). It should be noted that such distance calculations can also be performed in near real time. Therefore, compared with the first embodiment, the second embodiment requires a reduction in the computation time for distance calculations, because by excluding the distance dimension (i.e., by using standardized joint coordinates reflecting the ratio between body parts rather than using the absolute distance measure between corresponding joints), the search space for appropriate 2D projections is reduced from a group of 2D projections on multiple concentric spheres to a group of 2D projections on a single standardized sphere. The posture matching module finally selects the 2D projection with the minimum distance from the identified 2D skeleton. As a result, the selected 2D skeleton projection represents the posture image of the reference video, which is closest to the posture performed by the second person in the current frame, for which the skeleton detector has identified the corresponding 2D skeleton. This means that the field of view of the camera used to capture the test frame sequence approximately corresponds to the field of view associated with the imaginary camera position, which is associated with the selected 2D skeleton projection. The real distance between the camera and the user no longer works. It should be noted that in embodiments where the posture data file includes 2D projections of all frames of the reference frame sequence, each test frame of the test frame sequence can produce a positive match result with the corresponding 2D projection on the virtual sphere, whereas in embodiments where the posture data file only includes 2D projections of a subset of feature frames, positive match results can only be provided for such (current) test frames, i.e., those in which the user assumes a posture corresponding to the characteristic posture of the exercise in one of the feature reference frames.

[0017] In one embodiment, the selection of the 2D projection with the minimum mathematical distance to the identified 2D skeleton can be performed by a corresponding trained neural network. For example, a standard multilayer perceptron can be trained in the following manner so that it can perform such selection in near real time. A set of videos showing users in different postures while exercising is used as a basic data set. Such videos can be obtained, for example, from the so-called CMU panoramic data set. Then, the previously described 2D projection algorithm is applied to such videos to generate a corresponding 2D skeleton projection for each frame of a subset of frames marked as a feature of the exercise. For example, the corresponding tags of the feature frames can be set accordingly by the user annotating the reference frame sequence. Therefore, for each posture projection, a vector with corresponding polar coordinates (r, Φ, θ) is known and can now be used as a label for the corresponding 2D projection. Then the 2D skeleton projection is used as input to train the neural network to predict (r, Φ, θ) for any 2D posture, where the corresponding label provides ground truth for training. Once the neural network is fully trained, it can predict the (r, Φ, θ) vector corresponding to any 2D skeleton identified by a real-time 2D skeleton detector in near real time. The predicted (r, Φ, θ) vector points directly to the 2D skeleton projection of the reference video stream, which is estimated (by the neural network) to have the minimum mathematical distance to the pose in the current test frame. In other words, the neural network selects in real time the 2D projection at the field of view angle that is closest to the field of view angle between the user and the camera that acquires the test frame sequence. An advantage of this embodiment is also that it can automatically adapt to any position changes of the user immediately while the exercise is being performed. The neural network immediately selects the 2D projection of the reference video stream that provides the best match to the current pose at the changed position or perspective (if the user only slightly changes the viewing direction). The perspective describes the polar coordinates of the camera relative to the user captured by the camera. That is, the perspective is described by the distance r between the user and the camera and the field of view angle (as defined by the azimuth angle Φ and the polar angle θ).

[0018] Therefore, the 2D skeleton projection sequence associated with the hypothetical camera position represents a reference video stream with frame images that can be directly compared with the frame images of the test frame sequence. That is to say, the selected reference video stream provides an ideal basis for matching the posture performed by the second person when performing physical exercise with the posture of the first person in the reference video. Such posture matching provides a basis for more detailed analysis to identify the posture deviation of the corresponding posture of the second person and the coach / instructor, which may damage the health of the second person. Advantageously, posture matching can be performed by the claimed embodiment based on the video stream of the standard RGB camera device of the test frame sequence of the second person exercising. There is no need for complex video equipment with a camera with a depth sensor, etc. In addition, it is advantageous that the camera for capturing the second person exercising can be positioned at any field of view angle in the second person's exercise environment at a flexible distance. For the first embodiment using concentric virtual spheres, the distance can be flexible within a predetermined distance range. For the second embodiment using a standardized sphere, the distance does not work, and the camera can be placed at any distance in the user's exercise space. That is, the proposed pose matching is view angle agnostic and distance agnostic, because the pose matching component always finds the appropriate 2D skeleton projection with the minimum mathematical distance to the identified 2D skeleton of the second person in the test frame captured by the corresponding camera at an arbitrary view angle. In other words, the desired pose matching effect is achieved independently of the actual camera settings with respect to the view angle and the physical distance between the camera and the user. The selected 2D projection frame sequence can then be used as a reference pose image for pose analysis, as described below.

[0019] A computer system configured as described above is technically capable of performing a near real-time analysis of a sequence of test frames to assess whether the second person is performing the exercise correctly or if there is a risk of negatively affecting the second person's health due to incorrect exercise performance. In the latter case, the system may even provide instructions to the second user on how to improve the exercise to prevent potential health damage.

[0020] To achieve this effect, in one embodiment, the computer system also includes a posture checking module, which is configured to determine the distance between the second person's current posture and the corresponding reference posture in the selected 2D skeleton projection frame sequence. Therefore, this distance is a measure indicating whether the second person is performing physical exercise correctly. In other words, if the distance between the joints in the identified 2D skeleton and the corresponding joints in the selected 2D skeleton projection exceeds one or more corresponding predetermined thresholds, the second person is not exercising correctly. The one or more predetermined thresholds correspond to the tolerance for evaluating correctness. Such tolerance values ​​can be customized by the second person. For example, older users can use a larger tolerance than younger users.

[0021] In the case where the user does not perform physical exercise correctly, the posture check module can provide posture correction feedback instructions on how to correct the current posture. The feedback is then communicated to the user. Different modalities can be used to output the correction feedback to the user. For example, a text message with correction feedback can be merged into the visualization of the current frame represented by the 2D skeleton. Alternatively, the text message can be output to the user via a voice interface as a voice command. Alternatively, a graphic correction instruction can be merged into the current frame represented by the 2D skeleton at a position where the distance exceeds the threshold. For example, at the corresponding joint position, an arrow can indicate the direction in which the current posture needs to be modified for correct exercise.

[0022] In one embodiment, the posture data file can also include annotations for each frame that group joint subsets into corresponding body parts. That is, all joints belonging to the right arm of the first person can be annotated with the group indicator "right arm". This can be done for each body part related to the correct performance of the exercise. Based on such annotations, the posture check module can indicate the body parts that exceed the predetermined critical distance for the current test frame. The feedback provided to the user (the second person) conveys to the user how to change the current posture about the indicated body part until the current distance of the body part drops below the critical distance. In particular, when the modality provided for feedback is used as a text message, the user can quickly (in real time) understand how to change the current posture by following the instructions about the body part rather than by viewing the feedback involving a specific single joint. Alternatively, the corresponding body part can be highlighted in the visualization of the current frame represented by the 2D skeleton. For example, the body part with incorrect posture can be highlighted in red, and the graphic indicator (such as an arrow) associated with the highlighted body part can indicate the correction direction.

[0023] Further aspects of the invention will be realized and attained by the elements and combinations particularly described in the appended claims.It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention described. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A depicts a block diagram of a computer system for matching a test frame sequence capturing a user's physical exercise performance with a corresponding reference frame sequence according to an embodiment;

[0025] Figure 1B depicts concentric spheres for 2D skeleton projection according to an embodiment;

[0026] Figure 1C depicts a single normalized sphere for 2D skeleton projection according to an embodiment;

[0027] Figure 2 is a simplified flow chart of a computer-implemented method for matching a test frame sequence capturing a user's physical exercise performance with a corresponding reference frame sequence according to an embodiment;

[0028] Figure 3 The human shape is depicted using a two-dimensional overlay of a schematic 2D skeleton of a person;

[0029] Figure 4A , 4B Two postures of a physical exercise performed by a person are depicted with a two-dimensional overlay of a schematic 2D skeleton of the person;

[0030] Figure 5A , 6A Two frames of a reference frame sequence are each depicted, in which the coach performs a physical exercise;

[0031] Figure 5B , 5C A user interface is shown having a visualization of a user performing a physical exercise, including feedback to the user regarding correctness of the exercise being performed;

[0032] Figure 5D An example of a priority annotation is shown;

[0033] Figure 6B shows a user interface with a visualization of a user performing a physical exercise, including feedback based on priority annotations according to an embodiment;

[0034] Figure 7 An example embodiment of pose matching using a trained neural network is depicted;

[0035] Figure 8 A data augmentation method for training a neural network is depicted; and

[0036] Fig. 9 is a diagram illustrating examples of general purpose computer devices and general purpose mobile computer devices that may be used with the techniques described herein. DETAILED DESCRIPTION

[0037] Figure 1A A block diagram of a computer system 100 for matching a test frame sequence 201 capturing a performance of a physical exercise t1 - t2 - t3 by a user 2 with a corresponding reference frame sequence 203 is included according to an embodiment. Figure 2 is a simplified flowchart of a computer-implemented method 1000. The method 1000 may be performed by the system 100. To this end, refer to Figure 2 describe Figure 1A ,and Figure 1A The following description also refers to Figure 2 It should be noted that Figure 2The order of the method steps in is exemplary only. At least some of the steps may be performed in a different order. For example, the order of receiving steps 1100 and 1200 may be switched.

[0038] The system 100 is communicatively coupled to a camera 200, which is positioned at an appropriate angle to capture the user 2 while performing the exercise t1-t2-t3. The camera is a standard RGB camera device that provides a video stream including a test frame sequence 201 as an input to the system 100. For example, the camera can be an integrated camera in a smartphone, a notebook or tablet PC, an external webcam, etc. The reference numerals t1-t2-t3 of the exercise indicate the user performing the movement over time (from left to right as time passes). At time t1, the user 2 assumes a first posture with bent knees, from which the user moves continuously through an intermediate posture at t2 toward an upright position at t3.

[0039] The test frame sequence 201 is received 1200 by the real-time two-dimensional skeleton detector module 110 of the system 100. The real-time 2D skeleton detector detects 1300 the two-dimensional skeleton 20 of the user 2 in a specific received test frame of the test frame sequence 201. Typically, such a 2D skeleton detector can detect the 2D skeleton in each frame. However, it is also sufficient to detect the 2D skeleton in every n-th frame (n=2, 3, 4, etc.). As already described in the Summary of the Invention section, such a real-time 2D skeleton detector is known in the art and is commercially available. The two-dimensional skeleton 20 of the user 2 is a two-dimensional representation of the posture of the second person in the corresponding test frame. In this example, the camera is positioned in front of the user 2, slightly moved to the left from the perspective of the user 2. Therefore, the detected 2D skeleton shows the current posture of the user in the current frame Fc from the perspective of the camera 200.

[0040] Figure 3 Such a dual 2D skeleton identified by the skeleton detector 110 is schematically depicted within the outline of the person 300. In principle, the skeleton detector 110 detects the user's joints in the test frame (represented by circles) and connects such joints by thick connecting lines representing the user's skeleton. Figure 3 In the example of , the skeleton detector is further trained to recognize the user's eyes j2, j3 and ears j4, j5. Regarding the 2D skeleton, the following joints can be used: nose j1, neck j8, shoulders j6, j7, elbows j9, j10, wrists j11, j12, hips j13, j14, j15 (the middle hip j13 is also called the pelvis), knees j16, j17, ankles j18, j19 and toes j20, j21. The thick lines between the various joints are simplified representations of the skeleton of the person 300. For example, the human spine, although composed of multiple dorsal vertebrae, is simply represented by connecting lines j1-j8 and j8-j13.

[0041] Figure 4A and 4B Two test frames 410o, 420o are shown in which a user performs two postures as part of a physical exercise, with the corresponding identified 2D skeletons as overlays. Although there is no bone between the user's shoulders and ears, the overlays in the test frames also include ear-shoulder connection lines. These lines provide valuable information about the distance between the ears and shoulders, which is a good indicator of the user's head posture in the corresponding test frames. Therefore, in the schematic diagrams 410, 420 of the postures, the ear-shoulder connection lines are drawn by dashed lines to distinguish them from the simplified skeleton representation.

[0042] The computer system also has an interface to receive 1100 a gesture data file 203 associated with a reference frame sequence 202. The reference frame sequence 202 includes frames Fr1 to Frn showing a first person (coach / instructor) demonstrating the correct performance of a physical exercise to be performed by a second person 2. Of course, the reference frame sequence 202 has been previously recorded (that is, it was not performed by the first person in parallel with the second person). Figure 5A An example of two reference frames 510ru, 510rd is depicted. In frame 510ru, an initial posture of a physical exercise is shown, wherein the first person (coach) is raising the right arm. During the exercise, the coach moves the right elbow to touch the right knee by also raising the right knee. The posture of the exercise is depicted in reference frame 510rd.

[0043] The pose data file of the entire reference frame sequence is generated based on a list of three-dimensional joint coordinates, which represent the positions of multiple joints of the three-dimensional skeleton 10 of the first person in at least one (characterizing) frame subset Fr1 to FRn of the reference frame sequence 202, and for each frame of the subset Fr1 to Frn of the reference frame sequence 202, multiple two-dimensional skeleton projections 10p1, 10p2 on the virtual spherical surface 203s are represented, wherein a specific joint 10cj of the first person (usually the pelvis) is located at the geometric center of the spherical surface 203s. Therefore, for a specific reference frame, each two-dimensional skeleton projection 10p1, 10p2 on the virtual spherical surface corresponds to a two-dimensional reference pose image of the three-dimensional skeleton 10 of the first person from a different field of view angle. The dotted arrows between the three-dimensional skeleton 10 and the 2D skeleton projections 10p1, 10p2 represent two different field of view angles on the 3D skeleton 10. For a simple exercise where the user must repeatedly move from a starting pose to a target pose and back to the starting pose, the feature frame subset may include only two feature frames—a first feature frame having an image of the starting pose and a second feature frame having an image of the target pose. In this example, frames representing the motion between the two poses may not necessarily be reflected in the pose data file (i.e., such frames between the 2D skeleton projections may not necessarily be available in the reference frame sequence 202).

[0044] Briefly go to Figure 1B , depicts an embodiment using multiple concentric virtual spheres. Two such multiple concentric spheres 203s, 203s' are shown. Sphere 203s has a larger radius r than sphere 203s'. Δr determines the distance resolution of the system. Concentric spheres of higher density result in higher resolution about the potential distance between the user and the camera. The 2D projection 10p1' on sphere 203' and the 2D projection 10p1 on sphere 203 reflect the same field of view, but the distance between the camera and the user is different. Therefore, compared with the smaller projection 10p1, the larger projection 10p1' is associated with a shorter distance. The same applies to projections 10p2 and 10p2'. That is, assuming that the virtual camera will be installed at the center of a specific 2D projection, the size of the 2D projection is calculated so that it corresponds to the image of the coach to be captured by the virtual camera. In other words, the distance between the virtual camera and the coach corresponds to the radius of the corresponding sphere, and the correct field of view is defined by the two angular coordinates of the virtual camera position on the sphere.

[0045] exist Figure 1A In the example of , 2D skeletal projection 10p1 reflects the field of view angle associated with an imaginary camera position at a distance from the person corresponding to the camera position of camera 200 relative to user 2 in the user's real-world situation. 2D skeletal projection 10p2 reflects the field of view angle associated with an imaginary camera position that reflects the elevated position of the imaginary camera behind the coach (the pelvis of projection 10p2 is in the upper half of sphere 203s), slightly shifted to the right from the coach's perspective.

[0046] How to derive such a 2D skeleton projection of the pose data file 203 from the original 3D skeleton 10 is described below.

[0047] The spherical coordinate system {r, Φ, θ} is defined by

[0048] r: radial distance

[0049] Φ: azimuth

[0050] θ: polar angle

[0051] Assume that the middle hip joint (pelvis) is located at the center (0, 0, 0) of the coordinate system (e.g., the geometric center of the sphere 203s). It is also assumed that a list of 3D joint coordinates has been generated for at least the feature frames of the reference frame sequence of the corresponding identified joints of the coach's 3D skeleton using any of the previously cited methods. The 3D skeleton of each frame J is then projected to a point on the virtual sphere 203s. Since usually only the upper half of the sphere is relevant, the sphere is also referred to as the wall of the virtual dome in the following. The projection function is:

[0052]

[0053] Next, define intervals with corresponding interval values ​​for r, Φ, and θ. Example definitions with appropriate interval values ​​are given below:

[0054]

[0055]

[0056]

[0057] If appropriate, a person skilled in the art may select other increments between the r-, Φ-, θ-interval values. The increments of the corresponding coordinate values ​​determine the resolution provided by the virtual dome for pose matching under arbitrary field of view angles and variable distances between the user and the camera. It should be noted that the r-coordinate corresponds to the distance between the user and the camera. In the above example, the r-coordinate ranges from 2m to 10m, with a resolution (increment) of 0.1m. The projection on the virtual dome with r=10m is smaller than the projection on the virtual dome with r=2m. In order for the pose matching to work properly, the actual distance is not actually important, because it is sufficient to find a 2D skeleton projection on any one of the virtual domes (i.e., concentric spheres) that matches the size of the 2D skeleton identified in the test frame sequence.

[0058] The polar angle θ specifies the angle between the north pole of the sphere (θ=0) (corresponding to the camera view from the top) and the actual position of a specific projection on the corresponding sphere. That is, θ=180° corresponds to the south pole and reflects the camera view from below the user. θ=90° corresponds to a camera height located at the same height as the user's pelvis. In real camera setup scenarios, the actual range of θ-interval values ​​is usually well covered by the interval θ∈[45°, 46°, ..., 90, ..., 135°].

[0059] The following pseudocode illustrates an example algorithm for projection:

[0060]

[0061] The pose data file (pose file) may be saved in a suitable hierarchical format (e.g., json, xml, ...) and contains the projected 2D skeletons (reference pose images) of all frames of the reference frame sequence, and all projections according to predetermined interval values ​​(i.e., projections of all projection points on the concentric spheres 203s defined by the corresponding interval values ​​of the coordinates r, Φ, θ), as follows:

[0062]

[0063] In other words, the received pose data file 203 includes the 2D skeleton projections of each point on the surface of all concentric spheres according to the resolution defined by the r-, Φ- and θ-interval values.

[0064] Now go to Figure 1C , depicts an alternative embodiment of the virtual dome. This alternative embodiment uses only a single sphere 204s with standardized 2D skeleton projections 10p3, 10p4. In this embodiment, a single sphere 204s is used to generate the standardized 2D skeleton projections 10p3, 10p4. In this embodiment, the r-dimension is fixed (r=1). Only the Φ- and θ-angles determine the position of the corresponding 2D skeleton projection. This is achieved by normalizing the corresponding projected 2D skeleton. Figure 1C On the right side of , the user's original height h0 is shown. This corresponds to the zero distance d0 between the user and the camera. When generating projections for a series of distances, the individual bone lengths between the corresponding joints of the user (for example, the bone length BL0 between the user's left hand and left elbow) are scaled by the same factor as the user's total height in a particular frame. That is, for the user's 2D projection at distance d1 (where the user's projected height corresponds to 2 / 3h0), the bone length BL1 also corresponds to 2 / 3h0. Similarly, at distance d2 (where the user's projected height corresponds to 1 / 2h0), the bone length BL2 corresponds to 1 / 2h0. That is, when each bone length in the projection is normalized by the corresponding projected user height, the same normalized 2D skeleton projection is obtained for any distance. The normalized projection is then projected onto a sphere 204s with r=1.

[0065] Then, the posture matching module 120 of the system 100 selects 1400 a specific two-dimensional skeleton projection 10p1 of the first person having the minimum mathematical distance to the two-dimensional skeleton 20 of the second person in the current test frame Fc. In other words, the posture matching module compares the 2D skeleton of the current test frame Fc with the 2D projections on the concentric spheres by calculating the corresponding mathematical distances between the 2D skeleton and the available 2D skeleton projections. Then, the posture matching module selects the 2D skeleton projection with the minimum distance to the 2D skeleton of the current test frame. The selected 2D projection corresponds to the reference posture image of the posture data file that best matches the current posture of the second person 2 in the current test frame Fc. That is, the selected 2D projection represents the reference posture image at the field of view angle corresponding to the specific field of view angle of the camera device 200 relative to the user 2 and at the distance corresponding to the specific distance between the camera device 200 and the user 2. In the case of using a multi-sphere implementation scheme, the posture matching module can directly use the output of the skeleton detector 110 (the identified 2D skeleton) to compare with the available 2D skeleton projections. In the case of a normalization sphere implementation, the pose matching module 120 has a normalization function that normalizes the identified 2D skeleton received from the skeleton detector before comparing with the available 2D skeleton projections. That is, the length of each bone of the 2D skeleton 20 is divided by the height of the user in the corresponding frame Fc.

[0066] Once the correct 2D projection is selected, the frame sequence of the selected 2D projection 10p1 can be used as a reference frame sequence, for comparison of the upcoming exercise actually performed by the user 2 with the exercise demonstrated by the coach in the reference video. As long as the user does not significantly change its position relative to the camera 200, the selected reference frame sequence remains valid. However, even if the user 2 changes position, for example, by moving close to the screen showing the coaching video, the system 100 can automatically adapt to the new position of the user by simply selecting another reference frame sequence with a 2D projection that matches the new position of the user 2. It should be noted that the posture matching module 120 can select the matching reference posture image in near real time by applying an appropriate algorithm to calculate the mathematical distance between the 2D projection and the current 2D user skeleton. Examples of such algorithms are known in the art and are described in detail in, for example, the following publications. In Ada Wai-chee Fu et al., “Dynamic vp-tree indexing for n-nearest neighbor search given pairwise distances” (VLDB Journal (2000) 9:154–173), a distance-based indexing method is proposed that can be applied to calculate the distance between the corresponding joints in the 2D skeleton of the current frame and the corresponding 2D skeleton projection. Kumar N., Zhang L., Nayar S. (2008) describe alternative methods for mathematical distance calculation in "What Is a Good Nearest Neighbors Algorithm for Finding Similar Patches in Images?" (In: Forsyth D., Torr P., Zisserman A. (eds.) Computer Vision - ECCV 2008. ECCV2008, Lecture Notes in Computer Science, Vol. 5303, Springer, Berlin, Heidelberg)

[0067] Figure 7An embodiment of a pose matching module 120 is depicted, wherein an appropriate 2D projection 10p1 having a minimum mathematical distance to the identified 2D skeleton 20 is selected by a corresponding trained neural network (NN) 122. NN 122 receives as input the 2D skeleton 20 (or a standardized skeleton) of the current frame Fc as a representation of the user's current pose. NN 122 has been trained to predict the (r, Φ, θ) vector associated with the current pose. In other words, NN 122 predicts the coordinates of the center of the 2D skeleton projection, which reflects the field of view angle at which the test frame sequence is obtained. As previously described, depending on the embodiment of the pose data file (multiple concentric spheres or a single standardized sphere), the pose matcher 120 may also include a normalization function 121 to calculate a standardized identified 2D skeleton when a single standardized sphere is used for the 2D skeleton projection.

[0068] For example, NN 122 can be implemented by a standard multilayer perceptron with a relatively small topology, which includes 3 to 6 layers, each layer having 50 to 100 neurons. The input layer IL is configured to receive a detected 2D skeleton (or a standardized 2D skeleton) as a test input. The output layer is configured to provide a prediction of the viewing angle (r, Φ, θ) associated with the test input. Other suitable NN topologies can also be used by those skilled in the art. NN 122 can be trained as previously described so that it can be selected 1400 in near real time. The basic training data set includes a set of videos showing different postures of users while exercising. Then, a 2D projection algorithm is applied to such videos to generate corresponding 2D skeleton projections for each feature frame. Therefore, for each posture projection, a vector with corresponding polar coordinates (r, Φ, θ) is known and can now be used as a label for the corresponding 2D projection. Then the 2D skeleton projection is used as input to train the neural network to predict the (r, Φ, θ) vector of any 2D posture, where the corresponding label provides ground truth for training.

[0069] Once the neural network is fully trained, it can predict in near real time the (r, Φ, θ) vector corresponding to any 2D skeleton identified by the real-time 2D skeleton detector. The predicted (r, Φ, θ) vector points directly to the 2D skeleton projection of the reference video stream that is estimated (by NN 122) to have the minimum mathematical distance to the pose in the current test frame Fc. In other words, the neural network selects in real time the 2D projection at the field of view angle that is closest to the field of view angle between the user and the camera that acquired the test frame sequence. This embodiment is also advantageous in that it can automatically adapt to any position changes of the user immediately while being exercised. The neural network immediately selects the 2D projection of the reference video stream that provides the best match to the current pose at a changed position or perspective (for example, when the user only slightly changes the orientation toward the camera).

[0070] Figure 8 A method for optional data augmentation of a training data set is depicted, which results in improved accuracy when the view angle ((r, Φ, θ) vector) is predicted by the NN 122. For the poses in the basic training data set, the data augmentation includes generating additional pose images by manipulating the body skeleton proportions. In this example, it is assumed that the original 3D skeleton 8 of the person in the video is part of the basic training data set that reflects the body skeleton proportions SP1 of the person. Now, by scaling up and down specific body parts, the data augmentation function can create additional 3D skeletons from the original 3D skeleton 8 for each frame (in which the person is shown). In this example, the generated artificial 3D human skeleton 8' is shown to have a skeleton proportion SP2, which has shorter arms, longer legs and a shorter middle body compared to the original SP1 of a real person. A further generated artificial 3D human skeleton 8" is shown to have skeleton proportions SP3, which has shorter arms, shorter legs and a longer mid-body compared to the original SP1 of the real person. When such modifications of body proportions are applied to all frames of a video of a particular person, additional training video data is generated using artificial workers. For each of these artificial workers, the 2D skeleton projections on the virtual dome are calculated again. By using the additionally generated 2D skeleton projections as training data for training of NN 122, the accuracy of view prediction achieved by NN 120 is improved.

[0071] Returning to FIG. 1 , in one embodiment, the system 100 supports a display function. The visualizer function can be used to present a frame Fc' including the current test frame Fc and a representation of the corresponding two-dimensional skeleton 20 of the second person in near real time, and visualize 1450 the presentation result Fc' to the user 2 via an appropriate display. In a typical setting, the coaching video stream from which the reference frame sequence is generated will be presented to the user, so that the user 2 can watch his / her own exercise at the same time as the presented result, thereby being able to compare his / her own movement with the movement of the coach. However, it is difficult for users without medical knowledge to assess whether the exercise is performed correctly.

[0072] To support the user in this regard, in one embodiment, the system 100 may have a posture checking module 130. The posture checking module 130 determines 1500 the (mathematical) distance of the current posture from the corresponding reference posture, wherein the distance is a measure indicating whether the second person is performing the physical exercise correctly. The distance between the 2D skeleton of the current test frame and the corresponding selected 2D skeleton projection of the reference frame sequence may be calculated using the same algorithm as used to calculate the above minimum distance to select a matching reference posture image. In case the physical exercise is not performed correctly (e.g. the distance between the current posture and the reference posture exceeds a predetermined threshold), the system indicates posture correction feedback 10pf to the user 2 on how to correct the current posture. In the Figure 1AIn FIG. 1 , the posture correction feedback is represented by a small arrow as a graphical element incorporated into the presented output Fc′. However, other modalities may also be used for posture correction feedback, as described below.

[0073] In one embodiment, the pose data file includes annotations for each frame that group joint subsets into corresponding body parts. Figure 3 For example, the joint group j7, j10 and j12 can be annotated as "right arm". Joint 10 can be additionally annotated as "right elbow", and joint j12 is annotated as "right hand". Joints j15, j16, j18 and j20 can be grouped as "right leg". Joints j18, j20 can be grouped as "right foot" as a subgroup of "right leg". Joint j16 can be additionally annotated as "right knee", etc. That is, in the case where body parts have sub-parts (subgroups), there can be a hierarchical relationship between groups of body parts, where such subgroups may include only a single joint.

[0074] Using such annotations, the posture checking module 130 can indicate body parts that exceed a predetermined critical distance for the current test frame. For example, if a knee or elbow of a particular posture shows a significant deviation from the corresponding reference posture, corresponding feedback on how to change the current posture is provided with respect to the indicated body part until the current distance of the body part drops below the critical distance.

[0075] Figure 5B Two screenshots 520ou, 520od of two rendered output frames are shown, wherein the respective current frames show the user 3 performing a Figure 5A 510rd. In the first output frame 520ou, the user raises the right arm according to the corresponding posture "raise". In addition, the 2D skeleton of user 3 is visualized in the output frame 520ou via white black dots and white connecting lines, the white black dots represent the user joints, and the white connecting lines are used as a simplified representation of the user skeleton as an overlay. The output frame can provide further information to the user, such as the posture name 520pn and feedback 520fb, to indicate whether the exercise is performed correctly. It should be noted that in the output frames 520ou, 520od, user 3 seems to be exercising by raising the left knee and lowering the right elbow. However, the impression is generated because the frame sequence showing user 3 is actually captured by the selfie camera of the smartphone, resulting in a mirror effect in the video stream. In fact, user 3 performed the right knee / elbow exercise according to the reference frame sequence. Those skilled in the art can reconcile the view by adjusting the mirror effect accordingly.

[0076] The feedback 520fb provided for both gestures in this example is "Well done!" In these examples, the user has performed the exercise within a predetermined tolerance range according to the demonstrated exercise (i.e., the calculated distance from the reference gesture does not exceed a predetermined critical distance for any of the user's 3 body parts). The critical distance can be customized by the user to take into account specific user characteristics, such as the user's age, degree of mobility limitation (e.g., due to some previous surgery with other limitations), etc. In this example, the modality of the feedback output is a written text message on the display, which assumes that the display is large enough to make the feedback message 520fb readable by the user 3. For example, the output 520ou can be projected to a television screen, etc.

[0077] Figure 5C A screenshot of an output frame 520oc with correction feedback 520pf1 "Improve right knee" is depicted. In this example, the user's right knee is positioned too low, so that the distance between the position of the user's 2D skeleton and the corresponding annotated joint of the corresponding 2D skeleton projection exceeds a predetermined critical distance. Based on the sign of the distance, the posture check module can derive whether the deviation corresponds to a position that is too low or too high, and generate corresponding correction feedback.

[0078] exist Figure 5C In the example of , the right foot of user 3 is certainly also too low compared to the corresponding reference posture. In addition, the user's right elbow is still too high. Therefore, the system can also generate corrective feedback, such as "raise right foot" or "lower right elbow". However, for example, "raise right foot" may mislead the user to extend the right leg instead of raising the knee. Therefore, the posture data file can include further annotations for each reference frame, with different priorities for various body parts in the exercise environment. Figure 5D An example of such a priority annotation is given in . For example, for the reference posture of frame 510rd, the highest priority P10 may be given to the right knee, the second highest priority P9 to the right elbow and a lower priority P8 to the right hand and right foot. The corresponding body parts on the left side of the coach are not considered relevant for the performance of the exercise. Therefore, priority P0 (i.e., no priority) is used as the priority annotation for these body parts of the coach. In Figure 5C In the example of , the system has been customized to provide corrective feedback only for the body part with the highest priority P10. In the case where corrective feedback is also provided for the priority P9 body part, the system will also create corrective feedback about the right elbow (e.g., "lower your right elbow"). Instead of using priority values ​​(e.g., P0 to P10) for priority annotations of body parts, the body parts can also be organized in a hierarchical structure, where the root node indicates the body part with the highest priority, and each body part parent node has a higher priority than its child nodes.

[0079] In other words, in order to generate the most appropriate correction feedback to the user, the posture checker can use the priority annotation in the posture data file or associated with the posture data file to select the correction feedback associated with the body part with the highest priority for the current exercise. In this example, it is assumed that the position of the right knee is the most critical for performing the correct posture, and therefore, the selected correction feedback 520pf1 is related to the right knee. The priority annotation reflects which body parts are important and which body parts are not so important for a specific exercise. For example, for the squat exercise, the priority annotation can indicate that the position of the knee is the most important for evaluating the correctness of the exercise, and the position of the arm can be annotated as unimportant. In this case, even if the arm is not moved according to the reference frame sequence, positive feedback will also be given. For example, in the reference video, the coach can keep the arm in a straight horizontal position, and the user can fold his / her arm behind the head or can simply let the arm hang down. In all cases, if the arm annotation has a low priority for exercise, correction feedback will not be given. However, if the straight horizontal position of the arms is deemed critical to the correct performance of the exercise, they may be annotated with a similar priority as the knees, and any negative assessment of the arm movement will similarly result in corrective feedback to the user.

[0080] Fig. 6A , 6BA scenario is depicted in which deviations in exercise performance from a reference frame sequence 610ru, 610rd do not result in corrective feedback because the deviated body part is not annotated as being relevant to achieving the desired therapeutic effect when performing the exercise. In this example, the coach in the reference frame sequence 610ru, 610rd places his left hand on his left chest in two postures, and the user 3 performs two postures with his left arm hanging down. However, the left hand and left elbow are annotated with the "no relevance" priority annotation of the exercise 620pn. Therefore, in both cases, positive feedback 620fb is provided to the user to indicate that no correction is needed to improve the posture performed by the user 3. The system 100 can use different modalities to communicate corrective feedback to the user 3. As described above, a simple text message 520pf1 can be incorporated into the corresponding output frame 520oc. Alternatively, the posture checker 130 can incorporate a graphical object 520pf2 into the output presented. In this example, the right knee is identified as the most relevant body part to be corrected because it needs to be lifted higher. For example, an arrow starting from the corresponding joint and pointing to the direction of the desired correction is used as such a graphic feedback object. In another embodiment, the system can output a voice instruction 520pf3 as correction feedback via a speaker coupled to the system 100 for communication. The voice instruction can be the same as the text message, or it can be different from the text message. It should be noted that various feedback modes using multiple modalities can be combined. For example, a graphic object 520pf2 can be displayed, and a voice instruction 520pf3 can be issued at the same time. The user can also customize the system 100 in terms of selecting a combination of correction feedback modalities that best suits the user. For example, a visually impaired person may be only interested in voice output, while a hearing-impaired person may be only interested in visual correction feedback.

[0081] Fig. 9 9 is a diagram illustrating examples of general purpose computer devices 900 and general purpose mobile computer devices 950 that can be used with the techniques described herein. Computing device 900 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Ideally, device 900 has a GPU suitable for processing machine learning algorithms. General purpose computer device 900 may correspond to Figure 1AThe computer system 100 of the present invention. The computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. For example, the computing device 950 can be used as a GUI front end for a user to capture test input images and provide them to the computer device 900, and in turn receive the predicted PDCU value from the computer device. The components shown here, their connections and relationships, and their functions are exemplary only and are not meant to limit the embodiments of the present invention described and / or claimed in this document.

[0082] The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connected to the memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connected to a low-speed bus 914 and the storage device 906. Each of the components 902, 904, 906, 908, 910 and 912 is interconnected using various buses and can be installed on a common motherboard or installed appropriately in other ways. The processor 902 can process instructions for execution within the computing device 900, including instructions stored in the memory 904 or on the storage device 906, to display graphical information of a GUI on an external input / output device (such as a display 916 coupled to the high-speed interface 908). In other embodiments, multiple processing units and / or multiple buses and multiple memories and memory types can be used appropriately. Moreover, multiple computing devices 900 can be connected, each of which provides a portion of the necessary operations (for example, as a server library, a blade server group or a processing device).

[0083] The memory 904 stores information within the computing device 900. In one embodiment, the memory 904 is a volatile memory unit(s). In another embodiment, the memory 904 is a non-volatile memory unit(s). The memory 904 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0084] Storage device 906 can provide mass storage for computing device 900. In one embodiment, storage device 906 can be a computer-readable medium or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state storage device, or an array of devices (including devices in a storage area network or other configuration). A computer program product can be tangibly embodied in an information carrier. A computer program product can also contain instructions that, when executed, perform one or more methods (such as those described above). The information carrier is a computer or machine-readable medium, such as memory 904, storage device 906, or memory on processor 902.

[0085] The high-speed controller 908 manages bandwidth-intensive operations of the computing device 900, while the low-speed controller 912 manages less bandwidth-intensive operations. Such functional allocation is exemplary only. In one embodiment, the high-speed controller 908 is coupled to the memory 904, the display 916 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 910 that can accept various expansion cards (not shown). In this embodiment, the low-speed controller 912 is coupled to the storage device 906 and the low-speed expansion port 914. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0086] The computing device 900 can be implemented in a variety of different forms, as shown. For example, it can be implemented as a standard server 920, or multiple implementations in a group of such servers. It can also be implemented as part of a rack server system 924. In addition, it can be implemented in a personal computer such as a laptop computer 922. Alternatively, the components of the computing device 900 can be combined with other components (not shown) in a mobile device (such as device 950). Each of such devices can contain one or more of the computing devices 900, 950, and the entire system can be composed of multiple computing devices 900, 950 that communicate with each other.

[0087] The computing device 950 includes a processor 952, a memory 964, an input / output device (such as a display 954), a communication interface 966, and a transceiver 968, among other components. The device 950 may also be provided with a storage device, such as a micro drive or other device, to provide additional storage. Each of the components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and several of the components may be mounted on a common motherboard or otherwise appropriately mounted.

[0088] Processor 952 can execute instructions within computing device 950, including instructions stored in memory 964. The processor can be implemented as a chipset of chips that include separate and multiple analog and digital processing units. The processor can provide, for example, coordination of other components of device 950, such as control of a user interface, applications run by device 950, and coordination between wireless communications performed by device 950.

[0089] The processor 952 can communicate with the user through the control interface 958 and the display interface 956 coupled to the display 954. The display 954 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 956 can include appropriate circuits for driving the display 954 to present graphics and other information to the user. The control interface 958 can receive commands from the user and convert them to submit to the processor 952. In addition, an external interface 962 in communication with the processor 952 can be provided to enable near-field communication of the device 950 with other devices. The external interface 962 can, for example, provide wired communication in some embodiments, or provide wireless communication in other embodiments, and multiple interfaces can also be used.

[0090] The memory 964 stores information within the computing device 950. The memory 964 may be implemented as one or more of (one or more) computer-readable media, (one or more) volatile memory units, or (one or more) non-volatile memory units. An expansion memory 984 may also be provided and connected to the device 950 via an expansion interface 982, which may include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 984 may provide additional storage space for the device 950, or may also store applications or other information for the device 950. Specifically, the expansion memory 984 may include instructions to perform or supplement the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 984 may act as a security module for the device 950, and may be programmed with instructions that allow the secure use of the device 950. In addition, a security application may be provided via a SIMM card together with additional information, such as placing identification information on a SIMM card in a non-decipherable manner.

[0091] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods (such as those described above). The information carrier is a computer or machine readable medium, such as memory 964, expansion memory 984, or memory on processor 952, which may be received, for example, via transceiver 968 or external interface 962.

[0092] The device 950 can communicate wirelessly via a communication interface 966, which may include digital signal processing circuitry where necessary. The communication interface 966 may provide communications under various modes or protocols (such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc.). Such communications may occur, for example, via a radio frequency transceiver 968. In addition, short-range communications may occur, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 980 may provide additional navigation and location-related wireless data to the device 950, which may be used appropriately by applications running on the device 950.

[0093] Device 950 may also communicate audibly using audio codec 960, which may receive verbal information from a user and convert it into usable digital information. Audio codec 960 may similarly generate audible sounds for the user, such as through a speaker, such as a speaker in a telephone handset of device 950. Such sounds may include sounds from a voice cell phone call, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 950.

[0094] The computing device 950 can be implemented in many different forms, as shown in the figure. For example, it can be implemented as a cellular phone 980. It can also be implemented as a part of a smart phone 982, a personal digital assistant or other similar mobile device.

[0095] Various embodiments of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0096] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0098] The systems and techniques described herein may be implemented in a computing device that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0099] Computing devices may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0100] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the invention.

[0101] In addition, the logic flows depicted in the accompanying drawings do not require the particular order or sequence shown to achieve the desired results. In addition, other steps can be provided, or steps can be excluded from the described processes, and other components can be added to or removed from the described systems. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (1000) for matching a test frame sequence (201) to a reference frame sequence (202), the reference frame sequence (202) demonstrating a physical exercise performed by a first person, the method comprising: Receiving (1100) a posture data file (203), the posture data file representing a plurality of two-dimensional skeleton projections (10p1, 10p2, 10p1', 10p2') on a virtual spherical surface (203s, 203') based on a list of three-dimensional joint coordinates representing positions of a plurality of joints of the three-dimensional skeleton (10) of the first person in corresponding frames (Fn), for at least one subset of frames (Fr1 to Frn) of the reference frame sequence (202), wherein a specific joint (10cj) of the three-dimensional skeleton is located at a geometric center of the spherical surface (203s, 203s'), wherein each two-dimensional skeleton projection (10p1, 10p2) of a specific frame of the subset (Fr1 to Frn) corresponds to a two-dimensional reference posture image of the three-dimensional skeleton (10) of the first person from different field of view angles, wherein the two-dimensional reference posture image is a characteristic posture of the physical exercise; receiving (1200) the test frame sequence (201), the test frame sequence representing the movement of the second person (2) when imitating the physical exercise (t1-t2-t3), the test frame sequence being captured by a standard RGB camera device (200) at a specific angle; detecting (1300) a two-dimensional skeleton (20) of the second person (2) in a current test frame of the sequence of test frames using a real-time two-dimensional skeleton detector, wherein the two-dimensional skeleton of the second person is a two-dimensional representation of a pose of the second person in a corresponding test frame; and Selecting (1400) a specific two-dimensional skeleton projection (10p1) of the first person to match a current pose of the second person (2) in the current test frame (Fc) with a corresponding reference pose representation in the pose data file, the specific two-dimensional skeleton projection representing the corresponding reference pose at the field of view angle corresponding to the specific angle of the standard RGB camera device, wherein the selecting (1400) is achieved by any of the following means: selecting a specific two-dimensional skeleton projection (10p1) of the first person having the smallest mathematical distance to the two-dimensional skeleton (20) of the second person in the current test frame (Fc); or The two-dimensional skeleton of the second person in the current test frame (Fc) is provided to a neural network (122) to predict a view angle (r, Φ, θ) associated with the current test frame, the neural network (122) being trained with a plurality of training frames showing a user's posture while performing an exercise as input, wherein each training frame is trained with a corresponding view angle (r, Φ, θ) θ) is annotated as the ground truth; and selecting the specific two-dimensional skeleton projection (10p1) of the first person located at the spherical coordinates corresponding to the predicted viewing angle (r, Φ, θ), wherein the viewing angle (r, Φ, θ) is described by the distance r between the user and the camera and the field of view, which is defined by the azimuth angle (Φ) and the polar angle (θ).

2. The method according to claim 1, further comprising: A frame (Fc') including the current test frame (Fc) and a representation of the corresponding two-dimensional skeleton of the second person is visualized (1450) to the second person in near real time.

3. The method according to claim 1 or 2, further comprising: determining (1500) a mathematical distance of the current posture to the corresponding reference posture, wherein the distance is a measure indicating whether the second person is performing the physical exercise correctly; and In case the physical exercise is not performed correctly, posture correction feedback (10 pf) is indicated (1600) to the second person on how to correct the current posture.

4. The method of claim 3, wherein the pose data file includes annotations for each frame grouping joint subsets into corresponding body parts, the method further comprising: indicating, for the current test frame, a body part exceeding a predetermined critical distance; and Feedback is provided regarding how the indicated body part changes the current posture until the current distance of the body part drops below the critical distance.

5. The method according to claim 3 or 4, wherein the posture correction feedback (10pf) is output to the second person (2) as visual information or sound information.

6. A computer program product for matching a test frame sequence with a reference frame sequence, wherein the reference frame sequence demonstrates physical exercise performed by a first person, and when the computer program product is loaded into the memory of a computing device and executed by at least one processor of the computing device, causes the at least one processor to perform the steps of the computer-implemented method according to any one of claims 1-5.

7. A computer system (100) for matching a test frame sequence (201) with a reference frame sequence (202), the reference frame sequence (202) demonstrating a physical exercise performed by a first person, the system comprising: An interface is configured to receive a pose data file (203), wherein the pose data file represents a plurality of two-dimensional skeleton projections (10p1, 10p2, 10p1', 10p2') on a virtual spherical surface (203s, 203') based on a list of three-dimensional joint coordinates representing positions of a plurality of joints of a three-dimensional skeleton (10) of the first person in corresponding frames (Fn), for at least one subset of frames (Fr1 to Frn) of the reference frame sequence (202), wherein a specific joint (10cj) of the three-dimensional skeleton is located at the geometric center of the spherical surface (203s), wherein the projections of the two-dimensional skeletons are represented by the projections of the two-dimensional skeletons on the virtual spherical surface (203s, 203'). At a specific frame of the subset (Fr1 to Frn), each two-dimensional skeleton projection (10p1, 10p2) on the spherical surface (203s) corresponds to a two-dimensional reference pose image of the three-dimensional skeleton (10) of the first person from different field of view angles (Φ, θ), wherein the two-dimensional reference pose image is a characteristic pose of the physical exercise, and the interface is further configured to receive the test frame sequence (201) representing the movement of the second person (2) when imitating the physical exercise (t1-t2-t3), the test frame sequence being captured by a standard RGB camera device (200) at a specific angle; a real-time two-dimensional skeleton detector module (110) configured to detect a two-dimensional skeleton (20) of the second person (2) in a current test frame of the sequence of test frames, wherein the two-dimensional skeleton of the second person is a two-dimensional representation of a posture of the second person in a corresponding test frame; and a pose matching module (120) configured to select (1400) a specific two-dimensional skeleton projection (10p1) of the first person to match a current pose of the second person (2) in the current test frame (Fc) with a corresponding reference pose representation of the pose data file, the specific two-dimensional skeleton projection representing the corresponding reference pose at a field of view angle (Φ, θ) corresponding to the specific angle of the standard RGB camera device, wherein the pose matching module (120) is configured to select the specific two-dimensional skeleton projection (10p1) of the first person having the smallest mathematical distance to the two-dimensional skeleton (20) of the second person in the current test frame (Fc); or Further comprising a neural network (122) having: an input layer (IL) for receiving a representation of the two-dimensional skeleton as a test input; and an output layer (OL) for predicting a view angle (r, Φ, θ) associated with the received test input, the neural network (122) being trained with a plurality of training frames showing a user's pose while performing an exercise as input, wherein each training frame is annotated with a corresponding view angle (r, Φ, θ) as ground truth, wherein The predicted viewing angle (r, Φ, θ) represents a pointer to the specific two-dimensional skeleton projection (10p1) of the first person to be selected, where the viewing angle (r, Φ, θ) is described by the distance r between the user and the camera and the field of view, which is defined by the azimuth angle (Φ) and the polar angle (θ).

8. The system according to claim 7, further comprising: A visualizer function configured to visualize, in near real time, to the second person (2), a frame (Fc') comprising the current test frame (Fc) and a representation of the corresponding two-dimensional skeleton of the second person.

9. The system according to claim 7 or 8, further comprising a posture checking module (130), wherein the posture checking module is configured to: determining a mathematical distance from the current posture to the corresponding reference posture, wherein the mathematical distance is a measure indicating whether the second person is performing the physical exercise correctly; and In case the physical exercise is not performed correctly, posture correction feedback (10 pf) is indicated to the second person on how to correct the current posture.

10. The system of claim 9, wherein the pose data file includes annotations for each frame grouping joint subsets into corresponding body parts, the pose checking module (130) being further configured to: indicating, for the current test frame, a body part exceeding a predetermined critical distance; and Feedback is provided regarding how the indicated body part changes the current posture until the current distance of the body part drops below the critical distance.

11. A system according to any one of claims 7 to 10, wherein the posture matching module (120) further comprises a normalizer function (121), which is configured to convert the detected 2D skeleton into a normalized two-dimensional skeleton, wherein the length of each bone of the 2D skeleton is divided by the user's height in the corresponding frame.

12. The system of claim 11, wherein the virtual sphere (203s) is normalized to a radius equal to one, wherein each two-dimensional skeleton projection (10p1, 10p2) is a normalized two-dimensional representation of the three-dimensional skeleton from a different field of view angle (Φ, θ).