Method, computer readable medium and system for synchronizing video playback with user motion

Abstract

Method, computer-readable medium, and system for synchronizing video playback with user motion. A computer-implemented method for coordinating a sensed gesture associated with real-time motion of a first object in a live video feed and a pre-recorded gesture associated with motion of a second object in the video is provided. The computer-implemented method includes: applying a matching function to determine a match between points of one of the sensed gestures and corresponding points of the pre-recorded gesture; and based on the match, determining a play time and outputting the play time for at least one frame of the video associated with the second object.

Description

Technical Field

[0001] Aspects of example implementations relate to methods, systems, and user experiences that provide video playback synchronized with user motion in real time to facilitate user observation of video associations. Background Art

[0002] In the prior art, a user may view a video that is intended to provide the user with training on performing an activity. For example, a prior art video may provide the user with training on a series of operations required to perform assembly of a kit of parts or facilitate the user to move his or her body in a prescribed manner. Some prior art videos may provide training on how a user may move his or her body to perform activities such as yoga, dance, physical therapy, etc.

[0003] The video playback method of the prior art can play a series of operations, and when the user needs to view a subset of the operations, it is difficult for the user to return to the desired position. For example, the user may need to rewind or fast-forward the video to the desired operation.

[0004] Prior art methods may have disadvantages or problems. For example, but not by way of limitation, requiring the user to rewind the video to a desired point can be awkward for the user's training experience and may divert the user's attention from the actual task the user is trying to learn from the video. The user is focused on performing the task of controlling the video, rather than providing video playback of training on performing a series of tasks.

[0005] Furthermore, in training activities such as yoga, dance, physical therapy, etc., prior art methods of controlling video prevent users from effectively using prior art video playback methods to perform learning. As a result of these prior art shortcomings, users often become discouraged and do not continue the activity, and may not be able to further improve skills in areas such as yoga or dance, or may actually continue to suffer physical discomfort or hinder improvement for video playback associated with physical therapy, etc.

[0006] Other prior art methods of controlling video may include voice commands. However, these methods also have disadvantages and problems. When the user must use or remember a voice command for rewinding or rewinding the video to a desired point, the user is distracted from the training activity, similar to the prior art methods that require manipulation using a remote control, etc. Prior art methods may also employ gesture-related commands, as well as handheld devices. However, these methods also have problems and disadvantages because the user may be unable to perform the training activity, particularly where the activity includes positioning of an arm, hand, finger, etc.

[0007] Other prior art methods may include the use of computer-generated graphics (e.g., avatars or animations of avatars) as an alternative to video playback. However, these prior art methods also have various disadvantages and problems. For example, but not by way of limitation, these prior art methods may have relatively high development costs. In addition, those methods do not provide the photorealism that users expect for training activities.

[0008] Therefore, there is an unmet need in the prior art to be able to track the user's movements and control and present the playback in an appropriate manner to provide feedback as to whether the user has successfully followed the training operations provided in the video without requiring the user to be distracted or stop the training activity. Summary of the invention

[0009] According to aspects of an example implementation, there is provided a computer-implemented method for coordinating sensed gestures associated with real-time movement of a first object in a live video feed with pre-recorded gestures associated with movement of a second object in the video, the computer-implemented method comprising the steps of: applying a matching function to determine a match between points of one of the sensed gestures and corresponding points of the pre-recorded gesture; and based on the match, determining a play time and outputting at least one frame of the video associated with the second object for the play time.

[0010] According to some aspects, the point of one of the sensed gestures and the corresponding point of the pre-recorded gesture include a single point of the first object and the second object in the first mode and multiple points of the first object and the second object in the second mode. Optionally, a user control is provided to select between the first mode and the second mode, and for the second mode, the number and position of the multiple points of the first object and the second object are selected.

[0011] According to other aspects, the first object includes a user receiving the video output and performing gestures in sequence to coordinate with pre-recorded gestures, and the second object includes a coach displayed in the video performing the pre-recorded gestures in sequence. In addition, the movement can include one of a physical therapy activity, at least a portion of an exercise, at least a portion of a dance, or at least a portion of a performance.

[0012] According to other aspects, the computer-implemented method includes: determining an instantaneous speed estimate over the time window based on the results of applying the matching function and determining the play time; applying a smoothing method to the instantaneous speed estimate to generate a play speed; and updating the play time based on the play speed. In addition, the smoothing method may include using a fixed smoothing factor determined empirically or a sliding average of a variable that changes based on the degree of motion of the second object over the time window.

[0013] According to another aspect, the video includes one or more visual feedback indicators associated with the pre-recorded gesture of the second object. Optionally, the one or more visual feedback indicators may include a track indicating the motion path of the corresponding point of the second object, a track indicating the relative speed and relative direction of the corresponding point of the second object along the track, and one or more of a retractable band indicating the difference between the position of the corresponding point of the second object and the position of the point of the first object.

[0014] Example implementations may also include a non-transitory computer-readable medium having a storage device and a processor capable of executing instructions for implementing the methods of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Various aspects of a system according to example implementations are shown.

[0016] Figure 2(a) to Figure 2(c) Various aspects of a system according to example implementations are shown.

[0017] Figure 3 Example hardware aspects are shown for use in some example implementations.

[0018] Figure 4 Example processes for some example implementations are shown.

[0019] Figure 5 An example computing environment is shown with an example computer device suitable for use with some example implementations.

[0020] Figure 6 An example environment suitable for some example implementations is shown. DETAILED DESCRIPTION

[0021] The following detailed description provides further details of the drawings and example implementations of the present application. For clarity, the reference numerals and descriptions of redundant elements between the drawings are omitted. The terms used throughout the specification are provided as examples and are not intended to be limiting.

[0022] Various aspects of example implementations relate to methods and systems for video playback. More specifically, example implementations include a video playback scheme in which playback of a video is synchronized in real time with the movements and actions of a user who may be viewing the video. According to this example implementation, a user may be able to participate in and focus on physical activities and or tasks associated with video playback at a pace of their choice while viewing the video, and without the need to distract themselves in order to rewind or seek to a specific point in the video playback. Example implementations may be applied to a variety of videos, including but not limited to physical instructions such as dancing, yoga, tai chi, etc., as well as physical therapy, how-to videos, and entertainment.

[0023] According to these example implementations, the system tracks the user's movements in real time during playback and provides the user with output that reacts to the movements to play the video in a manner that matches or coordinates with the user's body movements. Thus, the example implementations will provide the user with control over the video playback of a video, which may include, but is not limited to, a recorded expert performance. More specifically, through the user's movements, the system provides the user with the experience of being in the position of the expert in the video being played back. The example implementations are provided without the use of computer-generated graphics or avatars.

[0024] In an example implementation, the video playback system tracks motion in the environment. More specifically, the user's motion is tracked in a manner that allows the user to continue playing the video or interact with the video in a manner that provides a useful feedback mechanism without the user having to divert attention from the physical activity. For example, but not by way of limitation, the video may include an expert performing a physical activity. In one example implementation, a user may demonstrate a "cloud hand" (a move in Tai Chi) that the user observing the video is trying to learn or practice.

[0025] Figure 1 An example implementation 100 is shown. In the example implementation 100, a user 101 is viewing a display 103, such as a video screen, a projected screen, etc. One or more video cameras 105 are provided, which sense the user 101. A speaker may also be provided. More specifically, the user's 101 physical activity, such as physical movement in response to information provided on the display 103, is sensed.

[0026] In display 103, video playback includes object 109. In this example implementation, object 109 is a pre-recorded video of a teacher performing one or more movements. Video playback is synchronized with the movement of user 101. For example, but not by way of limitation, if the user increases his or her speed of movement, the playback speed of the video also increases at a corresponding rate. Similarly, if user 101 slows down, stops or moves in reverse, the video will correspondingly slow down, stop or move in reverse. As a result, if user 101 moves slower than the speed of object 109 in the video, the video reacts to the speed difference and slows down. As a result, user 109 does not need to rewind or reverse the position of the video playback to a previous point, and the attention of user 101 will not change or be scattered.

[0027] Additionally, visual indicators 111, 113 are provided to give user 101 feedback on her position, as well as corrective motion (e.g., user 101 moves her hands as shown at 115 and 117). For example, but not by way of limitation, in Figure 1 In the “cloud hands” example shown, as the user 101 moves, the video plays together so that the posture of the object 109 shown in the video is synchronized with the movement of the user 101.

[0028] To implement the example implementation, the motion of the object 109 in the video is tracked, and the real-time motion of the user 101 is tracked. In the example implementation, the body of the user 101 is being tracked. The tracking provides estimates of the joint positions (e.g., 115 and 117) associated with the body of the user 101 at various times. However, the example implementation is not limited thereto, and other objects may replace the joint positions of the user without departing from the scope of the invention, such as tracking individuals or objects in the video, and in addition to the object 109 being the body.

[0029] The estimates generated in example implementations are described in more detail below. For example, the estimates may be generated in 3D world space or 2D image space. Furthermore, the estimates may be generated concurrently with the recording of the video (e.g., live with the initial recording of the video) prior to playback to the user. Furthermore, as an alternative to or in addition to the live initial recording estimates, the estimates may be generated after the initial video has been recorded.

[0030] Figure 2(a) to Figure 2(c) Additional example descriptions 200 of visual indicators according to an example implementation are shown. For example, as shown in FIG. 2( a), at 201, an object 109 such as a teacher (in this example implementation, a Tai Chi master) has various indicators on a display 103 shown in a video. More specifically, colored tracks 203 (green), 205 (vermillion) show the movement of the master's hands as he demonstrates the movements. Arrows 207 (purple), 209 (purple) are provided that indicate the relative speed and direction along the respective tracks 203, 205.

[0031] As shown in Fig. 2(b), at 211, a plurality of users 101 are sensed and tracked by a video camera. Signals received from the video camera are processed to indicate colored tracks 213 (vermilion), 215 (green), 221 (green), 223 (vermilion) indicating past and suggested future movements of the hands of the users 101. Arrows 217 (purple), 219 (purple), 225 (purple), 227 (purple) are provided to show the speed and direction of the movement of the hands of the users 101 along the corresponding tracks.

[0032] As shown in FIG2(c), the subject 109 of the video (in this case, the master (e.g., coach)) is shown with colored tracks 231 (green), 233 (vermilion) and arrows indicating relative speed and direction along the tracks 235 (purple), 237 (purple). In addition, indicators 239, 231 (sometimes referred to as "stretch bands") indicate the relative position of the user's 101 hand compared to the matching joint from the recorded video of FIG2(a), as shown in FIG2(c) above.

[0033] With the above example implementation, no additional visual feedback is provided or required for further training of the user, other than the playback of the video itself. Optionally, the above indicator is associated with the degree of match of the user's real-time movement compared to the movement of the corresponding part in the video. For example, but not by way of limitation, if the synchronization between the user and the pre-recorded video is based on the position of one or more joints or positions on the user's body (e.g., the position of the hand), the video displayed to the user may show the trajectory of the hand in the recorded video, as well as instructions on how the user can follow those trajectories.

[0034] Thus, the user can move his or her hand along the trajectory at the user's desired speed or even in the opposite direction of the trajectory in the video, and the video in conjunction with the camera follows the user's activities.As also explained above, in addition to the trajectory itself, arrows may be provided to indicate the direction and speed of the trajectory.

[0035] Similarly, the user may select various display options with respect to viewing of the video. According to one example implementation, the user may only view the recorded video and follow (or lead) the recorded video. According to another example implementation, the user may view the recorded video while viewing their own video in live mode to be able to view visual feedback of their own movement alongside the recorded video. In another example implementation, when the recorded video and the live video are displayed simultaneously as described herein, movement feedback may be provided in the live video in addition to or as an alternative to displaying movement feedback in the recorded video.

[0036] According to another example implementation, the user's movements may be recorded while the user is following a video or moving freely. The recording of the movements associated with the user may then be used by another user and / or by a teacher. For example, but not by way of limitation, the example implementation may be used in a telemedicine example implementation.

[0037] In this method, a physical therapist can record a therapeutic movement (e.g., exercise) for the patient to perform. The patient can watch a video to perform the exercise or movement while recording it. After performing the exercise or movement, the recording can be provided to the physical therapist. The physical therapist can later review the patient's movement and provide additional movement or feedback on the recorded movement. In addition, the physical therapist can use a pre-recorded video compared to the user's recorded video and incorporate tracks, arrows, and retractable bands as described above to provide detailed guidance and feedback to the user.

[0038] Although the above example implementations involve the movement of the user's body in real-time synchronization with the movement of the body of the object in the video, the example implementations are not limited thereto, and other example implementations may be substituted without departing from the scope of the invention. For example, but not by way of limitation, a pre-recorded video may include multiple people and tracked objects. In addition, during playback, the user's finger position may be tracked and used to drag the video based on the movement trajectory of the finger position.

[0039] Figure 3 A schematic diagram of a system according to an example implementation is shown. More specifically, the system is physically located in an area 300 so that a user 301 having joints 303 and 305 can perform physical activities that require movement of one or more joints 303 and 305. Area 300 also includes a display 307 that the user can view. A playback of a previously recorded video is output to display 307 and includes a series of movements or gestures that the user 301 is trying to follow or learn.

[0040] A sensor 309, such as a video camera, is provided with the display 307. The sensor 309 may be separate from the display 307, or may be integrally formed with the display 307, such as a camera device built into the display 307. The sensor 309 senses the motion of the user 301, including the motion of the joints 303 and 305 that the user moves based on the output of the pre-recorded video on the display 307. Optionally, for example, audio outputs 311a, 311b (e.g., speakers, etc.) may also be provided so that the user may also hear guidance, commands, music, or other audio information associated with the pre-recorded video on the display 307.

[0041] The display 307, camera 309, and optional audio outputs 311a, 311b may be communicatively coupled to a processor 313 via wired or wireless communications. For example, but not by way of limitation, the processor 313 may be a personal computer or laptop computer located in the area 300. Alternatively, the processor 313 may be a server located remotely from the area 300. The processor 313 may be used to adjust one or more settings associated with the display 307 and / or the sensor 309 to adjust, for example, but not by way of limitation, a matching mode, a playback mode, a timing mode, a speed, a track, an arrow, a retractable band, or other characteristics associated with the output of the display 307 and the input of the sensor 309.

[0042] More specifically, the display 307 may include an object 315, such as a pre-recorded video of a coach performing a sport or activity. In this example implementation, the object 315 is shown with joints 317, 319, and the user 301 follows the movement of the joints 317, 319 and attempts to move his or her joints 303, 305. In addition, as described above, indicators 321, 323 such as tracks, arrows, or retractable bands may also be provided alone or in combination.

[0043] As described above, an example implementation involves performing a matching operation between a video including a first sequence of video gestures captured in real time while the video is being recorded or generated after offline processing of the video. A second sequence of video gestures is obtained in real time as live tracking during use to generate a second sequence of video gestures.

[0044] For the case where the first sequence of poses and the second sequence of poses are each a set of 2D image positions of tracked joints, a basic playback mode is provided for each new time step in the sequence to indicate the frames of the first sequence that are most similar to the frames of the second sequence. In order to obtain the frames, a matching function is performed. The matching function may be different for different modes or applications. However, according to an example implementation, focus is placed on specified joint positions and other joint positions are ignored. In this method, the match will be indicated by the Euclidean distance of the image coordinates of the specified joint positions. For example, the joint positions can be those associated with the right hand, left hand, or other body parts.

[0045] More specifically, the first sequence can be represented as a sequence of video poses Pv(t) available for the video, and the second sequence can be represented as a sequence of live poses PL(t c ), where t C represents the real-time clock time and t represents the playback time in the video. For Pv(t) and PL(t), each being a set of 2D image positions of a tracked joint, the basic playback mode shows that for each new time step t C , the recorded video has the same C ) is the frame with the most similar pose Pv(t). For example, for the frame shown at time t*, equation (1) provides the following relationship:

[0046] t*=argmin match[Pv(t), PL(tc)] (1)

[0047] where match[.,.] is the matching function. For example, match[Pv(t),PL(tc)] can be the Euclidean distance between the image coordinates of the right hand position. If P(t)[j] represents the 2D position of joint j, then for this example implementation, the matching function is represented by the following equation (2):

[0048] match[Pv(t), PL(tc)]=dist[Pv(t)['right hand'], PL(tc)['right hand']] (2)

[0049] The example implementation provides one or more matching modes. For example, but not by way of limitation, single joint point and drag matching modes may be provided, and a multi-joint tracking mode may be provided.

[0050] According to the single joint point and drag mode of the example implementation, a joint such as a hand can be used to select a point in the video to be tracked, and the user moves the specific joint to fast-forward along the trajectory of the joint in the video. In order to help the user make the correct movement without distracting attention, a portion of the trajectory is displayed in the video, and the user can understand where to move his or her hand in real time. According to the single joint point and drag mode, when the distance from the hand point to the trajectory is less than the start threshold, the drag is performed, and when the distance from the hand point to the trajectory is greater than the stop threshold, the drag is released.

[0051] Alternative methods may be used, including but not limited to using specific gestures or hand movements, such as making and loosening a fist corresponding to start and stop. Thus, the user's joint is used as a pointer in real time, which can also be used to drag objects or joints in the recorded video. Furthermore, the dragged joint is not limited to the joint used for the pointer in the video.

[0052] According to another example implementation, multiple joints may be tracked on the user, and the tracked multiple joints of the user may be matched with corresponding joints in the recorded video. To implement this method, a matching function of the relationship identifier in the following equation (3) may be provided:

[0053] match[Pv(t), PL(tc)]=Σdist[Pv(t)[j], PL(tc)[j]]2 (3)

[0054] Wherein, J represents the set of joints to be tracked, such as the left hand and the right hand. According to the matching function, the set of joints to be tracked may include at least a single joint and at most a complete set of joints to be tracked. Although using fewer joints may be cognitively easier for the user to maintain attention and understand feedback, as well as to adapt the movement to the record they are being trained, using more joints may provide the user with a more comprehensive characterization of the movement, and therefore a more complete training opportunity.

[0055] According to one example implementation, a user may be able to control the number of joints tracked based on their focus level or their intention regarding the training opportunity. For example, but not by way of limitation, in the early training stages where one or a few joints are selected, the user may focus on understanding the feedback to adapt fine movements to the video recording, and then gradually increase the number of joints to obtain a more comprehensive training approach.

[0056] In addition, example implementations may include one or more playback timing modes. For example, according to a first mode, a pre-recorded video provides a basis for the user to follow the motion in the pre-recorded video (e.g., prior art methods). According to a second mode, the user generates the motion, and the video tracks the user. According to a third mode, the user also generates the motion, and the video tracks the speed.

[0057] In the first mode (e.g., the prior art method), a pre-recorded video is played, for example, at a fixed rate. The user follows the motion of the object in the pre-recorded video in real time and tries to maintain his or her motion using the pre-recorded video. For each time step t C , the first mode can be expressed by the following formula (4):

[0058] playTime=playTime+playSpeed*Δt (4)

[0059] According to an exemplary implementation in the first mode, the recorded frames are displayed with a playing time.

[0060] According to this example implementation, a user interface element may be presented to the user that allows the user to set or change certain variables. For example, but not by way of limitation, the play time or play speed may be modified by the user. In this example implementation, the system does not track the user. However, if the user falls behind the pace of the video and wants to rewind or slow down the video, he or she must explicitly control the video to revert to an earlier play time or change the speed.

[0061] According to the second mode, the pre-recorded video can be paused and rewound to match the motion associated with one aspect of the user's motion detected by the system. Thus, the playback time is determined by the user's motion. In addition, the continuity of the playback speed is based on the user's motion. As described above, for each real-time t C , the time of the best matching frame is calculated by equation (1) shown and described above, and the frame starting from t* is used.

[0062] More specifically, for each time step t C , observe the live frame, and determine the value of PL(tc). In addition, determine the value of t* as described above with respect to equation (1). Therefore, the play time is set to t*, and the recorded frame of that play time is displayed.

[0063] According to a third mode, the video playback system tracks the motion of the user along the track of the video. Based on the tracking, the user's motion speed is estimated. The speed can be positive forward motion on the track, negative or reverse motion on the track, or zero for a user paused along the track. More specifically, an instantaneous speed estimate is determined over a time window based on the matching function and the result of determining the play time. A smoothing method is applied to the instantaneous speed estimate to generate a play speed, and the play time is updated based on the play speed. In addition, the smoothing method may include a running average of a variable that changes based on an empirically determined fixed smoothing factor or a second object's degree of motion over a time window.

[0064] According to this example implementation, for each time step t C , observe the live frame associated with the user's activity, and determine PL(tc). In addition, t* is determined according to equation (1) as described above. In addition, the value of the instantaneous speed estimate s is determined based on the following equation (5):

[0065] s=(t*-playTime) / Δt (5)

[0066] Based on the determined value of S, the playback speed is defined by the following equation (6):

[0067] playSpeed=w*s+(1-w)*playSpeed ​​(6)

[0068] In equation (6) above, w is provided as a smoothing weight that determines how quickly the running estimate of speed adapts to the newly observed speed estimate. For example, a value of 1 for w can effectively eliminate the speed pattern aspect. In this case, each time step of a frame of t* will be generated. On the other hand, when the system is tracking the user's motion speed, a value of w close to zero will result in a more significant degree of smoothing being indicated.

[0069] Therefore, the playback speed is used to determine the playback time, as shown in the following formula (7):

[0070] playTime=playTime+playSpeed*Δt (7)

[0071] Based on the above, the recorded frames are displayed for the playback time.

[0072] In the above example implementation associated with the third mode, the value of w is associated with the strength of the change in the detected position of the user in terms of its impact on the updated playback speed estimate. For example, but not by way of limitation, w can be a fixed value determined empirically. Alternatively, w can be determined in another way that is not a constant, but can be associated with how fast the trajectory in the recording changes with respect to time. For example, the speed can be defined in the gesture space of the recorded trajectory motion at time t according to equation (8):

[0073] v(t)=dist[Pv(t), Pv(t+Δt)] / Δt (8)

[0074] A larger value of v(t) indicates a more significant amount of motion. The detected position change provides an estimate of the change in time, and therefore, w may be indicated as being higher. On the other hand, a smaller value of v(t) may indicate that there was little motion for a period of time. In this case, the estimate of speed should not be updated when there is little motion, and the value of w may be maintained close to zero so that when motion resumes, the video continues to play at the same speed until a later time in the recording.

[0075] In the case where the trajectory interacts with itself or generates "flicker" near an intersection, where a slight disturbance in the tracked position may result in discontinuity in video playback, consecutive frames may be presented from significantly different recording times. To address this event, items may be added to the match score in a manner that does not favor the time difference from the current playback time. In addition, a synchronized playback mode may be provided in which the system maintains continuity but adapts the playback speed to follow the user. In addition, tracking of additional joints may be added. In the case where there are time periods in the recording and there is no apparent movement of the tracked joints, the third mode as described above may be used in conjunction with the motion-related update rate also described above.

[0076] According to some example implementations, a user may position himself or herself so that the location of his or her joints in the video image can be used to control the system. For example, but not by way of limitation, the user may stand in a posture where his or her joints are positioned close to the joints in the video that the user may wish to control or associate with.

[0077] Alternatively, a transformation may be applied from the tracked joint positions to registered positions that are approximately aligned with the joints in the video. For example, but not by way of limitation, the tracked joints may be translated by a translation function T that aligns the head position, center of mass, joint points, etc. Registration may be performed at the start of use and updated upon explicit request or periodically (e.g., every frame). For updates that occur frequently, update weights may be set for small amounts of motion in a time period so that the relative time scale associated with the registration is slower than the time scale of typical motion of the activity associated with the training.

[0078] For 3D positions, the registration may implement a Procrustes fit to obtain a least squares alignment of points with respect to translation, rotation, and optionally scaling transformations. Additional registration methods may employ methods for motion adaptation between skeletonized models. Advantages may exist by using a 3D example implementation. For example, but not by way of limitation, when a user is performing motions that involve forward and backward motions relative to the position of the camera, joint tracking may be significantly more effective in detecting and processing these motions with precision and accuracy compared to a 2D example implementation.

[0079] According to an example implementation, the system can automatically recognize when a user switches between, for example, one exercise and another exercise. In one example implementation, the motion being performed live can be used as a query to a library of information such as historical data, and the system can automatically learn or predict activity using a matching function.

[0080] Additionally, the user's activity being sensed by the system can be recorded and provided to a coach or, for example, a physiotherapist. Thus, the user's activity can be recorded, fed through a tracker and converted into recorded content, and provided to a coach for comparison with pre-recorded video using a matching function.

[0081] Figure 4 An example process 400 is shown according to an example implementation. As described herein, the example process 400 can be performed on one or more devices.

[0082] At 401, a playback video or training video is received. For example, but not by way of limitation, the training video may be a recording of a coach performing a movement. During training, the training video will be displayed to the user on a display.

[0083] At 403, the training video is processed to define a sequence of video gestures in the training video. This processing can be performed in real time as the video is recorded. Alternatively, the video gestures can be captured after the training video has been recorded (e.g., during offline processing of the video). As described above, the gestures are associated with the tracking of one or more joints.

[0084] Therefore, after 403, the training video has been generated. The training video can be used by one or more users in the process of training the movement performed in the video content. Operations 401 and 403 only need to be performed when the video is generated for the first time, and do not need to be performed every time the user watches the video.

[0085] At 405, the user performs a motion sensed by the sensor. For example, but not by way of limitation, the sensor may be a camera that senses the user's real-time motion as a video input. Depending on the playback timing mode, the user may perform the motion simultaneously with the video playback, with the video leading and the user following. Alternatively, as described herein, the user may lead and the video may track the user.

[0086] Live tracking is also performed in real time from the sensed video input at 407. As a result of the live tracking, one or more live gestures of the user are detected in real time in the sensed video input.

[0087] Once one or more live gestures of the user are detected in real time in the sensed video input, and as described above, the video gestures of the training video have been previously defined, a matching function can be applied at 409. Aspects of the matching function are described above. As a result of the matching function, a play time (e.g., t*) is defined for a frame. Therefore, the playback of a pre-recorded playback video can be determined based on the play time based on the gesture of the tracked joint. As described above, the matching mode can include single joint point and dragging, multi-joint tracking, or a combination thereof. In this example implementation, the argmin function is used as described above. However, as will be appreciated by those skilled in the art, other functions may replace it.

[0088] At 411, it is evaluated whether the playback timing mode is a mode in which the video playback system tracks the user's motion and the trajectory of the video and estimates the speed of the motion (e.g., "speed mode"). As described below, if the playback timing mode is in "speed mode", operations 415 to 419 are performed. If the playback timing mode is not in "speed mode", operation 413 is performed.

[0089] For playback timing modes that are not in "speed mode," at 413, the recorded frames of the training video are output for the play time defined in operation 409. Operations then continue as the user continues to change his or her posture.

[0090] For playback timing mode in "Speed ​​Mode," at 415, an instantaneous speed estimate is determined as described above.

[0091] At 417, the determined instantaneous speed estimate is applied to the smoothing weight w as described above to determine the playback speed. Optionally, and as described above, the value of w may be determined empirically or based on a relationship between how fast the tracks in the recorded video change over time (e.g., v(t)).

[0092] At 419, the play time is updated to increase the play speed multiplied by the time interval, and then operation 413 is performed as described above.

[0093] Figure 5An example computing environment 500 is shown with an example computer device 505 suitable for use with some example implementations. The computing device 505 in the computing environment 500 may include one or more processing units, cores, or processors 510, memory 515 (e.g., RAM, ROM, etc.), internal storage 520 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 525, any of which may be coupled on a communication mechanism or bus 530 for communicating information or embedded in the computing device 505.

[0094] The computing device 505 may be communicatively coupled to an input / interface 535 and an output device / interface 540. Either or both of the input / interface 535 and the output device / interface 540 may be wired or wireless interfaces and may be detachable. The input / interface 535 may include any device, component, sensor, or interface (physical or virtual) that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Braille, motion sensors, optical readers, etc.).

[0095] Output devices / interfaces 540 may include displays, televisions, monitors, printers, speakers, Braille, etc. In some example implementations, input / interfaces 535 (e.g., user interfaces) and output devices / interfaces 540 may be embedded in or physically coupled to computing device 505. In other example implementations, other computing devices may serve as or provide functionality for input / interfaces 535 and output devices / interfaces 540 for computing device 505.

[0096] Examples of computing devices 505 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machines, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices that are not designed for mobility (e.g., desktop computers, server devices, other computers, kiosks, televisions, radios embedded with and / or coupled to one or more processors, etc.).

[0097] Computing device 505 may be communicatively coupled to external storage device 545 and network 550 (e.g., via I / O interface 525) for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 505 or any connected computing device may function as, provide services for, or be referred to as a server, client, thin server, general purpose machine, special purpose machine, or another label. For example, and not by way of limitation, network 550 may include a blockchain network and / or a cloud.

[0098] I / O interface 525 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11xs, universal system bus, WiMAX, modems, cellular network protocols, etc.) for communicating information to and / or from at least all connected components, devices, and networks in computing environment 500. Network 550 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0099] The computing device 505 may use and / or communicate using computer-usable or computer-readable media (including both transitory and non-transitory media). Transitory media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., magnetic disks and tapes), optical media (e.g., CD ROMs, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage devices), and other non-volatile storage devices or memories.

[0100] Computer device 505 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. Computer-executable instructions can be retrieved from transient media, as well as stored on and retrieved from non-transitory media. Executable instructions can originate from one or more of any programming, scripting, and machine languages ​​(e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0101] The processor 510 can be executed under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, which include a logic unit 555, an application programming interface (API) unit 560, an input unit 565, an output unit 570, a matching unit 575, a tracking unit 580, a playback timing unit 585, and an inter-unit communication mechanism 595 for different units to communicate with each other, with the OS, and with other applications (not shown).

[0102] For example, matching unit 575, tracking unit 580, and playback timing unit 585 may implement one or more of the processes shown above for the above structures. The design, function, configuration, or implementation of the described units and elements may vary and are not limited to the description provided.

[0103] In some example implementations, when information is received or instructions are executed via API unit 560, it may be communicated to one or more other units (eg, logic unit 555, input unit 565, matching unit 575, tracking unit 580, and playback timing unit 585).

[0104] For example, the matching unit 575 may receive and process information associated with a pre-recorded playback video with a captured video gesture and a real-time live video feed to detect the gesture. The output of the matching unit may provide a value indicating the time t* associated with the comparison of the tracked joint between the pre-recorded playback video and the user's live gesture. For example, the output of the matching unit 575 may be used to determine the play time, which in turn is used to determine how long the recorded frame should be displayed. Similarly, a tracking unit 580 may be provided to perform the function of tracking the user's motion speed along the track of the video as described above. In addition, a playback timing unit 585 may perform playback based on the information obtained from the matching unit 575 and the tracking unit 580.

[0105] In some cases, in some of the example implementations described above, the logic unit 555 may be configured to control the flow of information between the units and direct the services provided by the API unit 560, the input unit 565, the matching unit 575, the tracking unit 580, and the playback timing unit 585. For example, the flow of one or more processes or implementations may be controlled by the logic unit 555 alone or in conjunction with the API unit 560.

[0106] Figure 6 An example environment suitable for some example implementations is shown. Environment 600 includes devices 605 to 645, and each device is communicatively connected to at least one other device via, for example, network 660 (e.g., by wired and / or wireless connection). Some devices may be communicatively connected to one or more storage devices 630 and 645.

[0107] Examples of one or more devices 605 to 645 may be Figure 5 6. Devices 605 at 645 may include, but are not limited to, a computer 605 (e.g., a laptop computing device) with a monitor and associated webcam as described above, a mobile device 610 (e.g., a smart phone or tablet), a television 615, a device associated with a vehicle 620, a server computer 625, computing devices 635-640, storage devices 630 and 645.

[0108] In some implementations, devices 605-620 may be considered user devices associated with a user who may remotely receive a display of a training session at various locations. Devices 625-645 may be devices associated with a service provider (e.g., for storing and processing information associated with the operation of the display device, the sensing device, and settings associated with the matching mode and the playback timing mode). In this example implementation, one or more of these user devices may be associated with a sensor location proximate to the user to be able to sense the user's real-time movements and provide a real-time live video feed to the system to facilitate processing gestures based on tracking of one or more joints of the user as described above.

[0109] Example implementations may have various benefits and advantages. For example, but not by way of limitation, example implementations of the present invention do not require the user to hold an object such as a remote control or mouse that may alter the user's body movements, or cause the user to become distracted from the training process. Instead, example implementations track the user's body movements so that the user can maintain focus on the training activity.

[0110] Additionally, according to an example implementation, multi-joint tracking is provided. As described above, the user can select the number of joints to be tracked based on the user's intent (e.g., the user is only focusing on the movement of a single joint, as opposed to the user focusing on learning a series of movements as part of a more comprehensive training, or focusing on learning a single activity associated with a single track).

[0111] In addition, the example implementation provides multiple playback modes that can be interactively determined by the system itself or manually. For example, and not by way of limitation, the example implementation provides not only a mode where the user leads and the video follows, but also a mode where the video leads and the user follows, and a mutually synchronized mode. In contrast, prior art methods do not provide synchronization of the video with the user as described herein with respect to the example implementation.

[0112] In addition, according to the example implementation, there is no need to predefine events that the user must perform in order to advance to other parts of the training. Instead, through the motion analysis scheme described herein, the example implementation provides a matching of gestures from a pre-recorded video relative to the live motion of the user's body and the user's motion, so that the user can experience the actions from the video without having to predefine the actions.

[0113] According to various aspects of example implementations, simulation training may be used where a user performs a set of gestures or movements in real time in comparison to a prescribed set of training gestures or movements, without requiring the user to use any device to track joint positions for the presence of gestures.

[0114] According to another example implementation, the aspects described herein may be applied to entertainment activities. For example, a video of a celebrity performing or dancing may be a playback video, and the user may be a student of the performance or entertainment, aiming to learn the technique of the particular actor. By applying tracks, arrows, and / or retractable bands, the actor may learn to control the fine movements of the tracked joints to be able to imitate the style or technique of the celebrity. In addition, as the actor becomes more and more skilled in specific movements, a multi-joint function may be applied to allow the user to learn additional movements, and ultimately to be able to learn the entire body movement and movement of the celebrity. Since the speed of the pre-recorded video is based on the user's movement, the actor may practice certain movements, styles, or techniques without being distracted and having to rewind or return to a prescribed point in the video as in the prior art methods.

[0115] In addition, there may be other aspects and example implementations associated with the inventive concept. For example, but not by way of limitation, a query mechanism may be provided so that for a desired posture sequence, a search may be performed to find similar available online videos. In this example implementation, the user's real-time live video is replaced with pre-recorded online video clips that are compared to the pre-recorded video using a matching function to determine similarity and matching of joint motions.

[0116] In addition, in the case where the user performs one or more movements and the one or more movements performed by the user include multiple gestures, a query can be performed on the user's movement to determine the closest set of pre-recorded videos. The most relevant query results are then classified, and the user can be informed of how similar his or her movement looks to the most relevant categories. For example, but not by way of limitation, the user may attempt to perform a certain type of dance. However, the user's movement may be more similar to another type of dance that is different from the type of dance in the pre-recorded training video. The user can be informed that the dance he or she is performing is a different dance, and the user can be provided with the option of training for the different dance that is opposite to the dance in the pre-recorded training video.

[0117] Similarly, a user performing a certain type of exercise or physical therapy may be provided with an indication that the most relevant joint motion associated with the user is actually different from that in the pre-recorded video. In this case, the user may be recommended to try the type of exercise or physical therapy that is shown to be most relevant to the query, rather than the certain type of exercise or physical therapy selected by the user.

[0118] As an additional aspect of an example implementation, and as described above, "Speed ​​Pattern" may provide an indication that the user has paused an activity or that the motion has begun to slow down. The pause or slow down may be an indication that the user is becoming fatigued or has lost interest in the activity or is experiencing pain (e.g., in the case of physical therapy). Thus, for example, the results of "Speed ​​Pattern" may be used as an indicator to recommend that the user consider taking a break, trying another activity, or restarting a pre-recorded video.

[0119] Optionally, the numerical values ​​associated with the matching function can be used to generate similarity scores between the user's movements and the pre-recorded video. These scores can be used to determine which of a plurality of users engaging in a movement compared to a common pre-recorded video most closely matches the pre-recorded video. For example, in a competition to determine which user most closely imitates a prescribed set of movements associated with sports, dancing, acting, etc., the user with the higher score can be determined based on the numerical values ​​associated with the matching function.

[0120] According to other example implementations, various aspects may be integrated into an augmented reality or virtual reality scene. For example, but not by way of limitation, for evidence collection and identification of an individual, the body movement of an individual shown in a security camera recording may be compared to a real-time live video feed of the same location to determine the frequency of occurrence of individuals with gait and body movement. Furthermore, including the use of a 3D camera may allow a user in an augmented reality or virtual reality scene to be able to perform body movements in a manner appropriate to the scene.

[0121] In the above example implementations, various terms are used to describe aspects and features. However, the example implementations are not limited thereto, and other terms and descriptions may be substituted without departing from the scope of the invention. For example, but not by way of limitation, the computer-implemented method includes coordinating sensed gestures and pre-recorded gestures. The sensed gestures may be sensed by a 2D or 3D video camera as described above. In addition, the sensed gestures may be associated with the real-time motion of a first object in a live video feed (e.g., a user sensed by a video camera, but not limited thereto). The pre-recorded gestures may be associated with the motion of a second object (e.g., a coach in a video). However, the first object need not be a user, and the second object need not be a coach.

[0122] In addition, the computer-implemented method includes applying a matching function to determine a match between a point of one of the sensed gestures and a corresponding point of a pre-recorded gesture, and based on the match, determining a play time and outputting at least one frame of a video associated with a second object for the play time. In some example implementations, the point of one of the sensed gestures and the corresponding point of the pre-recorded gesture may be a joint or multiple joints of the user (e.g., a hand, a foot, etc.). In addition, as will be appreciated by those skilled in the art, the matching functions to be applied may include those described above, as well as other matching functions that may be substituted therefor.

[0123] Although some example implementations have been shown and described, these example implementations are provided in order to convey the subject matter described herein to those familiar with the art. It should be understood that the subject matter described herein can be implemented in various forms, without being limited to the described example implementations. The subject matter described herein can be practiced without those specifically defined or described matters or with other or different elements or matters that are not described. Those familiar with the art will understand that these exemplary implementations can be changed without departing from the subject matter defined in the attached claims and their equivalents as described herein.

Claims

1. A computer-implemented method for coordinating sensed gestures associated with real-time motion of a first object in a live video feed and pre-recorded gestures associated with motion of a second object in the video, the computer-implemented method comprising the steps of: applying a matching function to determine a match between points of one of the sensed gestures and corresponding points of the pre-recorded gestures; Based on the matching, determining a play time and outputting at least one frame of the video associated with the second object for the play time; determining an instantaneous speed estimate over a time window based on the play time determined by applying the matching function and a current play time; applying a smoothing method to the instantaneous speed estimate to generate a playback speed; as well as The current play time is updated based on the play speed.

2. The computer-implemented method of claim 1 , wherein: The point of one of the sensed gestures and the corresponding point of the pre-recorded gesture include a single point of the first object and the second object in a first mode and a plurality of points of the first object and the second object in a second mode.

3. The computer-implemented method of claim 2, wherein: User controls are provided to select between the first mode and the second mode, and for the second mode, the number and location of the plurality of points of the first object and the second object are selected.

4. The computer-implemented method of claim 1 , wherein: The first object includes a user who receives the output of the video and performs gestures in sequence to coordinate with the pre-recorded gestures, and the second object includes a coach displayed in the video who performs the pre-recorded gestures in sequence.

5. The computer-implemented method of claim 4, wherein: The movement includes one of a physical therapy activity, at least a portion of an exercise, at least a portion of a dance, or at least a portion of a performance.

6. The computer-implemented method of claim 1 , wherein: The smoothing method includes utilizing an empirically determined smoothing factor or a sliding average of a variable that changes based on the degree of motion of the second object over the time window.

7. The computer-implemented method of claim 1 , wherein: The video includes one or more visual feedback indicators associated with the pre-recorded gesture of the second object.

8. The computer-implemented method of claim 7, wherein: The one or more visual feedback indicators include one or more of a trajectory indicating a motion path of the corresponding point of the second object, a track indicating a relative speed and a relative direction of the corresponding point of the second object along the trajectory, and a retractable band indicating a difference between a position of the corresponding point of the second object and a position of the point of the first object.

9. A system capable of coordinating sensed gestures associated with real-time movement of a first object in a live video feed and pre-recorded gestures associated with movement of a second object in the video, the system being configured to: applying a matching function to determine a match between points of one of the sensed gestures and corresponding points of the pre-recorded gestures; Based on the matching, determining a play time and outputting at least one frame of the video associated with the second object for the play time; determining an instantaneous speed estimate over a time window based on the play time determined by applying the matching function and a current play time; applying the instantaneous speed estimate to a smoothing factor to generate a playback speed; as well as The current play time is updated based on the play speed.

10. The system according to claim 9, wherein: The point of one of the sensed gestures and the corresponding point of the pre-recorded gesture include a single point of the first object and the second object in a first mode and multiple points of the first object and the second object in a second mode, wherein a user control is provided to select between the first mode and the second mode, and for the second mode, the number and position of the multiple points of the first object and the second object are selected.

11. The system according to claim 9, wherein: The first object includes a user receiving the output of the video and performing gestures in sequence to coordinate with the pre-recorded gestures, and the second object includes a coach shown in the video performing the pre-recorded gestures in sequence, wherein the movement includes one of a physical therapy activity, at least a portion of an exercise, at least a portion of a dance, or at least a portion of a performance.

12. The system according to claim 9, in, The smoothing factor includes a fixed value determined empirically, or a variable that changes based on a degree of motion of the second object over the time window.

13. The system according to claim 9, wherein: The video includes one or more visual feedback indicators associated with the pre-recorded gesture of the second object, wherein the one or more visual feedback indicators include one or more of a trajectory indicating a motion path of the corresponding point of the second object, a track indicating a relative speed and relative direction of the corresponding point of the second object along the trajectory, and a telescopic band indicating a difference between a position of the corresponding point of the second object and a position of the point of the first object.

14. A non-transitory computer readable medium having a storage device storing instructions for coordinating sensed gestures associated with real-time movement of a first object in a live video feed and pre-recorded gestures associated with movement of a second object in the video, the instructions being executable by a processor, the instructions comprising: applying a matching function to determine a match between points of one of the sensed gestures and corresponding points of the pre-recorded gestures; Based on the matching, determining a play time and outputting at least one frame of the video associated with the second object for the play time; determining an instantaneous speed estimate over a time window based on the play time determined by applying the matching function and a current play time; applying the instantaneous speed estimate to a smoothing factor to generate a playback speed; as well as The current play time is updated based on the play speed.

15. The non-transitory computer readable medium of claim 14, wherein: The point of one of the sensed gestures and the corresponding point of the pre-recorded gesture include a single point of the first object and the second object in a first mode and multiple points of the first object and the second object in a second mode, wherein a user control is provided to select between the first mode and the second mode, and for the second mode, the number and position of the multiple points of the first object and the second object are selected.

16. The non-transitory computer readable medium of claim 14, wherein: The first object includes a user receiving the output of the video and performing gestures in sequence to coordinate with the pre-recorded gestures, and the second object includes a coach shown in the video performing the pre-recorded gestures in sequence, wherein the movement includes one of a physical therapy activity, at least a portion of an exercise, at least a portion of a dance, or at least a portion of a performance.

17. The non-transitory computer readable medium of claim 14, in, The smoothing factor includes a fixed value determined empirically, or a variable that changes based on a degree of motion of the second object over the time window.

18. The non-transitory computer readable medium of claim 14, wherein: The video includes one or more visual feedback indicators associated with the pre-recorded gesture of the second object.

19. The non-transitory computer readable medium of claim 18, wherein: The one or more visual feedback indicators include one or more of a trajectory indicating a motion path of the corresponding point of the second object, a track indicating a relative speed and a relative direction of the corresponding point of the second object along the trajectory, and a retractable band indicating a difference between a position of the corresponding point of the second object and a position of the point of the first object.

Images