Method and system for training a user to perform an activity

Abstract

The present disclosure relates to a method and system for training a user to perform a physical activity. The method includes capturing a real-time video of a user performing an activity based on an activity option selected by the user, extracting an AI model based on the activity option, processing the real-time video of the user by the AI ​​model in real time to determine a set of user performance parameters based on the user's current activity performance, overlaying the user in the real-time video with a pose skeleton model, comparing the set of user performance parameters with a set of target activity performance parameters, generating feedback for the user based on a comparison of the set of user performance parameters with the set of target activity performance parameters, and rendering the feedback on a rendering device.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 63 / 074539, filed September 4, 2020, which is incorporated herein by reference in its entirety. [Technical Field]

[0002] This disclosure generally relates to: activity Regarding training, more specifically, activity The present invention relates to a method and system for training a user to perform a task. [Background technology]

[0003] In an era of rapid urbanization and fast-paced living, people in many places find it difficult to carve out daily time for physical well-being and find a proper work-life balance. Moreover, during a pandemic, when lockdowns are imposed, gyms and parking lots are usually closed to public access. In such cases, it is much more convenient for people to train at home.

[0004] Many exercise machines and training methods are monitored and configured with adjustable parameter settings based on the user's desired abilities, goals, and specific training method. However, for best results and to reduce the possibility of muscle damage, many exercises require precise control of complex movements by the user during an exercise routine. performance , and requires skilled adjustment of weight or force resistance.

[0005] A drawback associated with conventional workout machines is that the setting parameters used may vary and change over time during a workout session. Another drawback is that the exercise machine does not remember previously entered customized settings or preferences. Another problem is that workout history is not saved. Also, conventional exercise machines may include a set of pre-configured programs that may not be appropriate for all users. Another problem associated with conventional exercise machines is that the pre-configured programs do not consider other parameters such as posture, body movement, and other parts.

[0006] Furthermore, at most, gyms typically have a set of mirrors that allow people to view, confirm, or adjust their posture and movement to account for proper posture and movement. However, unless a person has a posture and movement analyzer, they may perform with improper posture and movement, potentially resulting in injury. Furthermore, exercise trainers cannot be present during every exercise. Also, trainers cannot simultaneously monitor and guide all exercisers during a group exercise. Furthermore, in most cases, trainers are not available to motivate / encourage exercisers.

[0007] Additionally, sensors are known to record various information about the human body. For example, electromyography (EMG) electrodes can measure the electrical activity generated by a person's muscles. Similarly, there are motion sensors that record a person's movements / movements. Thus, in the context of individual training, particularly in the context of self-training or individual training or remote training, current technology does not enable a coaching / training entity to monitor an individual's physiological state during coaching / training and / or to efficiently manage an exercise regimen for an individual in an individualized, real-time manner. An individual may: performanceSince people may have personalized needs for improvement, it is desirable for the system to automatically adjust metrics and instructions by taking into account physiological conditions.

[0008] On the other hand, monitoring and evaluating exercise / fitness, matching exercise sequences, counting sequences, and tracking real-time progress of exercises through the presence of an instructor can be time-consuming, and the reliability of the results can be low depending on the instructor's subjective evaluation criteria. Therefore, it is beneficial to use video displays, artificial intelligence (AI), and augmented reality (AR) technologies to solve these problems.

[0009] Therefore, an interactive exercise machine that uses sensors to track a user's posture and body movements and further provides an interactive rendering device to display and manage exercise is highly desirable from a health and fitness perspective for many users. In addition to fitness, interactive rendering devices are useful in a variety of other scenarios, such as rehab, physical therapy, yoga, dance, theater, and other situations where feedback regarding calmness and observation is important. activity It may also be desirable in Summary of the Invention

[0010] In one embodiment, the body activity In one example, the method includes, via a graphical user interface (GUI) of a rendering device, displaying a plurality of activity types. Choices This includes rendering multiple activity type Choices Each of the activity The method includes: activity Selected by the user from a selection activityThe method further includes receiving a selection, the user input including at least one of a gesture, a touch, or an audio command. activity Based on the choice activity Each of the at least one camera captures real-time video of the user performing the action. capture The real-time video captures the postures and movements made by the user to perform the activity. flow The method includes: activity The AI ​​model further includes extracting an AI model based on the selection. activity Expert Goal Activity Performance Based on activity Responding to choices activity The method is configured to determine deviations of the user from a plurality of correct movements associated with the user. At that time Activities performance Based on user performance The method further includes processing a real-time video of the user with the AI ​​model in real time to determine the set of parameters. posture The posture skeleton model is superimposed on the skeleton model. activity Each of the plurality of key points is based on a corresponding key point of the user in the real-time video. joint The method involves generating a set of user performance parameters using an AI model. the goal Further comprising comparing with a set of activity performance parameters. Goal Activity The set of performance parameters is activity The method uses an AI model to generate a set of user performance parameters and the goalThe method further includes generating feedback for the user based on a comparison with the set of activity performance parameters. The feedback includes at least one of a corrective action or a warning. The feedback includes at least one of visual feedback, auditory feedback, or haptic feedback. The method further includes rendering the feedback on a rendering device by the AI ​​model. Rendering the feedback includes overlaying one of the at least one corrective action on a postural skeletal model overlaid on the real-time video of the user. Rendering the feedback includes overlaying one of the at least one corrective action on a GUI of the rendering device. caveat Rendering the feedback further includes outputting auditory feedback to the user via a speaker.

[0011] In one embodiment, the body activity A rendering device for training a user to perform the steps of the method is disclosed. In one example, the rendering device includes a display device. The display device displays a plurality of activity type Choices It includes a GUI configured to render multiple activity type Choices Each of the activity The GUI responds to user input by activity Selected by the user from a selection activity The rendering device is further configured to receive a selection, the user input comprising at least one of a gesture, a touch, or an audio command. activity Based on the choice activity Real-time video of the user running capture and at least one camera configured to capture real-time video of the user from an associated predetermined angle. capture The real-time video captures the postures and movements made by the user to perform the activity. flow The rendering device further includes a processor and a memory communicatively coupled to the processor, the memory storing processor instructions that, when executed by the processor, cause the processor to generate a user-selected image. activity The AI ​​model extracts the answer based on the choice. activity Expert Goal Activity Performance Based on activity Responding to choices activity The processor-executable instructions are further configured, when executed, to cause the processor to determine a deviation of the user from a plurality of correct movements associated with the user. At that time Activities performance Based on user performance The processor-executable instructions further cause the processor, when executed, to process a real-time video of the user with the AI ​​model in real time to determine a set of parameters. posture The posture skeleton model is superimposed on the skeleton model. activity Each of the plurality of key points is based on a corresponding key point of the user in the real-time video. joint The processor-executable instructions further, when executed, cause the processor to implement a set of user performance parameters according to the AI ​​model. the goal The activity is compared against a set of performance parameters. Goal Activity The set of performance parameters is activity The processor-executable instructions, when executed, further inform the processor of the AI ​​model, a set of user performance parameters and the goalThe processor-executable instructions, when executed, further cause the processor to render, via the AI ​​model, the feedback on a rendering device. Rendering the feedback includes overlaying one of the at least one corrective action on a postural skeletal model overlaid on a real-time video of the user. Rendering the feedback includes overlaying one of the at least one corrective action on a postural skeletal model overlaid on a real-time video of the user on a GUI of the rendering device. caveat Rendering the feedback further includes outputting auditory feedback to the user via a speaker.

[0012] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed. [Brief explanation of the drawings]

[0013] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles. [Figure 1] 1A-1E illustrate block diagrams of exemplary rendering devices for training a user to perform a physical activity, according to some embodiments. [Figure 2] FIG. 1 illustrates a functional block diagram of an exemplary system for training a user to perform a physical activity, according to some embodiments. [Figure 3] 3A and 3B show a flow diagram of an exemplary process for training a user to perform a physical activity, according to some embodiments. [Figure 4]1 illustrates a flow diagram of an exemplary process for correcting an initial position of a user, according to some embodiments. [Figure 5] 1 illustrates a flow diagram of an exemplary process for comparing a trainer avatar corresponding to a target activity performance with a user avatar corresponding to a user's current activity performance, according to some embodiments. [Figure 6] 1 illustrates an exemplary graphical user interface (GUI) displaying multiple exercises, according to some embodiments. [Figure 7] 1 illustrates an exemplary GUI displaying a fitness application home page, according to some embodiments. [Figure 8] 1 illustrates an exemplary GUI for displaying exercise parameters, according to some embodiments. [Figure 9] 1 illustrates an exemplary GUI displaying a user's current activity performance and posture skeletal model, according to some embodiments. [Figure 10] 1 illustrates an exemplary GUI displaying a user's current activity performance and posture skeletal model, according to some embodiments. [Figure 11] 1 illustrates an exemplary GUI displaying a user's current activity performance and posture skeletal model, according to some embodiments. [Figure 12] FIG. 1 is a block diagram of an exemplary computer system for implementing embodiments consistent with the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014]

[0013] Exemplary embodiments are described with reference to the accompanying drawings. For convenience, the same reference numerals are used throughout the drawings to refer to the same or similar parts. While examples and features of the disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. It is intended that the following detailed description be considered as exemplary only, with a true scope and spirit being indicated by the following claims.

[0015] 1A-E, in accordance with some embodiments, activity An exemplary rendering device 100 is shown for training a user 102 to perform a virtual reality game. By way of example, the rendering device may include, but is not limited to, a smart television (TV), a desktop, a laptop, a personal computer, a smartphone, a smart mirror, a voice assistant, or any computing device. In an exemplary scenario, a user 102 may use the rendering device 100 in a room 104. The room 104 may be part of, for example, a gymnasium, a physical therapy facility, a rehab facility, a yoga studio, a dance studio, a dojo, a martial arts center, a theater coaching center, etc. The gymnasium may accommodate multiple users. activity The rendering device 100 may include a plurality of smart mirrors configured to analyze performance. In yet another embodiment, the rendering device 100 may be used in an outdoor environment (e.g., a park). In some embodiments, the rendering device 100 may be used to monitor the progress of a patient undergoing rehab or physical therapy.

[0016] The rendering device 100 may include a display device 106, at least one camera 108, at least one external camera 110, one or more processors (not shown), and a storage unit (not shown) communicatively coupled to the one or more processors. The display device 106 may include a graphical user interface (GUI). It should be noted that the at least one camera 108 may be located in the center, along an edge, or at the bottom of the smart mirror 100. The rendering device 100 may be configured to display images via the at least one camera 108 and the at least one external camera 110. capture Using real-time video captured by the activity Users can be trained to perform

[0017] As will be described in more detail in connection with FIGS. 2-11, rendering device 100 renders device Multiple activity types via a graphical user interface (GUI) Choices to the user. activity type Choices Each of the activity The rendering device 100 may generate a plurality of activity Selected by the user from a selection activity The rendering device 100 further receives a selection, the user input including at least one of a gesture, a touch, or an audio command. The rendering device 100 further receives a selection by the at least one camera. activity Based on the choice activity Each of the at least one camera captures real-time video of the user performing the action. capture The real-time video captures the postures and movements made by the user to perform the activity. flow The rendering device 100 includes a activity Based on the choices, the AI ​​model is further extracted. activity Expert Goal Activity Performance Based on activity Responding to choices activity The rendering device 100 is further configured to process the real-time video of the user in real time with the AI ​​model to determine deviations of the user from a plurality of correct movements associated with the user. At that time Activities performance Based on user performance The rendering device 100 further determines a set of parameters for the user in the real-time video by the AI ​​model. posture Overlay with a skeleton model. The posture skeleton model is activity Each of the plurality of key points is based on a corresponding key point of the user in the real-time video. joint The rendering device 100 further calculates a set of user performance parameters using an AI model. the goal The activity performance is compared to a set of parameters. Goal Activity The set of performance parameters is activity The rendering device 100 further comprises an AI model that determines a set of user performance parameters and the goal and generating feedback for the user based on the comparison to the set of activity performance parameters, the feedback including at least one of a corrective action or a warning, the feedback including at least one of visual feedback, auditory feedback, or haptic feedback. device 100 rendered by AI model device Rendering feedback includes overlaying one of the at least one corrective action on a posture skeletal model overlaid on a real-time video of the user. Rendering feedback includes overlaying one of the at least one corrective action on a GUI of the rendering device. caveatRendering the feedback further includes outputting auditory feedback to the user via a speaker.

[0018] The memory may include an AI model that, when executed by one or more processors, may provide the one or more processors with a physical response to the user 102 in real time, in accordance with aspects of the present disclosure. activity The memory may also store instructions that train the rendering device 100 to execute capture Various data that may be processed and / or requested (e.g., real-time video AI model data, multiple Activity Type , multiple activity ,Real-time video ,set of user performance parameters, Goal Activity performance data, etc.

[0019] The rendering device 100 can interact with the user 102 via a GUI accessible via the display device 106. By way of example, the display 106 can be a liquid crystal display (LCD), a light-emitting diode (LED)-backlit LCD, a thin-film transistor (TFT) LCD, an LED display, an organic LED (OLED) display, an active-matrix organic LED (AMOLED) display, a plasma display panel (PDP) display, a quantum dot LED (QLED) display, or the like. The rendering device 100 can also include one or more external devices (not shown). In some embodiments, the rendering device 100 can interact with one or more external devices via a communication network (e.g., a universal serial bus (USB) data cable, a high-definition multimedia interface (HDMI) cable, wireless fidelity (Wi-Fi), optical fidelity (Li-Fi), Bluetooth, and other similar communication networks) to transmit or receive various data. The external device can include, but is not limited to, a remote server, a digital device, or another computing system.

[0020] The GUI provides multiple Activity Type Render multiple Activity Type Each of the activity The user 102 may Activity Type from Activity Type Select and via user command Activity Type Multiple associated with activity from activity Furthermore, the rendering device 100 starts the activity. activity Work with the experts Goal Activity Performance 112 is displayed on the screen. Goal Activity Performance 112 is activity Expert video recording, activity Expert three-dimensional (3D) models, two-dimensional (2D) models, or activity The user 102 may be an expert in four dimensions (4D). At that time Activities performance in the goal activity performance The display device 106 can follow the user 102. At that time of activity Shows real-time video of the performance.

[0021] Additionally, at least one camera 108 and at least one external camera 110 may be used to capture images of the user 102. At that time of activity The at least one external camera 110 enhances the accuracy of determining the user's 102 head posture, movement, gaze, and orientation. Additionally, the at least one camera 108 may be used for facial recognition of the user 102. The facial data corresponding to the user 102 is associated with a user profile. The user profile is stored in a database and includes history, custom settings, activity Current and historical user data may be associated with the user, such as, but not limited to, messages from experts, profile data, and other similar data.

[0022] Also, rendering device 100 was selected by the user 102 activity Based on the choices, an AI model is extracted. activity Expert Goal Activity Performance Based on 112, activity Responding to choices activity Further, the AI ​​model is configured to receive the real-time video and process the real-time video to determine deviations of the user 102 from a plurality of correct movements associated with the user 102. At that time Activities performance Based on user performance In one embodiment, the rendering device 100 is configured to automatically adjust a set of parameters based on an estimated future orientation of the head of the user 102. For example, when the user 102 is in a reclining position, At that time of activity When executing, the rendering device 100 At that time of activity It can be rotated approximately 90 degrees to provide impromptu tracking of performance.

[0023] Additionally, the AI ​​model identifies users 102 in real-time video. posture The AI ​​model may be an AI predictive model. In some configurations, the posture skeleton model 114 may be determined based on the estimated future posture and motion of the user 102. The posture skeleton model 114 may be activity Type and activity Each of the plurality of key points corresponds to a joint of the user 102. Additionally, the plurality of key points can be connected with lines representing the bones of the user 102 to complete the posture skeleton model 114.

[0024] The display device 106 of the rendering device 100 displays a postural skeleton model 114 overlaid on the real-time video of the user 102, and a postural skeleton model 114 overlaid on the real-time video of the user 102. activity Expert Goal Activity Performance 112 and At that time of activity a set of performance-related user performance parameters; Goal Activity Related to Performance 112 Goal Activity and a set of parameters. posture The skeletal model 114 is configured to calculate the estimated future distance of the user 102 relative to the rendering device 100 and the estimated future position of the user 102. posture Note that the video is automatically adjusted and normalized to the real-time video of the user 102 based on the user's 102 posture and movement. posture The transparency of the skeletal model 114 may be adjustable by the user 102. In one embodiment, the postural skeletal model 114 is completely transparent and invisible to the user 102. In such an embodiment, the postural skeletal model 114 may be used by the AI ​​model for computational purposes only.

[0025] The AI ​​model uses a set of user performance parameters the goal The AI ​​model compares the activity performance parameters with the set of user performance parameters. the goal The AI ​​model generates feedback for the user 102 based on the comparison with the set of activity performance parameters. The feedback includes at least one of a corrective action or a warning. The feedback may be at least one of visual feedback, auditory feedback, or haptic feedback. Further, the AI ​​model renders the feedback. The rendering may include overlaying one of the at least one corrective action on the posture skeletal model 114 overlaid on the real-time video of the user 102. Further, the rendering may include overlaying one of the at least one corrective action on a GUI of the rendering device 100. caveatAdditionally, the rendering may include outputting auditory feedback to the user 102 via a speaker configured in the rendering device 100. The feedback may include displaying the user's 102's At that time Instructions for correcting the posture of the user 102 At that time instructions for correcting the user's movements relative to the user's posture, and At that time The method may include generating an alert to the user 102 that includes instructions for correcting the position of the vehicle.

[0026] In FIG. 1A, a user 102 is presented with a target Activity Performance In the exemplary scenario, user 102 At that time of Activity Performance , and the associated posture skeleton model 114 activity Expert Goal Activity Performance Follow 112. activity Expert Goal Activity Performance 112 completely overlaps with the real-time video of the user 102. In such a scenario, the rendering device 100 At that time of Activity Performance but Goal Activity Performance As long as you follow 112, Goal Activity Performance 112. In some embodiments, the rendering device 100 the goal activity performance According to 112 At that time Activities performance A motivational audio message can be output encouraging the user 102 to maintain good health.

[0027] In another exemplary scenario, user 102 may select, as shown on display device 106: the goal activity performance 112 cannot be traced accurately. At that time of Activity Performance , and the associated posture skeleton model 114, activity Expert Goal Activity Performance112. In such a scenario, the rendering device 100 At that time Activities performance but the goal activity performance 112 may generate feedback for the user 102 to ensure compliance. At that time Activities performance The deviation of performance and continues for a predetermined threshold time, the rendering device 100 performance A set of parameters and the goal activity performance The display of 112 can be paused.

[0028] In FIG. 1B, user 102 is viewing a target image as shown on display device 106. Activity Performance 112. In one embodiment, the goal activity performance 112 is displayed in an inset frame at the top left of the display device 106. In the exemplary scenario, At that time of activity Real-time video of the performance116, and the associated postural skeletal model114 activity Expert Goal Activity Follow performance 112. activity Expert Goal Activity Performance 112 completely overlaps with the real-time video 116 of the user 102. In such a scenario, the rendering device 100 At that time of Activity Performance but Goal Activity Performance As long as you follow 112, Goal Activity Performance 112. In some embodiments, the rendering device 100 the goal activity performance According to 112 At that time Activities performance A motivational audio message can be output encouraging the user 102 to maintain good health.

[0029] In another exemplary scenario, user 102 may select, as shown on display device 106: the goal activity performance 112 cannot be traced accurately. At that time of Activity Performance The real-time video 116, and associated pose skeleton model 114, activity Expert Goal Activity Performance 112. In such a scenario, the rendering device 100 At that time Activities performance but the goal activity performance 112 may generate feedback for the user 102 to ensure compliance. At that time Activities performance The deviation of performance and continues for a predetermined threshold time, the rendering device 100 performance A set of parameters and the goal activity performance The display of 112 can be paused.

[0030] In FIG. 1C, a front view of the rendering device 100 is shown on the display device 106, with the user 102 aiming activity It is shown when following the performance 112. At that time of activity Real-time video corresponding to the performance is captured in real time by at least one camera 108. capture Additionally, the AI ​​model of the rendering device 100 may process the real-time video and generate the posture skeleton model 114. In an exemplary scenario, At that time of Activity Performance teeth Goal Activity Performance 112, the rendering device 100 At that time of Activity Performance but Goal Activity Performance As long as you follow 112, Goal Activity Performance 112. In some embodiments, the rendering device 100 the goalactivity performance According to 112 At that time Activities performance A motivational audio message can be output encouraging the user 102 to maintain good health.

[0031] In another exemplary scenario, user 102 may select, as shown on display device 106: the goal activity performance 112 cannot be traced accurately. At that time of Activity Performance However, the specified performance Above-threshold goals Activity Performance 112, rendering device 100 At that time of Activity Performance The goal is Activity Performance Feedback (video, graphical, auditory, or tactile) can be generated to the user 102 to ensure compliance with 112. At that time Activities performance The deviation of performance and continues for a predetermined threshold time, the rendering device 100 performance A set of parameters and the goal activity performance The display of 112 can be paused.

[0032] In FIG. 1D, the rendering device 100 simultaneously activity The rendering device 100 trains each of the users 102 and 118 to perform the following: Activity Performance Note that the rendering device 100 may be shared by a family member at home, a gym member, a patient undergoing rehab treatment and / or physical therapy at a hospital, etc. At that time of activity Performance is user 118 At that time of activityThe rendering device 100 is selected by each of the users 102 and 118. activity Targets corresponding to activity View performance and rendering device The rendering device 100 can display a postural skeletal model superimposed on real-time video of the user 102 and the user 118. In some embodiments, the rendering device 100 can display a target skeletal model overlaid on the real-time video of each of the user 102 and the user 118. activity In some embodiments, rendering device 100 may display the performance in an inserted frame on display device 106. Goal Activity Alternatively, rendering device 100 may include multiple displays or screens, one for each of multiple users (e.g., user 102 and user 118). A single display or screen in such rendering device 100 may be used by one user at a time.

[0033] In FIG. 1E, rendering device 100 is activity The performance is analyzed and the smart mirror 120 determines the activity The performance is analyzed. The smart mirror 120 includes a camera and a GUI. The functionality of the smart mirror 120 is similar to that of the rendering device 100. device 100 displays real-time video corresponding to user 102, and smart mirror 120 displays a reflection or real-time video corresponding to user 118. Smart mirror 120 may be communicatively coupled to rendering device 100. In one embodiment, a combination of multiple devices (such as rendering device 100 and smart mirror 120) may be used in a public environment (e.g., a gymnasium or park) for multiple users.

[0034] Referring now to FIG. 2, a real-time body activity2 shows a functional block diagram of an exemplary system 200 for training a user 202 to perform the following: The system 200 includes a rendering device 204. In some embodiments, the rendering device 204 of the system 200 device 204 rendered device 100. The rendering device 204 includes a display device 206, a camera 208, a microphone 210, a speaker 212, a processor 214, and a memory 216. The memory 216 includes a GUI module 218, an AI model 220, and a database 222. One or more wearable sensors 224 may be worn by the user 202. By way of example, the one or more wearable sensors 224 may include an electrocardiogram (ECG) sensor, an electroencephalogram (EEG) sensor, an electromyogram (EMG) sensor, a pulse oximeter, etc. Each of the one or more wearable sensors 224 is communicatively coupled to the rendering device 204 via a communications network. Additionally, the rendering device 204 may include one or more built-in sensors (e.g., a proximity sensor, an audio sensor, a light detection and ranging (LIDAR) sensor, an infrared (IR) sensor, and other motion-based sensors) to receive additional data that can be processed and analyzed for the user 202.

[0035] Additionally, the display device 206 is configured to display real-time video of the user 202. The GUI module 218 is accessible to the user 202 via the display device 206. The GUI module 218 includes a plurality of Activity Type is provided to the user 202. As an example, Activity Type may include, but are not limited to, physical exercise, guided meditation, yoga, physical therapy, flower arranging, origami, dance, theatre, performing arts, martial arts, speech therapy, rehabilitation, drawing, painting, any form of physical therapy and rehabilitation, CrossFit, Les Mills, F45, Zamba, Bikram Yoga, Orange Theory, etc. Activity Type Each of the activity The user 202 can, via user command, Activity Typefrom Activity Type and, Activity Type Multiple associated with activity from activity and (iii) the user command may be at least one of a voice command (received via microphone 210), a touch gesture, an air gesture, an eye gesture, or a signal generated by an input device (e.g., a mouse, a touchpad, a stylus, a keyboard, an associated connected device or controller (e.g., a game controller), etc.). Rendering device 204 may include multiple displays and multiple cameras to handle multiple users simultaneously.

[0036] Furthermore, the camera 208 activity Type and activity of user 202 corresponding to At that time of Activity Performance The real-time video is captured in real time. In some embodiments, the rendering device 204 may include one or more additional cameras (such as at least one external camera 110). The real-time video received from the camera 208 is stored in a database 222. In some embodiments, a user may edit the real-time video based on one or more user commands. The user commands may be at least one of a text command, a voice command, a touch command, or a display gesture. The one or more user commands include at least one of setting a start point for the real-time video, setting an end point for the real-time video, removing a background from the real-time video, assigning one or more tags to the real-time video, and sharing the real-time video with a set of other users.

[0037] Additionally, the rendering device 204 may be selected by the user 202. activity Based on the selection, an AI model 220 can be extracted. activity Expert Goal Activity Performance Based on activityResponding to choices activity Further, the AI ​​model 220 is configured to receive real-time video from the camera 208 via the processor 214 and process the real-time video to determine deviations of the user 202 from a plurality of correct movements associated with the user 202. At that time Activities performance Based on user performance A set of parameters is determined.

[0038] Additionally, the AI ​​model 220 may identify the user 202 in the real-time video. posture Skeletal models (e.g., posture Superimpose it onto the skeleton model 114). posture The skeleton model is activity Type and activity Each of the plurality of key points corresponds to a joint or feature of the user 202 in the real-time video. In addition, the plurality of key points posture To complete the skeletal model, it may be connected with lines representing the bones of the user 202. In one embodiment, a 3D rendering of the user 202 may be generated as a pose skeletal model. posture The skeletal model is the current state of the user 202 relative to the rendering device 204. posture and estimated future distance and viewing position, current posture and the estimated future field of view, and the current posture and estimated future posture Note that the time domain is automatically adjusted and normalized to the real-time video of the user 202 based on the time domain and the motion. posture The transparency of the skeletal model 114 may be adjustable by the user 202. In one embodiment, the postural skeletal model is completely transparent and invisible to the user 202. In such an embodiment, the postural skeletal model may be used by the AI ​​model 220 for computational purposes only.

[0039] The AI ​​model 220 receives a set of user performance parameters. the goalThe activity performance parameters are compared with a set of user performance parameters. the goal Based on a comparison with a set of activity performance parameters, posture and generated via AI deviation processing. The feedback includes at least one of a corrective action or a warning. The feedback may be at least one of visual feedback, auditory feedback, or haptic feedback. Further, the AI ​​model 220 renders the feedback. The rendering may include overlaying one of the at least one corrective action on a real-time video of the user 202 on the rendering device 204. Further, the rendering may include overlaying one of the at least one corrective action on a GUI of the rendering device 204. caveat Further, the rendering may include outputting auditory feedback to the user 202 via the speaker 212. In one embodiment, the speaker 212 may be a directional speaker that provides a more personalized training experience for the user 202. In some embodiments, the system 200 includes multiple speakers located in different areas of a room (e.g., a home theater system). In some embodiments, the rendering device 204 is configured to output audio feedback via a Bluetooth headset or speaker. The feedback may be provided to the user 202 when the user 202 is at least partially outside the field of view of the camera 208. At that time Instructions for correcting the posture of the user 202 At that time and instructions for correcting user movements related to the user's posture. At that time In some embodiments, the AI ​​model 220 may generate a warning to the user 202, including instructions to correct the position of the body. activityIn one embodiment, the AI ​​model 220 of the rendering device 204 may include a plurality of sub-modules that function in combination to perform the above steps to train the user to perform the following: activity By way of example, the performance data may include a recommendation engine for providing recommendations for exercises performed, circuits performed, duration of exercises, activity This may include performance, personal goals, user profile, age, weight, body mass index (BMI), etc.

[0040] The rendering device 204 At that time of Activity Performance but Goal Activity Performance As long as you follow Goal Activity Performance However, the user 202 may continue to display At that time of Activity Performance , and related posture The skeletal model activity Expert Goal Activity Performance When indicating a deviation from At that time of Activity Performance but Goal Activity Performance The system may generate feedback for the user 202 to ensure that the user is following the instructions. At that time Activities performance The deviation of performance and continues for a predetermined threshold time, the rendering device 204 performance A set of parameters and the goal activity performance The display of the can be paused.

[0041] As can be seen, the execution of the user 102 activityThe feedback based on the activity may not be limited to instructions to perform corrective actions. The feedback may also include biometric feedback or alerts, such as any irregularities or problems in one or more of the user's 102 pulse rate or heart rate, the user's 102 body temperature, muscle spasms, pupil dilation, and other similar health issues. In some embodiments, the feedback may be in the form of motivation or encouragement provided to the user 102 while performing an activity or after completing an activity. As an example, in the form of audio feedback, messages such as "Great job," "Amazing," "Great action," "Perfectly done," "Done like a pro," "You're the best," "The best I've seen," and other similar messages may be provided to the user 102. Sounds of applause, cheers, or various exclamations may also be provided to the user 102 as feedback. These messages may also be provided in the form of visual feedback, such that the messages may be displayed in text format on the GUI of the rendering device 100. Additionally or alternatively, graphical elements, such as popping crackers, flying balloons, the sound of a stadium crowd, or avatars of chair leaders, instructors, celebrities (e.g., Kai Greene, Phil Health, Ronnie Coleman, Arnold, and other well-known personalities), may also be displayed to the user 102. In some configurations, gamification and reward mechanisms for activities performed by the user may also be used as feedback provided to the user. As a result of such feedback, the user 102 may be constantly motivated and motivated to complete any given task in a silo. activity Sometimes I don't feel like I'm doing my job.

[0042] In some configurations, the user 102 also activity In such a case, the feedback may include a status regarding the percentage of the goal that has been achieved by the user 102.

[0043] In some embodiments, to provide feedback to the user 102 on their own personal smartphone device, i.e., a third-party smartphone device, the rendering device 100 may be configured with an open application programming interface (API), which may seamlessly enable such integration. Furthermore, data received from the third-party smart device may also be captured in the rendering device 100 via the open API and further provided to the user 102 via the rendering device 100 using visual elements (such as graphs or charts), verbal and audio cues, or haptic cues. The data may also correspond to alerts and warnings generated by the third-party smart device. As an example, a smartwatch configured to sense the user 102's blood pressure can transmit data regarding the user 102 having high blood pressure to the rendering device 100. Thus, the rendering device 100 can verbally or visually render the message "Your blood pressure is too high, relax and break" to the user 102. Thus, the rendering device 100 can act as a feedback collator and a single-point smart device for viewing all feedback. In other words, as the smart mirror 100 generates feedback itself and also receives feedback from other smart devices, the rendering device 100 assimilates all the feedback, refines it, and presents it to the user 102 via the rendering device 100. Thus, the user does not need to rely on multiple devices to receive different types of feedback.

[0044] Additionally, the user 102 may communicate with other remote users on various social networks or who may also use the smart mirror 100. activityYou may wish to share your performance with friends. To this end, the rendering device 100 may be configured with various integrations with social media applications. Examples of these social media applications may include, but are not limited to, FACEBOOK®, WHATSAPP®, YOUTUBE®, and / or INSTAGRAM®. In some embodiments, the smart mirror 100 may have these social media applications already installed. Also, the rendering device 100 may have the social media applications already installed. device 100 specific and other rendering device There may be a social media application configured to connect only users of the 100 and / or smart mirrors.

[0045] Thus, through integration with these social media applications, user performance may be posted and published to one or more of these social media platforms and made available as online content for other users to access. The aforementioned reward mechanisms may also be used for sharing media on the social media platforms. In some configurations, users activity Scores associated with the exercises performed may be presented on a leaderboard as points for various users using the smart mirror 100 and / or display device 200. Badges may also be assigned to various users based on the level of activity performed by them and may be displayed on social media platforms. Additionally, records associated with the exercises performed may also be displayed. activity The goals set by various users for the game and the percentage completion of each of the goals may also be displayed on the social media platform. As can be appreciated, the feedback provided to the user may also be shared within the user's groups on social media, including friends, social circles, and classes that may be connected in real time.

[0046] In some embodiments, the rendering device 204 can generate audio messages for the user 202 in audio form via the speaker 212 (e.g., number of reps in audio form, auditory feedback to the user 202, new achievements, personal bests, messages from other users, advertisements, challenges, errors, warnings, etc.). Note that when generating audio messages, the timing and duration of the audio output can be important. For example, when the user 202 is exercising at a high speed, some of the audio messages may become stale before they are generated. Furthermore, some of the audio messages may become repetitive and unnatural. Furthermore, some of the audio messages may be of higher priority (e.g., warnings and errors). In such scenarios, audio messages may be generated via a priority queue-based mechanism. In one embodiment, an AI-based approach can be used for audio messages to generate more natural interactions. posture Note that the exercise matching and keypoint recognition can be performed on a remote server, and the keypoint recognition can be performed on an edge node. Therefore, the transfer of heavy video data to the server can be avoided. posture and exercise matching is unknown on the end edge device, which may enhance overall security.

[0047] It should be noted that all such modules 206-224 may be represented as a single module or a combination of different modules. Furthermore, as will be appreciated by those skilled in the art, each of the modules 206-224 may reside, in whole or in part, on one device or multiple devices that communicate with each other. In some embodiments, each of the modules 206-224 may be implemented as dedicated hardware circuitry comprising custom application-specific integrated circuits (ASICs) or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. Each of the modules 206-224 may also be implemented in programmable hardware devices such as field programmable gate arrays (FPGAs), programmable array logic, programmable logic devices, etc. Alternatively, each of the modules 206-224 may be implemented in software for execution by various types of processors (e.g., processor 214). An identified module of executable code may comprise, for example, one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, function, or other construct. Nevertheless, the identified modules or component executables need not be physically located together, but may comprise different instructions stored in different locations that, when logically coupled together, comprise a module and achieve the module's specified purpose. Indeed, a module of executable code can be a single instruction, or many instructions, and can even be distributed across several different code segments, among different applications, and across the internet, the cloud, and several parallel memory devices.

[0048] As will be appreciated by those skilled in the art, activity Various processes may be used to train a user to perform the steps described herein. For example, the exemplary system 200 and associated rendering device 204 may perform the steps described herein to train a user to perform the steps described herein. activityIn particular, as will be appreciated by those skilled in the art, control logic and / or automated routines for carrying out the techniques and processes described herein may be implemented by system 200 and associated rendering device 204 either through hardware, software (such as neural networks or other computational models), or a combination of hardware and software. For example, suitable code may be accessed and executed by one or more processors on rendering device 204 to perform some or all of the techniques described herein. Similarly, application specific integrated circuits (ASICs) configured to perform some or all of the processes described herein may be included in one or more processors on rendering device 204.

[0049] 3A and 3B, in accordance with some embodiments, activity An exemplary process 300 for training a user to perform a plurality of tasks is illustrated via a flowchart. In one embodiment, the process 300 may be implemented by a rendering device 100 in a room 104. The process 300 begins at step 302 by prompting a user, via a graphical user interface (GUI) of the rendering device, to perform a plurality of tasks. activity type Choices This includes rendering multiple activity type Choices Each of the activity , and in one embodiment, the GUI may be rendered on the display device 206 via the GUI module 218.

[0050] Additionally, the process 300 may, at step 304, generate a plurality of activity Selected by the user from a selection activityThe user input includes at least one of a gesture, a touch, an audio command, or a signal generated by an input device (such as a keyboard, a mouse, a stylus, a graphic pen, etc.). In one embodiment, the audio command may be received by the microphone 210.

[0051] Further, the process 300 may, in step 306, select a camera image by at least one camera (e.g., at least one camera 108). activity Based on the choice activity Each of the at least one camera captures real-time video of the user performing the action. capture The real-time video captures the postures and movements made by the user to perform the activity. flow In some embodiments, a user can edit the real-time video based on one or more user commands. The user commands comprise at least one of a text command, a voice command, a touch command, or a display gesture. As an example, the one or more user commands can include setting a start point for the real-time video, setting an end point for the real-time video, removing a background from the real-time video, assigning one or more tags to the real-time video, and displaying the real-time video. activity Sharing with a set of experts or other users; or activity Expertise should be shared with users Goal Activity Performance It can record and edit real-time video corresponding to the

[0052] Furthermore, the process 300 continues at step 308 with the user-selected activity and extracting an AI model (e.g., AI model 220) based on the selection. activity Expert Goal Activity Performance Based on activity Responding to choices activity Further, the process 300 is configured to determine deviations of the user from a plurality of correct movements associated with the user's At that time Activities performance Based on user performance It involves processing a user's real-time video in real time with an AI model to determine a set of parameters.

[0053] Further, the process 300 may, in step 312, use the AI ​​model to identify the user in the real-time video. posture Skeletal models (e.g., posture This involves superimposing the posture skeleton model on the skeletal model114. activity Each of the plurality of key points is based on a corresponding key point of the user in the real-time video. joint In one embodiment, the AI ​​model 220 is overlaid on top of the user 202 At that time of activity In some embodiments, the AI ​​model 220 is configured to generate a postural skeletal model based on real-time video corresponding to the performance. In such embodiments, the AI ​​model 220 is configured to identify various joints or features of the user 202 in real time and assign importance points to each of the joints or features. In some embodiments, the AI ​​model 220 is configured to estimate the future distance of the user 202 relative to the rendering device 204, as well as the user's future posture and movement. Further, step 312 of the process 300 includes, in step 314, determining the user's position relative to the rendering device. At that time pose and estimated future distance and the user's At that time The method includes automatically adjusting and normalizing the posture skeleton model based on the current posture and estimated future posture and motion.

[0054] The process 300 further comprises, in step 316, determining a set of user performance parameters using the AI ​​model. the goal This includes comparing the activity performance to a set of parameters. Goal ActivityThe set of performance parameters is activity For example, the user performance The set of parameters is At that time of Activity Performance This information may include, but is not limited to, the speed of the user's workout, the number of repetitions completed, the overall completion of the workout, third-party smart device information, the user's pulse rate, the user's blood pressure, and the user's movement. Activity Performance The set of parameters is the goal Activity Performance These include, but are not limited to, the user's target speed, target number of repetitions, the user's target pulse rate, and the user's target effort.

[0055] The process 300 further comprises, in step 318, determining a set of user performance parameters and the goal generating feedback for the user based on a comparison to the set of activity performance parameters, the feedback including at least one of a corrective action or a warning, the feedback including at least one of visual feedback, auditory feedback, or tactile feedback, the feedback being a feedback of the user At that time The user's posture correction instructions and At that time and instructions for modifying the user's movement associated with the user's posture when the user is at least partially outside the field of view of at least one camera. At that time and instructions to correct the position of the

[0056] Process 300 further includes rendering feedback to a rendering device by the AI ​​model at step 320. Process 300 further includes overlaying one of the at least one corrective action on the postural skeletal model overlaid on the real-time video of the user at step 322. Process 300 further includes displaying a warning on a GUI of the rendering device at step 324. Process 300 further includes outputting auditory feedback to the user via a speaker at step 326.

[0057] In some embodiments, the process 300 At that time of Activity Performance is compared based on a predefined threshold for a predefined threshold time. performance Goals that exceed Activity Performance A set of users performance Parameters and Goals Activity Performance Further, in such an embodiment, the process 300 may include, by the AI ​​model, pausing the display of the set of user performance parameters and the goal generating feedback for the user based on a comparison of the activity performance parameters to a set of predefined thresholds; performance Note that the time is correlated with a predefined threshold time. For example, if a user the goal If there is a misalignment from the activity performance, the smart mirror 100 may still pause the display. In some embodiments, the AI ​​model may set a predefined threshold based on the user's skill level. performance and a predefined threshold time. In some embodiments, the AI ​​model dynamically determines values ​​for each of the performance Various parameters other than time can be used.

[0058] The process 300 further comprises, at step 328, providing a user interface through the rendering device's GUI. performance A set of parameters, Goal Activity Performance a set of parameters, and activity Expert Goal Activity Performance In one embodiment, the real-time video includes displaying the user's At that time of Activity Performance In response to Goal Activity Performance corresponds to activity In such an embodiment, the user performance A set of parameters and activity Expert Goals Activity Performance is displayed in real time via the GUI via the rendering device. the goal activity performance is the user's At that time Activities performance The process 300 further comprises, in step 330, overlaying the user's image in the real-time video. At that time of activity performance the goal Includes real-time overlay of activity performance.

[0059] 4, an exemplary process 400 for correcting a user's initial position is illustrated via a flowchart, according to some embodiments. In one embodiment, the process 400 is implemented by the rendering device 100. The process 400 includes, at step 402, detecting the user's initial position via at least one camera (e.g., at least one camera 108). Further, the process 400 includes, at step 404, detecting the detected user's initial position relative to at least one activity Further, in step 406, the process 400 includes prompting the user to correct the initial position if the detected initial position does not match the initial position. Steps 402-406 are performed by the user. At that time Note that this may be performed iteratively throughout the activity performance.

[0060] Referring now to FIG. 5, according to some embodiments, the goal activity performance The trainer avatar corresponding to the user's At that time Activities performance An exemplary process 500 for comparing a user avatar with a corresponding user avatar is illustrated via a flowchart. In one embodiment, the process 500 is implemented by the rendering device 100. The process 500 begins in step 502 by capturing real-time video through at least one camera. capture Then, activity Expert Goals activity A trainer avatar that corresponds to the performance and the user's At that time of activity and a user avatar corresponding to the performance. The trainer avatar is activity The user avatar is a multi-dimensional model of the user, while the expert is a 3D model of the user.

[0061] Further, the process 500 continues at step 504 by activity At least one of a trainer avatar and a user avatar corresponding to the performance, activity Type and activity Based on the user's At that time of activity The process 500 further includes, in step 506, comparing the user's performance. performance Set of parameters, goal Activity Performance and at least one of a trainer avatar and a user avatar corresponding to the user's avatar through a GUI via a rendering device. At that time of Activity Performance This includes displaying the At that time Activities performance is in real time, the goal activity performance The user's avatar is overlaid on at least one of the trainer avatar and the user avatar corresponding to the user's avatar. At that time Note that this may be performed iteratively throughout the activity performance.

[0062] 6, an exemplary GUI 600 is shown displaying a plurality of exercises 602, according to some embodiments. According to one embodiment, the plurality of exercises 602 may include, but are not limited to, side squats, lunges, squats, burpees, push-ups, front triceps, front push-ups, dumbbell squat presses, front squats, and front lunges. Each of the plurality of exercises 602 may include a plurality of Activity Type Selected from 604 Activity Type As an example, the filtering may be based on, but is not limited to, a plurality of Activity Type 604 may include, but is not limited to, "arms," ​​"chest," "lunges," "legs," "quads," "shoulders," "squats," and "triceps." The plurality of exercises 602 may be sorted based on one of sorting criteria 606. By way of example, sorting criteria 606 may include, but is not limited to, new exercises, most recent exercises performed, most frequently performed exercises, and duration of exercises. Note that GUI 600 is not limited to fitness and may be customized based on user requirements and use cases. For example, GUI 600 may be configured to display exercises for a specific treatment in the case of rehab, meditation for yoga, physical therapy recommended by a medical professional, etc. activity may include:

[0063] Referring now to FIG. 7 , an exemplary GUI 700 displaying a home page of a fitness application is shown, according to some embodiments. The GUI 700 can include a menu 702, customization 704, and multiple languages ​​706. By way of example, the menu 702 can include options for the user, such as, but not limited to, “Exercise,” “Circuit,” “Dashboard,” “Goals,” “Connect,” and “Calendar.” Additionally, the customization 704 can provide the user with a theme color option. By way of example, the theme colors can be, but are not limited to, blue steel, dark steel, carbon, charcoal, pastel lady, pastel girl, zeon, and xanado. By way of example, the multiple languages ​​706 can include, but are not limited to, English, Japanese, or Hindi.

[0064] 8, an exemplary GUI 800 displaying exercise parameters 802 is shown, according to some embodiments. Upon receiving a user selection for an exercise, the GUI 800 may prompt the user for the exercise parameters 802. By way of example, the exercise parameters 802 may include, but are not limited to, the number of reps, the number of sets, the interval, and the level of exercise (e.g., beginner or advanced). In one embodiment, a predefined group of parameters (e.g., CrossFit) may be provided to the user.

[0065] Referring now to FIG. 9, a user's At that time of activity An exemplary GUI 900 is shown displaying performance 902 and a postural skeletal model 904. In one embodiment, the GUI 900 may display the postural skeletal model 904 of the user overlaid on a real-time video of the user by a rendering device (such as rendering device 100). Additionally, the GUI 900 may activity Work with the experts Goal Activity Performance906 in an inset frame at the bottom right of the display device. Additionally, GUI 900 displays a message 908 to prepare the user for the exercise. Message 908 may also be provided as an audio output. Additionally, GUI 900 displays a set of user performance parameters, such as, but not limited to, a rep / step counter 910, number of reps 912, exercise 914, heart rate 916, and calories 918. The rep / step counter 910 indicates the user's progress and the progress they should make. the goal Activity Performance target posture Note that this is a sequence of

[0066] Referring now to FIG. 10, a user's At that time of activity An exemplary GUI 1000 is shown displaying performance 1002 and posture skeleton model 1004. The user confirms the message 908 and selects the initial posture Upon entering, the AI ​​model of the smart mirror (e.g., rendering device 100) posture Furthermore, the GUI 1000 can determine whether the exercise is correct. posture The system displays a message 1008 to the user indicating that the user's posture has been successfully recognized. The message 1008 may also be provided as an audio output. The user may be notified via text, graphics, visual, haptic, or audio output to begin the exercise. In an exemplary scenario, when the user's initial posture is incorrect, the goal Deviations from activity performance are presented to the user via messages (textual, graphical, audio, visual, or haptic).

[0067] Referring now to FIG. 11, a user's At that time of activity An exemplary GUI 1100 is shown displaying performance 1102. Once the user begins performing the exercise, At that time Activities performance The real-time video associated with 1102 is analyzed by an AI model in the smart mirror (e.g., rendering device 100). At that time of Activity Performance 1102 real-time video activity Expert Goal Activity Performance 1104. As an example, GUI 1100 may display a rep / step counter, a percentage of reps completed, a percentage of exercise completed, the user's heart rate, calories burned by the user, or any other user performance parameter. posture From the subsequent posture Upon successfully navigating to step 2, the step counter will change from a value of "1" to a value of "2," indicating that the user is on step 2. activity Note that a rep of a burpee has multiple steps. For example, a rep of a burpee has eight steps. At that time of Activity Performance 1102 is a target that exceeds a predetermined threshold capacity Activity Performance Upon deviation from 1104, a message 1106 is displayed on the display with corrective action for the user (e.g., "straighten your back"). The message 1106 may also be provided as an audio output and a graphical display. Additionally, activity Expert Goal Activity Performance 1104 is the user At that time of Activity Performance Displayed to comply with 1102.

[0068] Some embodiments of the present disclosure may be used in gyms, rehabs, physical therapy in hospitals, dance studios, theaters, or any other use case scenario. activity The present invention may include a plurality of exercise machines and equipment for performing the following: A user may use a rendering device (e.g., rendering device 100) or any other display device (e.g., a smart mirror) to activity By category activity You can select activityassociated with activity The attributes can be selected accordingly. Multiple cameras can be activity of capture It can be done and is running activity The rendering device 100 can provide the user with relevant instructions and feedback to improve their performance. The camera can also be used for facial recognition to identify the user and provide them with history, customized settings, messages from the trainer, profile data, and other similar data. In a gym, a single rendering device can include multiple screens and GUIs to provide exercise training for multiple users. Additionally, the rendering device 100 can be configured to output audio feedback via a Bluetooth headset or speaker.

[0069] The camera allows users to activity of users in the gym when moving from one area or from one machine to another to perform activity The rendering device 100 may be used to track and record the movements of the camera-using users as they move from one area of ​​the gym to another. At that time The camera can track the progress of the user. The camera can enable continuity of user context and information across the monitoring. A gym (or any other use case scenario) can be used by a personal coach, trainer, or physical therapist. activity Various types of professionals, such as experts, physiotherapists, occupational therapists, physical education teachers, martial arts teachers, dance and choreography, sports personalities, team coaches, as well as demonstrators and other trainers in health and fitness. activity It may have experts.

[0070] It will be held in the gym activity and the goals achieved by the user are communicated with one or more other users practicing in the gymnasium, or with one or more remote users; activity It can be shared by experts or users.

[0071] Additionally, user performance may be posted and published on social media platforms and made available as online content for access by one or more remote users. This can be done through gamification of the activities performed by the user and using reward mechanisms. activity Scores associated with the activity may be presented as points on a leaderboard. Badges may be assigned to users based on the level of activity performed. Additionally, records associated with the activity performed may include, for example, accuracy, total number of exercises performed, rest periods between exercises, etc., and may be provided to one or more users and rendered. device and may be displayed on a smart mirror. Additionally, the rendering device 100 may include additional features, such as an unlocking feature, additional activities, designs, and other similar features.

[0072] In one embodiment, the rendering device 100 can be used to create content media, e.g., At that time Content media including information related to a user's health status, exercise routine, athletic performance, and previous records and earned rewards on social media platforms can be shared through an application programming interface (API) integrated with the rendering device 100.

[0073] The rendering device 100 can be used as a recording tool for creating new fitness content and as associated instructions for activities to be performed by a user received via voice-based input. Additionally, both display devices and smart mirrors, such as those used in gyms (or any other use case scenario), can be used to record and record user-selected fitness content. activity may be used to edit and review new content related to activityIt can be used by experts to review a user's session. device 100 is user rendered device 100 can be connected to a health and fitness application that can be logged into. Feedback received on the health and fitness application can be shared on social platforms for social engagement and displayed to other socially connected parties or groups in the formation of leaderboards. activity Relevant data may be provided.

[0074] Furthermore, the rendering device 100 may be activity As can be appreciated, the rendering device 100 may be used by a user and an expert as a recording device. activity Experts can crop, highlight, add audio, and provide voice-to-text feedback on the smart mirror. activity The expert may be allowed to add or remove background images to be used in the smart mirror. activity Experts can create metadata, instructions, threshold parameters, and combinations thereof. Recorded videos can be shared with other users. Additionally, parameters collected by the rendering device can be processed to create metadata, instructions, threshold parameters, and combinations thereof.

[0075] The rendering device 100 uses one or more cameras and one or more other sensors to determine the user's position while performing an activity. capture Feedback based on the activity being performed by the user can be the goalThe feedback may include, but is not limited to, instructions related to slowing down displays and other media components, such as video in motion, but may also include other feedback, such as pulsating and rhythmic audio cues, such as a metronome. performance Guidance clues, the goal A tight coupling of the user's movements can be performed to provide motion, media, and voice and audio feedback. The rendering device 100 can map and synchronize the media and information provided with the user's actual movements, and thus provide relevant and corresponding feedback.

[0076] In one embodiment, a user may activity By category activity A multilingual voice-based interface may be provided to allow users to navigate, select, schedule, and sequence activities. Voice-based input can create and save playlists, add metadata to playlists, add comments to playlists and activities using voice-to-text mechanisms and voice feedback, record new activity categories, edit and clip performed activities, tag exercises with hashtags, and tag, for example, exercises. type It can be used to tag exercise clips, muscle groups or difficulty levels, replace exercise clips with alternative versions, share playlists and exercises with other users, and indicate a message for another user when sharing a playlist.

[0077] Some embodiments of the present disclosure may be implemented as an AI-based health and fitness system training method, which includes detecting a user, determining the user's posture and body movements using a camera, further sensing the body or user's movements, motion, position, and / or movement using a sensor, directing and monitoring exercise through the user's posture determination and body movements on a smart mirror, overlaying the posture and movements on the user's mirror reflection, and providing real-time feedback. activity Overlaying postures on the expert's video stream, showing posture positions in conjunction with the training video and the user, tracking movements in real time, automatic recalculation, and in the movement sequence. the goal Includes correlation and accuracy, and real-time social media sharing to groups, friends, and others. In an alternative embodiment, an artificial intelligence based health and fitness training method includes a camera for determining a user's posture and body movements, a microphone for listening to the user's voice commands, and a speaker for providing feedback to the user regarding the user's movements, whereby the method provides for determining a user's posture and body movements in an exercise sequence, for automatic recalculation, for real-time tracking of the exercise.

[0078] In one embodiment, the real-time live feedback may be provided by voice control instructions, visuals including video or text or graphic elements based instructions for exercises, scripts, exercise sequences, and / or activity The rendering device 100 may include, for example, information related to the user's account access details, the user's workout history, and other information related to the user. posture and body movements and display the user's movements on the user's mirror reflection. posture and overlaying the movement, activityOn expert video streams posture overlays and interacts with training videos and users posture Based on the AI ​​model, the rendering device 100 may provide live feedback to the user, for example, via audio feedback, video feedback, text feedback, or graphical feedback, based on instructions related to the exercise / activity, the script, the sequence of the exercise, and / or the user's performance during the exercise.

[0079] In one embodiment, for efficient display of the guide steps and placement of visual information related to the guide steps, the GUI may be presented on a screen of a rendering device. The rendering device GUI may be adjusted based on the user's eye position, so that the information is displayed in the user's video stream. capture Instead, it is positioned appropriately in the reflection or in some fixed position.

[0080] Additionally, a 3D model of the user's posture and movement on the smart mirror is provided. This involves overlaying the provided 3D model on a reflection, which can then be rendered along with the analysis for display through the smart mirror. As can be seen, the user's posture and movement can be captured midway along the long side of the mirror. capture The camera or cameras may be adjusted to allow for a better aspect ratio of the user's pose.

[0081] The video recording can be created using at least one camera located at a distributed location. The rendering device 100 can be used to create new training content. activity To record content that will be reviewed later by experts, physiotherapists, teachers, choreographers, and activity May include a recording device for real-time sharing of expert or user live streams.

[0082] It will also be understood that the techniques described above can take the form of computer- or controller-implemented processes and apparatuses for performing those processes. The present disclosure can also be embodied in the form of computer program code including instructions embodied in a tangible medium, such as a floppy disk drive, diskette, solid-state drive, CD-ROM, hard drive, or any other computer-readable storage medium, such that when the computer program code is loaded into a computer or controller and executed, the computer becomes an apparatus for practicing the present invention. The present disclosure can also be embodied in the form of computer program code or signals, for example, whether stored on a storage medium, loaded and / or executed on a computer or controller, or transmitted over some transmission medium, such as via electrical wiring or cable, via optical fiber, or via electromagnetic radiation, such that when the computer program code is loaded into a computer and executed by the computer, the computer becomes an apparatus for practicing the present invention. When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits.

[0083] The disclosed methods and systems may be implemented on conventional or general-purpose computer systems, such as personal computers (PCs) or server computers. Referring now to FIG. 12 , an exemplary computing system 1200 is shown that may be employed to implement processing functionality for various embodiments (e.g., as a SIMD device, a client device, a server device, one or more processors, etc.). Those skilled in the art will recognize how to implement the present invention using other computer systems or architectures. Computing system 1200 may represent, for example, a user device, such as a desktop, laptop, mobile phone, personal entertainment device, DVR, or any other type of specialized or general-purpose computing device that may be desirable or appropriate for a given application or environment. Computing system 1200 may include, for example, one or more processors, such as processor 1202, which may be implemented using a special-purpose processing engine, such as a microprocessor, microcontroller, or other control logic. In this example, processor 1202 is connected to bus 1204 or other communications medium. In some embodiments, processor 1202 may be an artificial intelligence (AI) processor, which may be implemented as a tensor processing unit (TPU), a graphical processing unit, or a custom programmable solution field-programmable gate array (FPGA).

[0084] Computing system 1200 may also include memory 1206 (main memory), such as random access memory (RAM) or other dynamic memory, for storing information and instructions to be executed by processor 1202. Storage 1206 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1202. Computing system 1200 may also include read-only memory ("ROM") or other static storage devices coupled to bus 1204 for storing static information and instructions for processor 1202. Computing system 1200 also includes storage device 1208, which may include, for example, a media drive 1210 and a removable storage interface. Media drive 1210 may include a drive or other mechanism for supporting fixed or removable storage media, such as a hard disk drive, floppy disk drive, magnetic tape drive, SD card port, USB port, micro USB, optical disk drive, CD or DVD drive (R or RW), or other removable or fixed media drive. Storage medium 1212 may include, for example, a hard disk, magnetic tape, flash drive, or other fixed or removable media that is read or written by media drive 1210. As these examples illustrate, storage medium 1212 may include a computer-readable storage medium having specific computer software or data stored thereon.

[0085] In alternative embodiments, storage device 1208 may include other similar means for allowing computer programs or other instructions or data to be loaded into computing system 1200. Such means may include, for example, removable storage unit 1214, storage unit interface 1216 such as a program cartridge and cartridge interface, removable memory (e.g., flash memory or other removable memory module) and memory slots, and other removable storage units and interfaces that allow software and data to be transferred from removable storage unit 1214 to computing system 1200.

[0086] Computing system 1200 may also include a communications interface 1218. Communications interface 1218 may be used to allow software and data to be transferred between computing system 1200 and external devices. Examples of communications interface 1218 may include a network interface (such as an Ethernet or other NIC card), a communications port (e.g., a USB port, a micro USB port, etc.), a near-field communication (NFC), and other similar communications interfaces. The software and data transferred via communications interface 1218 are in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1218. These signals are provided to communications interface 1218 via channel 1220. Channel 1220 may carry signals and may be implemented using a wireless medium, wire or cable, optical fiber, or other communications medium. Some examples of channel 1220 may include a telephone line, a cellular telephone link, an RF link, a Bluetooth link, a network interface, a local or wide area network, and other communications channels.

[0087] Computing system 1200 may further include input / output (I / O) devices 1222. Examples include, but are not limited to, a display, a keypad, a microphone, an audio speaker, a vibration motor, LED lights, and other similar I / O devices. I / O devices 1222 may receive input from a user and may also display the output of computations performed by processor 1202. As used herein, the terms “computer program product” and “computer-readable medium” may generally refer to media such as, for example, memory 1206, storage device 1208, removable storage unit 1214, or signals on channel 1220. These and other forms of computer-readable media may be involved in providing one or more sequences of one or more instructions to processor 1202 for execution. Such instructions, generally referred to as “computer program code” (which may be grouped together in the form of a computer program or other grouping), when executed, enable computing system 1200 to perform the features or functionality of embodiments of the present invention.

[0088] In embodiments in which elements are implemented using software, the software may be stored on a computer-readable medium and loaded into computing system 1200 using, for example, removable storage unit 1214, media drive 1210, or communications interface 1218. The control logic (in this example, software instructions or computer program code) when executed by processor 1202 causes processor 1202 to perform the functions of the present invention as described herein.

[0089] As will be appreciated by those skilled in the art, the techniques described in the various embodiments above are not routine, conventional, or well understood in the art. activity The technique first provides training users to perform multiple activity types via a graphical user interface (GUI) on a rendering device. Choicesto the user. activity type Choices Each of the activity The technique then includes: activity Selected by the user from a selection activity The user input may include at least one of a gesture, a touch, or an audio command. The technology then captures the selected selection by at least one camera. activity Based on the choice activity Real-time video of the user running capture Each of the at least one camera may capture real-time video of the user from an associated predetermined angle. capture The real-time video captures the postures and movements made by the user to perform the activity. flow The technique then performs the steps of: activity Based on the choices, an AI model can be extracted. activity Expert Goal Activity Performance Based on activity Responding to choices activity The technique is then configured to determine the deviation of the user from a plurality of correct movements associated with the user. At that time Activities performance Based on user performance The AI ​​model may process a real-time video of the user in real time to determine a set of parameters. The AI ​​model then processes the user in the real-time video. posture The posture skeleton model can be overlaid with the skeletal model. activity Each of the plurality of key points is based on a corresponding key point of the user in the real-time video. joint The technology then uses an AI model to generate a set of user performance parameters. the goal It may be compared to a set of activity performance parameters. Goal Activity The set of performance parameters is activityThe technology then uses an AI model to generate a set of user performance parameters and the goal Feedback can be generated for the user based on a comparison with the set of activity performance parameters. The feedback includes at least one of a corrective action or a warning. The feedback includes at least one of visual feedback, auditory feedback, or haptic feedback. The technology can then render the feedback on a rendering device by the AI ​​model. Rendering the feedback includes overlaying one of the at least one corrective action on a postural skeletal model overlaid on a real-time video of the user. Rendering the feedback can include overlaying one of the at least one corrective action on a GUI of the rendering device. caveat Rendering the feedback further includes outputting auditory feedback to the user via a speaker.

[0090] In light of the above advantages and the technical advances provided by the disclosed methods and systems, the claimed processes discussed above are not routine, conventional, or well-understood in the art because the claimed processes enable the following solutions to existing problems in the prior art: Furthermore, the claimed processes clearly result in an improvement in the functionality of the device itself because the claimed processes provide a technical solution to a technical problem.

[0091] This specification activityThis specification describes a method and system for training a user to perform a program. The illustrated processes are presented to explain the illustrated exemplary embodiments, and it is to be expected that ongoing technological developments will change the way in which particular functions are performed. These examples are presented herein for purposes of explanation and not limitation. Moreover, the boundaries of functional components have been arbitrarily defined herein for convenience of description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, and deviations from those described herein) will be apparent to those skilled in the art based on the teachings contained herein. Such alternatives are within the scope and spirit of the disclosed embodiments.

[0092] Additionally, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory in which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD-ROMs, DVDs, flash drives, hard disks, and any other known physical storage medium.

[0093] It is intended that the present disclosure and examples be considered as exemplary only, with a true scope and spirit of the disclosed embodiments being indicated by the following claims.

Claims

1. 1. A method for training a user to perform a physical activity, comprising: Rendering a plurality of activity type options to a user via a graphical user interface (GUI) of a rendering device, each of the plurality of activity type options comprising a plurality of activities; receiving an activity option selected by a user from a plurality of activity options in response to a user input, the user input comprising at least one of a gesture, a touch, an audio command, or a signal generated from an input device; capturing, with at least one camera, real-time video of the user performing an activity based on the selected activity option, each of the at least one camera capturing real-time video of the user from an associated predefined angle, the real-time video comprising a sequence of postures and movements made by the user to perform the activity; extracting an AI model based on an activity selection selected by the user, the AI ​​model being configured to determine a deviation of the user from a plurality of correct movements associated with an activity corresponding to the activity selection based on a target activity performance of an activity expert; processing, in real time, a real-time video of the user with an AI model to determine a set of user performance parameters based on the user's current activity performance; overlaying, by the AI ​​model, a posture skeletal model on the user in the real-time video, the posture skeletal model comprising a plurality of activity-based key points, each of the plurality of key points being overlaid on a corresponding joint of the user in the real-time video; comparing, by the AI ​​model, the set of user performance parameters with a set of target activity performance parameters, the set of target activity performance parameters corresponding to the activity expert; generating, by the AI ​​model, feedback for the user based on a comparison of the set of user performance parameters to the set of target activity performance parameters, the feedback comprising at least one of a corrective action or a warning, and the feedback comprising at least one of visual feedback, auditory feedback, or haptic feedback; Rendering feedback on a rendering device with the AI ​​model, overlaying one of the at least one corrective action on a postural skeletal model overlaid on a real-time video of the user; Displaying a warning on the rendering device's GUI; and outputting auditory feedback to the user via a speaker. pausing the display of the set of user performance parameters and the target activity performance, wherein the display is paused if the activity performance at that time deviates from the target activity performance by more than a predefined threshold performance for a predefined threshold time based on the comparison; Including, overlaying the postural skeletal model on the user in the real-time video; automatically adjusting and normalizing the posture skeleton model based on the user's current posture and estimated future distance relative to the rendering device, and the user's current posture and estimated future posture and motion; the feedback includes generating an alert to the user; The warning may be: instructions to correct the user's current posture; instructions for correcting the user's movements relative to the user's current posture; and instructions for correcting the user's current position when the user is at least partially outside the field of view of the at least one camera; further comprising: method.

2. The method of claim 1 , further comprising displaying the set of user performance parameters, the set of target activity performance parameters, and the target activity performance of the activity expert through the GUI of the rendering device.

3. The method of claim 2 , further comprising overlaying the target activity performance in real time over the current activity performance of the user in the real-time video.

4. 2. The method of claim 1, wherein the set of user performance parameters includes the current activity performance speed, number of repetitions completed, overall completion of an activity circuit, third-party smart device information, the user's pulse rate, the user's blood pressure, and the user's movement, and the set of target activity performance parameters includes the target activity performance speed, target number of repetitions, the user's target pulse rate, and the user's target movement.

5. and rendering at least one of a trainer avatar corresponding to a target activity performance of the activity expert and a user avatar corresponding to a current activity performance of the user upon capturing real-time video via the at least one camera, wherein the trainer avatar is a three-dimensional (3D) model of the activity expert and the user avatar is a multi-dimensional model of the user. The method of claim 1.

6. comparing the trainer avatar and / or the user avatar corresponding to the target activity performance with the user's current activity performance based on the activity type and activity; displaying, via the rendering device and via the GUI, the set of user performance parameters, at least one of a trainer avatar and the user avatar corresponding to the target activity performance, and the user's current activity performance, wherein the current activity performance is overlaid in real time on at least one of the trainer avatar and the user avatar corresponding to the target activity performance. The method of claim 5.

7. storing the real-time video received from the at least one camera in a database; Editing the real-time video based on one or more user commands of the user, the user commands being at least one of a text command, a voice command, a touch command, or a visual gesture, the one or more user commands including: Setting a starting point for the real-time video; Setting an end point for the real-time video; Removing background from real-time video; assigning one or more tags to the real-time video; and sharing the real-time video with other users; The method of claim 1.

8. receiving real-time video from an activity expert in real time corresponding to a target activity performance according to the user's current activity performance; displaying the set of user performance parameters and the activity expert's target activity performance in real time through a GUI via a rendering device, wherein the target activity performance is overlaid over the user's current activity performance in the real-time video; and wherein the target activity performance is overlaid over the user's current activity performance in the real-time video. The method of claim 1.

9. Detecting an initial position of a user via at least one camera; determining whether the detected initial location of the user matches an initial location mapped to at least one activity; and if the detected initial position does not match the initial position, prompting the user to correct the initial position. The method of claim 1.

10. The method of claim 1, wherein the user command to select the activity option includes at least one of a voice command, a touch gesture, an air gesture, an eye gesture, or a signal generated by an input device.

11. 1. A rendering device for training a user to perform a physical activity, comprising: A display device including a GUI (Graphical User Interface), the GUI comprising: Rendering a plurality of activity type options to the user, each of the plurality of activity type options comprising a plurality of activities; a display device configured to receive a user-selected activity option from a plurality of activity type options in response to a user input, the user input comprising at least one of a gesture, a touch, and an audio command; at least one camera configured to capture real-time video of the user performing the activity based on the selected activity option, each of the at least one camera configured to capture the real-time video of the user from an associated predefined angle, the real-time video including a sequence of postures and movements made by the user to perform the activity; a processor; a memory communicatively coupled to the processor, the memory storing processor instructions that, when executed by the processor: extracting an AI model based on an activity selection selected by the user, the AI ​​model being configured to determine a deviation of the user from a plurality of correct movements associated with an activity corresponding to the activity selection based on a target activity performance of an activity expert; processing, in real time, a real-time video of the user with an AI model, to determine a set of user performance parameters based on the current activity performance; overlaying, by the AI ​​model, a posture skeletal model on the user in the real-time video having the posture skeletal model, the posture skeletal model comprising a plurality of activity-based key points, each of the plurality of key points being overlaid on a corresponding joint of the user in the real-time video; comparing, by the AI ​​model, the set of user performance parameters with a set of target activity performance parameters, the set of target activity performance parameters corresponding to the activity expert; generating, by the AI ​​model, feedback for the user based on a comparison of the set of user performance parameters to the set of target activity performance parameters, the feedback comprising at least one of a corrective action or a warning, and the feedback comprising at least one of visual feedback, auditory feedback, or haptic feedback; Rendering feedback on a rendering device with the AI ​​model, overlaying one of the at least one corrective action on a postural skeletal model overlaid on a real-time video of the user; Displaying a warning on the rendering device's GUI; and outputting auditory feedback to the user via a speaker. a memory for causing a processor to execute the Including, pausing the display of the set of user performance parameters and the goal activity performance when the activity performance at that time deviates from the goal activity performance by more than a predefined threshold performance for a predefined threshold time based on the comparison; and overlaying the posture skeleton model on the user in the real-time video, the processor instructions, when executed, cause the processor to automatically adjust and normalize the posture skeleton model based on the user's current posture and estimated future distance relative to a rendering device, and the user's current posture and estimated future posture and motion; the feedback includes generating an alert to the user; The warning may be: instructions to correct the user's current posture; instructions for correcting the user's movements relative to the user's current posture; instructions for correcting the user's current position when the user is at least partially outside the field of view of the at least one camera; Rendering device.

12. The processor instructions cause the processor to display, through a GUI of a rendering device, the set of user performance parameters, the set of target activity performance parameters, and the target activity performance of an activity expert; a real-time overlay of the target activity performance against the user's current activity performance in the real-time video; Rendering device according to claim 11.

13. The processor instructions, when executed, cause the processor to: Rendering at least one of a trainer avatar corresponding to a target activity performance of the activity expert and a user avatar corresponding to a current activity performance of the user as real-time video is captured via at least one camera, wherein the trainer avatar is a three-dimensional (3D) model of the activity expert and the user avatar is a multi-dimensional model of the user; comparing at least one of a trainer avatar and a user avatar corresponding to a target activity performance with the user's current activity performance based on the activity type and the activity; The rendering device of claim 11, further comprising a step of displaying, via the rendering device and via a GUI, the user performance parameter set, at least one of a trainer avatar and a user avatar corresponding to the target activity performance, and the user's current activity performance, wherein the current activity performance is overlaid in real time on at least one of the trainer avatar and the user avatar corresponding to the target activity performance.

14. When a processor instruction is executed, the processor: storing the real-time video received from the at least one camera in a database; Editing the real-time video based on one or more user commands of the user, the user commands being at least one of a text command, a voice command, a touch command, or a visual gesture, the one or more user commands including: Setting a starting point for real-time video; Setting the end point of the real-time video; Removing background from real-time video; assigning one or more tags to the real-time video; and and sharing the real-time video with other users.

15. The processor instructions, when executed, cause the processor to: receiving real-time video from the activity expert in real time corresponding to the target activity performance in response to the user's current activity performance; The rendering device of claim 11, wherein the rendering device displays the user performance parameter set and the target activity performance of the activity expert in real time via the GUI, and the target activity performance is overlaid on the user's current activity performance in the real-time video.

16. The processor instructions, when executed, cause the processor to: Detecting an initial position of a user via at least one camera; determining whether the detected initial location of the user matches an initial location mapped to at least one activity; The rendering device of claim 11 , further comprising the step of prompting a user to correct the initial position if the detected initial position does not match the initial position.

Images