Systems and methods for real-time interaction and guidance

Abstract

Methods and systems are described for real-time coaching and guidance using a virtual assistant that interacts with a user. A user can receive feedback inferences provided substantially in real-time after a video sample is collected from the user's device. Neural network architectures and layers can be used to determine motion patterns and temporal aspects of the video sample, as well as to detect the activity of the foreground user despite background noise. The methods and systems can have various capabilities, including but not limited to real-time feedback on the exercise activity performed, exercise score, calorie estimation, and repetition count.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 982,793, filed on February 28, 2020, the contents of which are incorporated herein by reference in their entirety.

[0003] field

[0004] The described embodiments generally relate to systems and methods for real-time interaction, and more particularly to real-time exercise guidance based on video data.

[0005] background

[0006] Fitness guidance and / or training provided by live coaches is very expensive and out of reach for many users.

[0007] Interaction with automated virtual assistants takes several different forms. Firstly, it can be achieved using smart speakers, such as… Alexa Siri and The assistants are virtual assistants. However, these virtual assistants only allow voice-based interaction and can only recognize simple queries. Secondly, many service robots exist, but most lack complex human interaction capabilities and are basic "blind chatbots with bodies."

[0008] These assistants do not provide visual interaction, including visual interaction using video data from the user's device. For example, existing virtual assistants cannot understand the surrounding video scene, objects and actions in the video, spatial and temporal relationships within the video, human behavior shown in the video, spoken language in the video, space and time described in the video, have visual-based concepts about the causes of real-world events, or have memory or an understanding of time.

[0009] One challenge in creating virtual assistants that provide visual interaction is determining the methodology for training data, as the various quantitative aspects of labeled data (e.g., the velocity labeling of video data by human reviewers) are inherently subjective. This makes it difficult to label large amounts of video using such labels, especially when multiple people are involved in the process (a common occurrence when labeling large datasets).

[0010] Virtual assistants still need improvement, with enhanced human-like interaction for personalized guidance, including video interaction using the camera of smart devices such as smartphones.

[0011] Overview

[0012] If a neural network is configured to process camera streams showing a user performing physical activities in real time, it can be used for real-time teaching and guidance. Such networks can drive teaching or guidance applications by providing real-time feedback and / or by collecting information about the user's activities, such as counts or intensity measurements.

[0013] In a first aspect, a method for providing feedback to a user at a user equipment is provided, the method comprising: providing a feedback model; receiving a video signal at the user equipment, the video signal comprising at least two video frames, a first video frame of the at least two video frames being captured prior to a second video frame of the at least two video frames; generating an input layer of the feedback model comprising the at least two video frames; determining a feedback inference associated with the second video frame of the at least two video frames based on the feedback model and the input layer; and outputting the feedback inference to the user using an output device of the user equipment.

[0014] In one or more embodiments, the feedback model may include a backbone network and at least one head network.

[0015] In one or more embodiments, the backbone network may be a three-dimensional convolutional neural network.

[0016] In one or more embodiments, each of the at least one head network may be a neural network.

[0017] In one or more embodiments, the at least one head network may include a global activity detection head network, which can be used to determine the activity classification of the video signal based on the layers of the backbone network; and the feedback inference may include the activity classification.

[0018] In one or more embodiments, the activity classification may include at least one selected from a group of exercise scores, calorie estimates, and exercise form feedback.

[0019] In one or more embodiments, the training score may be a continuous value determined based on a weighted sum of the softmax outputs of multiple activity markers from the global activity detection head network.

[0020] In one or more embodiments, the at least one head network may include a discrete event detection head network for determining at least one event from the video signal based on layers of the backbone network, each of the at least one event may include an event classification; and the feedback inference may include the at least one event.

[0021] In one or more embodiments, each of the at least one event may further include a timestamp corresponding to the video signal; and the at least one event may correspond to a portion of a user's exercise repetition.

[0022] In one or more embodiments, the feedback inference may include exercise repetition counts.

[0023] In one or more embodiments, the at least one head network may include a local activity detection head network for determining at least one bounding box from the video signal based on layers of the backbone network and determining an activity classification corresponding to each bounding box in the at least one bounding box; and the feedback inference may include the at least one bounding box and the activity classification corresponding to each bounding box in the at least one bounding box.

[0024] In one or more embodiments, the feedback inference may include an activity classification for one or more users, the bounding box corresponding to the one or more users.

[0025] In one or more embodiments, the video signal may be a video stream received from the video capture device of the user equipment, and the feedback inference may be provided in near real-time upon receiving the video stream.

[0026] In one or more embodiments, the video signal may be a video sample received from the storage device of the user equipment.

[0027] In one or more embodiments, the output device may be an audio output device, and the feedback inference may be an audio prompt to the user.

[0028] In one or more embodiments, the output device may be a display device, and the feedback inference may be provided as subtitles superimposed on the video signal.

[0029] In a second aspect, a system is provided for providing feedback to a user at a user equipment, the system comprising: a memory including a feedback model; an output device; and a processor communicating with the memory and the output device, wherein the processor is configured to: receive a video signal at the user equipment, the video signal including at least two video frames, a first video frame of the at least two video frames being captured prior to a second frame of the at least two video frames; generate an input layer including the feedback model of the at least two video frames; determine a feedback inference associated with the second video frame of the at least two video frames based on the feedback model and the input layer; and use the output device to output the feedback inference to the user.

[0030] In one or more embodiments, the feedback model may include a backbone network and at least one head network.

[0031] In one or more embodiments, the backbone network may be a three-dimensional convolutional neural network.

[0032] In one or more embodiments, each of the at least one head network may be a neural network.

[0033] In one or more embodiments, the at least one head network may include a global activity detection head network for determining the activity classification of the video signal based on the layers of the backbone network; and the feedback inference may include the activity classification.

[0034] In one or more embodiments, the activity classification may include at least one selected from a group of exercise scores, calorie estimates, and exercise form feedback.

[0035] In one or more embodiments, the training score may be a continuous value determined based on a weighted sum of the softmax outputs of multiple activity markers from the global activity detection head network.

[0036] In one or more embodiments, the at least one head network may include a discrete event detection head network for determining at least one event from the video signal based on layers of the backbone network, each of the at least one event may include an event classification; and the feedback inference may include the at least one event.

[0037] In one or more embodiments, each of the at least one event may further include a timestamp corresponding to the video signal; and the at least one event may correspond to a portion of a user's exercise repetition.

[0038] In one or more embodiments, the feedback inference may include exercise repetition counts.

[0039] In one or more embodiments, the at least one head network may include a local activity detection head network, the local activity detection network being used to determine at least one bounding box from the video signal based on layers of the backbone network and to determine an activity classification corresponding to each of the at least one bounding box; and the feedback inference may include the at least one bounding box and the activity classification corresponding to each of the at least one bounding box.

[0040] In one or more embodiments, the feedback inference may include an activity classification for one or more users, the bounding box corresponding to the one or more users.

[0041] In one or more embodiments, the video signal may be a video stream received from the video capture device of the user equipment, and the feedback inference is provided in near real-time upon receiving the video stream.

[0042] In one or more embodiments, the video signal may be a video sample received from the storage device of the user equipment.

[0043] In one or more embodiments, the output device may be an audio output device, and the feedback inference is an audio prompt to the user.

[0044] In one or more embodiments, the output device may be a display device, and the feedback inference may be provided as subtitles superimposed on the video signal.

[0045] In a third aspect, a method for generating a feedback model is provided, the method comprising: transmitting a plurality of video samples to a plurality of labeled users, each of the plurality of video samples including video data, each of the plurality of labeled users receiving at least two video samples from the plurality of video samples; receiving a plurality of ranking responses from the plurality of labeled users, each of the plurality of ranking responses indicating a relative ranking selected by a corresponding labeled user from the at least two video samples transmitted to the corresponding labeled user based on a ranking criterion; determining a sorting label for each of the plurality of video samples based on the plurality of ranking responses and the ranking criterion; organizing the plurality of video samples into a plurality of buckets based on the corresponding sorting label for each video sample; determining a classification label for each of the plurality of buckets; and generating a feedback model based on the plurality of buckets, the classification label for each corresponding bucket, and the video samples for each corresponding bucket.

[0046] In one or more embodiments, generating the feedback model may include applying gradient-based optimization to determine the feedback model.

[0047] In one or more embodiments, the feedback model may include at least one head network.

[0048] In one or more embodiments, each of the at least one head network may be a neural network.

[0049] In one or more embodiments, the method may further include determining that a sufficient number of ranking responses from the plurality of labeled users have been received.

[0050] In one or more embodiments, the ranking criteria may include at least one selected from a group of exercise speed, repetition, and range of motion.

[0051] In one or more embodiments, the ranking criteria may be associated with a specific type of physical exercise.

[0052] In a fourth aspect, a system for generating a feedback model is provided, the system comprising: a memory including a plurality of video samples; a network device; and a processor communicating with the memory and the network device, the processor being configured to: transmit a plurality of video samples to a plurality of labeled users using the network device, each of the plurality of video samples including video data, each of the plurality of labeled users receiving at least two of the plurality of video samples; receive a plurality of ranking responses from the plurality of labeled users using the network device, each of the plurality of ranking responses indicating a relative ranking selected by a corresponding labeled user from the at least two video samples transmitted to the corresponding labeled user based on a ranking criterion; determine a sorting label for each of the plurality of video samples based on the plurality of ranking responses and the ranking criterion; organize the plurality of video samples into a plurality of buckets based on the corresponding sorting label for each video sample; determine a classification label for each of the plurality of buckets; and generate a feedback model based on the plurality of buckets, the classification label for each corresponding bucket, and the video samples for each corresponding bucket.

[0053] In one or more embodiments, the processor may be further configured to apply gradient-based optimization to determine the feedback model.

[0054] In one or more embodiments, the feedback model may include at least one head network.

[0055] In one or more embodiments, each of the at least one head network may be a neural network.

[0056] In one or more embodiments, the processor may be further configured to determine that a sufficient number of ranking responses from the plurality of labeled users have been received.

[0057] In one or more embodiments, the ranking criteria may include at least one selected from a group of exercise speed, repetition, and range of motion.

[0058] In one or more embodiments, the ranking criteria may be associated with a specific type of physical exercise. Brief description of the attached diagram

[0060] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which:

[0061] Figure 1 This is a system diagram of a user device for real-time interaction and guidance according to one or more embodiments;

[0062] Figure 2 This is a diagram illustrating a method for real-time interaction and guidance according to one or more embodiments;

[0063] Figure 3It is a scene illustration for real-time interaction and guidance according to one or more embodiments;

[0064] Figure 4 It is a diagram illustrating a user interface for real-time interaction and guidance, including a virtual avatar, according to one or more embodiments;

[0065] Figure 5 These are user interface diagrams for real-time interaction and guidance according to one or more embodiments;

[0066] Figure 6 These are user interface diagrams for real-time interaction and guidance according to one or more embodiments;

[0067] Figure 7 This is another user interface illustration for real-time interaction and guidance according to one or more embodiments;

[0068] Figure 8 It is a tabular illustration for exercise scoring according to one or more embodiments;

[0069] Figure 9 This is another tabular illustration of exercise scoring according to one or more embodiments;

[0070] Figure 10 This is a system diagram for generating a feedback model according to one or more embodiments;

[0071] Figure 11 This is a diagram illustrating a method for generating a feedback model according to one or more embodiments;

[0072] Figure 12 It is a model diagram for determining feedback inference according to one or more embodiments;

[0073] Figure 13 This is a stepable convolution diagram for determining feedback inference according to one or more embodiments;

[0074] Figure 14 It is a user interface diagram for generating time stamps for a feedback model according to one or more embodiments;

[0075] Figure 15 It is a user interface diagram of paired markers for generating a feedback model according to one or more embodiments;

[0076] Figure 16 It is a comparison of the accuracy of pairwise ranking tags and manually labeled rankings, where pairwise rankings are generated by comparing each video with 10 other videos;

[0077] Figure 17It is another user interface for real-time interaction and guidance, according to one or more embodiments.

[0078] Description of exemplary embodiments

[0079] It should be understood that numerous specific details have been set forth in order to provide a thorough understanding of the exemplary embodiments described herein. However, those skilled in the art will understand that the embodiments described herein can be practiced without these specific details. In other instances, well-known methods, processes, and components have not been described in detail so as not to obscure the embodiments described herein. Furthermore, this specification and the accompanying drawings should not be construed as limiting the scope of the embodiments described herein in any way, but merely as illustrating implementations of the embodiments described herein.

[0080] It should be noted that, as used herein, degree terms such as “basically,” “about,” and “approximately” refer to a reasonable amount of deviation from the term they modify such that the final result is not significantly altered. These degree terms should be interpreted as including the deviation from the term they modify if such deviation does not negate the meaning of the term they modify.

[0081] Furthermore, as used herein, the term “and / or” is intended to indicate inclusion or. That is, for example, “X and / or Y” is intended to mean X or Y or both. As a further example, “X, Y and / or Z” is intended to mean X or Y or Z or any combination thereof.

[0082] The various embodiments of the systems and methods described herein can be implemented in hardware or software, or a combination of both. These embodiments can be implemented in a computer program that executes on a programmable computer, each computer including at least one processor, a data storage system (including volatile or non-volatile memory or other data storage elements or combinations thereof), and at least one communication interface. For example, but not limited to, a programmable computer (hereinafter referred to as a computing device) can be a server, network device, embedded device, computer expansion module, personal computer, laptop computer, personal data assistant, cellular phone, smartphone device, tablet computer, wireless device, or any other computing device that can be configured to perform the methods described herein.

[0083] In some embodiments, the communication interface may be a network communication interface. In various embodiments of the combined elements, the communication interface may be a software communication interface, such as those for inter-process communication (IPC). In other embodiments, combinations of implemented communication interfaces may exist, such as hardware, software, and combinations thereof.

[0084] Program code can be applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

[0085] Each program can be implemented using a high-level procedure or object-oriented programming and / or scripting language, or both, to communicate with the computer system. However, if desired, these programs can be implemented using assembly language or machine language. In any case, the language can be a compiled language or an interpreted language. Each such computer program can be stored on a general-purpose or special-purpose programmable computer-readable storage medium or device (e.g., ROM, disk, optical disk) for configuring and operating the computer when the computer reads the storage medium or device to perform the processes described herein. Embodiments of the system can also be considered as non-transitory computer-readable storage media configured with computer programs, wherein such a storage medium causes the computer to operate in a specific and predefined manner to perform the functions described herein.

[0086] Furthermore, the systems, processes, and methods of the described embodiments can be distributed in a computer program product, which includes a computer-readable medium carrying computer-usable instructions for one or more processors. This medium can be provided in various forms, including one or more floppy disks, compressed disks, magnetic tapes, chips, wired transmission, satellite transmission, internet transmission or download, magnetic and electronic storage media, digital and analog signals, etc. The computer-usable instructions can also be in various forms, including compiled code and uncompiled code.

[0087] As described herein, the term "real-time" refers to basic real-time feedback from the user device to the user. The term "real-time" as used herein may include short processing times (e.g., 100 ms to 1 second), and the term "real-time" may mean "approximately real-time" or "close to real-time."

[0088] First refer to Figure 1 This diagram illustrates a system illustration of a user equipment 100 for real-time interaction and guidance according to one or more embodiments. The user equipment 100 includes a communication unit 104, a processor unit 108, a memory unit 110, an I / O unit 112, a user interface engine 114, and a power supply unit 116. The user equipment 100 has a display 106, which may also be a user input device, such as a capacitive touch sensor integrated with the screen.

[0089] Processor unit 108 controls the operation of user equipment 100. Processor unit 108 can be any suitable processor, controller, or digital signal processor capable of providing sufficient processing power according to the configuration, purpose, and requirements of user equipment 100 known to those skilled in the art. For example, processor unit 108 can be a high-performance general-purpose processor. In alternative embodiments, processor unit 108 may include more than one processor, each configured to perform different dedicated tasks. In alternative embodiments, dedicated hardware may be used to provide some of the functions provided by processor unit 108. For example, processor unit 108 may include a standard processor, such as... processor, The processor or microcontroller.

[0090] The communication unit 104 may include wired or wireless connectivity. The communication unit 104 may include a radio that communicates using 4G, LTE, 5G, CDMA, GSM, GPRS, or Bluetooth protocols according to standards such as IEEE 802.11a, 802.11b, 802.11g, or 802.11n. The communication unit 104 can be used by the user equipment 100 to communicate with other devices or computers.

[0091] Processor unit 108 may also execute a user interface engine 114 for generating various user interfaces, some examples of which are shown and described herein, such as Figure 3 , 4 The interfaces shown in 5, 6, and 7. Optionally, in a user device such as... Figure 10 In the case of 1016 user devices, such as Figure 14 and 15 The user interface can be generated.

[0092] User interface engine 114 is configured to generate interfaces for receiving feedback inferences when a user performs physical activities, weightlifting, or other types of movements. The feedback inferences can be provided substantially in real-time along with video signals collected by the user equipment. The feedback inferences can be superimposed by user interface engine 114 onto the video signals received by I / O unit 112. Optionally, user interface engine 114 can provide a user interface for marking video samples. Various interfaces generated by user interface engine 114 are displayed to the user on display 106.

[0093] The display 106 can be an LED- or LCD-based display and can be a gesture-enabled touch-sensitive user input device.

[0094] I / O unit 112 may include at least one of a mouse, keyboard, touchscreen, touch wheel, trackpad, trackball, card reader, voice recognition software, etc., depending on the specific implementation of user device 100. In some cases, some of these components may be integrated with each other.

[0095] I / O unit 112 can further receive video signals from a video input device (such as a camera (not shown)) of user equipment 100. The camera can generate a video signal of the user when the user of the user equipment performs an action such as physical activity. The camera can be a CMOS active pixel image sensor, etc. The format of the video signal from the image input device can be provided to the video buffer 124 in 3GP format using an H.263 encoder.

[0096] The power supply unit 116 can be any suitable power source for supplying power to the user equipment 100, such as a power adapter or a rechargeable battery pack, depending on the implementation of the user equipment 100 known to those skilled in the art.

[0097] The memory unit 110 includes software code for implementing the operating system 120, program 122, video buffer 124, backbone network 126, global activity detection header 128, discrete event detection header 130, local activity detection header 132, and feedback engine 134.

[0098] Memory unit 110 may include RAM, ROM, one or more hard disk drives, one or more flash drives, or other suitable data storage elements, such as disk drives. Memory unit 110 is used to store operating system 120 and programs 122, which are well known to those skilled in the art. For example, operating system 120 provides various basic operating procedures for user device 100. For example, operating system 120 may be a mobile operating system, such as... Android operating system or The iOS operating system, or another operating system.

[0099] Program 122 includes various user programs that enable users to interact with user device 100 to perform various functions, such as, but not limited to, interacting with user device, recording video signals with a camera, and displaying information and notifications to users.

[0100] The backbone network 126, global activity detection header 128, discrete event detection header 130, and local activity detection header 132 can be used as sources from of or Play The software application is provided to user devices 100. Figure 12The backbone network 126, the global activity detection head 128, the discrete event detection head 130, and the local activity detection head 132 are described in more detail.

[0101] The video buffer 124 receives video signal data from the I / O unit 112 and stores it for use by the backbone network 126, the global activity detection head 128, the discrete event detection head 130, and the local activity detection head 132. The video buffer 124 can receive streaming video signal data from the camera device via the I / O unit 112, or it can receive video signal data stored on the storage device of the user equipment 100.

[0102] Buffer 124 allows for fast access to video signal data. Buffer 124 can have a fixed size and can use a first-in-first-out replacement strategy to replace the video data in buffer 124.

[0103] The backbone network 126 can be a machine learning model. The backbone network 126 can be pre-trained and can be provided in a software application provided to user device 100. For example, the backbone network 126 can be a neural network, such as a convolutional neural network. The convolutional neural network can be a three-dimensional neural network. The convolutional neural network can be a step-by-step convolutional neural network. The backbone network can be backbone network 1204 (see...). Figure 12 ).

[0104] The global activity detection header 128 can be a machine learning model. The global activity detection header 128 can be pre-trained and can be provided in a software application provided to user device 100. For example, the global activity detection header 128 can be a neural network, such as a convolutional neural network. The convolutional neural network can be a three-dimensional neural network. The convolutional neural network can be a step-through convolutional neural network. The global activity detection header 128 can be a global activity detection header 1208 (see...). Figure 12 ).

[0105] The discrete event detection header 130 can be a machine learning model. The discrete event detection header 130 can be pre-trained and can be provided in a software application provided to user device 100. For example, the discrete event detection header 130 can be a neural network, such as a convolutional neural network. The convolutional neural network can be a three-dimensional neural network. The convolutional neural network can be a step-through convolutional neural network. The discrete event detection header 130 can be a discrete event detection header 1210 (see...). Figure 12 ).

[0106] The local activity detection head 132 can be a machine learning model. The local activity detection head 132 can be pre-trained and can be provided in a software application provided to user device 100. For example, the local activity detection head 132 can be a neural network, such as a convolutional neural network. The convolutional neural network can be a three-dimensional neural network. The convolutional neural network can be a step-through convolutional neural network. The local activity detection head 132 can be a local activity detection head 1212 (see...). Figure 12 ).

[0107] Feedback engine 134 can collaborate with backbone network 126, global activity detection head 128, discrete event detection head 130, and local activity detection head 132 to generate user feedback inferences for actions performed by the user device 100's video input device.

[0108] Feedback engine 134 can execute Figure 2 The method is to determine feedback for the user based on the user's actions, given that the video input device of user equipment 100 is based on the user's actions.

[0109] Feedback engine 134 can generate feedback for users of user device 100, including audio, audiovisual, and visual feedback. The created feedback may include prompts to help users improve their physical activity, feedback on the form of their physical activity, exercise scores indicating the success rate of the user's exercise, estimates of calories burned by the user, and repetition counts of the user's activity. Furthermore, given the video input device connected to I / O unit 112, feedback engine 134 can provide feedback to multiple users.

[0110] Next reference Figure 2 Figure 200 illustrates a method for real-time interaction and guidance according to one or more embodiments.

[0111] Method 200 for real-time interaction and guidance may include outputting feedback inferences (including via audio or visual cues) to a user at a user device. To determine the feedback inferences, a video signal may be received, which may be processed by a feedback engine using a feedback model (see [link to relevant documentation]). Figure 12 ).

[0112] Method 200 can provide basic real-time feedback on an activity or exercise performed by a user. The feedback can be provided by an avatar or superimposed on the user's video signal, allowing them to see and correct their form of exercise. For example, the feedback could include information about the user's posture (allowing them to correct their posture based on the collected video signal), or feedback on the exercise based on the collected video signal. This can be useful for instruction, where a "coach" avatar provides real-time feedback on the form and other aspects of how an activity (e.g., exercise) is performed.

[0113] In 202, a feedback model is provided.

[0114] At 204, a video signal is received at the user equipment, the video signal comprising at least two video frames, the first video frame of the at least two video frames being captured before the second video frame of the at least two video frames.

[0115] At 206, an input layer for the feedback model is generated, which includes the at least two video frames.

[0116] At 208, based on the feedback model and the input layer, a feedback inference associated with the second video frame of the at least two video frames is determined.

[0117] In one or more embodiments, feedback inference can be used to output the user equipment's output device to the user.

[0118] In one or more embodiments, the feedback model may include a backbone network and at least one head network. Figure 12 The model architecture is described in more detail in the text.

[0119] In one or more embodiments, the backbone network may be a three-dimensional convolutional neural network.

[0120] In one or more embodiments, each of the at least one head network may be a neural network.

[0121] In one or more embodiments, the at least one head network may include a global activity detection head network for determining the activity classification of the video signal based on the layers of the backbone network; and the feedback inference may include the activity classification.

[0122] In one or more embodiments, the activity classification may include at least one selected from a group of exercise scores, calorie estimates, and exercise form feedback.

[0123] In one or more embodiments, the feedback inference may include a repetition score, which is determined based on the activity classification and the motion repetition count received from the discrete event detection head; and wherein the activity classification may include an exercise score.

[0124] In one or more embodiments, the exercise score may be a continuous value determined based on the inner product between a softmax output vector across multiple activity markers and a scalar reward value vector across multiple activity markers.

[0125] In one or more embodiments, the at least one head network may include a discrete event detection head network (see, for example...) Figure 12The discrete event detection head network is used to determine at least one event from the video signal based on the layers of the backbone network, each of the at least one event may include an event classification; and the feedback inference includes the at least one event.

[0126] In one or more embodiments, each of the at least one event may further include a timestamp corresponding to the video signal; and the at least one event corresponds to a portion of a user's exercise repetition.

[0127] In one or more embodiments, the feedback inference may include exercise repetition counts.

[0128] In one or more embodiments, the at least one head network may include a local activity detection head network (see...). Figure 12 The local activity detection head network is used to determine at least one bounding box from the video signal based on the layers of the backbone network and to determine the activity classification corresponding to each bounding box in the at least one bounding box; and the feedback inference may include the at least one bounding box and the activity classification corresponding to each bounding box in the at least one bounding box.

[0129] In one or more embodiments, the feedback inference may include an activity classification for one or more users, the bounding box corresponding to the one or more users.

[0130] Next reference Figure 3 The diagram 300 illustrates a scenario for real-time interaction and guidance according to one or more embodiments.

[0131] The scenario diagram 300 shown provides an example view of using a software application on a user device to assist in exercise activities. User 302 operates user device 304, which performs operations including... Figure 12 The software application of the feedback model described is illustrated. User equipment 304 captures video signals processed by the feedback model to generate feedback inferences, such as forming feedback 306. While user 302 is performing an activity, the associated feedback inference 306 is output to user 302, and this is done substantially in real time. The output can be in the form of audio prompts to user 302, messages from a virtual assistant or avatar, or subtitles superimposed on the video signal.

[0132] User device 304 may be provided by a fitness center, a fitness instructor, user 302, or other individuals, groups, or businesses. User device 304 may be used in a fitness center, at home, outdoors, or anywhere user 302 can use user device 304.

[0133] The software application of user device 304 can be used to provide feedback on the exercise completed by user 302. The exercise can be yoga, Pilates, weight training, bodyweight exercises, or other physical activities. When user 302 completes the exercise, the software application can acquire video signals from the video input device or camera of user device 304. The feedback provided can offer user 302 feedback on repetition count, set repetitions, positive reinforcement, available exercise corrections, form corrections, repetition speed, angles of body parts, stride or body placement width, exercise depth, or other types of feedback.

[0134] The software application can provide information to user 302 in the form of feedback to improve user 302's posture during exercise. The output may include corrections to limb placement, hold duration, body positioning, or other corrections that can only be obtained if the software application can detect user 302's body placement through video signals from user device 304.

[0135] The software application can provide feedback inference 306 to user 302 in the form of an avatar, virtual assistant, etc. The avatar can provide user 302 with appropriate body and limb placement, visual representations of exercise corrections to increase or decrease difficulty levels, or other visual representations. Feedback inference 306 may further include audio prompts for user 302.

[0136] The software application may provide feedback inference 306 to user 302 in the form of a video signal captured by the camera of user equipment 304. The video signal may have feedback inference 306 superimposed on the video signal, wherein feedback inference 306 includes one or more of the feedback options described above.

[0137] Next reference Figure 4 The diagram 400 illustrates a scene for real-time interaction and guidance, including a virtual avatar 408, according to one or more embodiments. Room 402 is shown as containing user 406 when a software application is used on user device 404, while user device 404 represents content output from user device 404 to user 406.

[0138] User 406 can perform operations on user equipment 404 including, for example Figure 12The software application of the feedback model described herein. User device 404 captures video signals processed by the feedback model to generate a virtual avatar 408. Virtual avatar 408 can be output to user 406 to guide user 406 through exercise routines, personal workouts, etc. Virtual avatar 408 can also provide user 406 with feedback such as repetition count, set number of repetitions, positive reinforcement, available exercise corrections, form corrections, repetition speed, angles of body parts, stride or body placement width, exercise depth, or other types of feedback. The feedback (not shown) provided to user 406 via user device 404 can be a visual or audio representation.

[0139] Next reference Figure 5 Figure 500 illustrates a user interface for real-time interaction and guidance according to one or more embodiments.

[0140] User 510 operates user interface 500, which includes functions such as... Figure 12 The software application of the feedback model described is shown. User interface 500 captures video signals processed by the feedback model via camera 506 and can generate feedback inference 514 and activity classification 512. The associated feedback inference 514 and activity classification 512 can be output to user 510 during and / or after the user 510 performs an activity. As shown, the output can be captions superimposed on the video signal.

[0141] The video signal can be processed by a global activity detection head and a discrete event detection head to generate feedback inference 514 and activity classification 512, respectively. Feedback inference may include repetition counts, stride length or body placement width, or other types of feedback as previously described. Activity classification may include formal feedback, fair exercise scores, and / or calorie estimation. The global activity detection head and the discrete event detection head can define the user's 510 motion to output a visual representation of the motion 516.

[0142] User interface 500 can provide user 510 with an output in the form of a video signal captured by camera 506 of user interface 500. The video signal may have feedback inference 514, activity classification 512 and / or visual representation of motion 516 superimposed on the video signal.

[0143] Next reference Figure 6 Figure 600 illustrates a user interface for real-time interaction and guidance according to one or more embodiments.

[0144] User 610 operates user interface 600, which performs operations including, for example... Figure 12The software application of the feedback model described is shown. User interface 600 captures video signals processed by the feedback model via camera 606 and can generate activity classification 612. Activity classification 612 can be output to user 610 during and / or after an activity performed by user 610. The output can be captions superimposed on the video signal.

[0145] The video signal can be processed by a discrete event detection head to generate an activity classification 612. The activity classification may include fair exercise scores, calorie estimates and / or formal feedback, such as the angle of body placement, repetition speed, or other types of feedback as described above.

[0146] The user interface 600 can provide the user 610 with an output in the form of a video signal captured by the camera 606 of the user interface 600. The video signal may have an activity classification 612 superimposed on the video signal.

[0147] Next reference Figure 7 Figure 700 illustrates another user interface diagram 700 for real-time interaction and guidance according to one or more embodiments.

[0148] User 710 operates user interface 700, which includes functions such as... Figure 12 The software application of the feedback model described is shown. User interface 700 captures video signals processed by the feedback model via camera 706 and can generate activity classification 712. Activity classification 712 can be output to user 710 during and / or after an activity performed by user 710. The output can be captions superimposed on the video signal.

[0149] The video signal can be processed by a discrete event detection head to generate an activity classification 712. The activity classification may include fair exercise scores, calorie estimates and / or form feedback, such as stride or body placement width, repetition speed, or other types of feedback as described above.

[0150] The user interface 700 can provide the user 710 with an output in the form of a video signal captured by the camera 706 of the user interface 700. The video signal may have an activity classification 712 superimposed on the video signal.

[0151] Next reference Figure 10 A system diagram 1000 for generating a feedback model according to one or more embodiments is shown. The system may have an auxiliary device 1002, a network 1004, a server 1006, and a user device 1016. Although three user devices 1016 are shown, there may be more than three user devices.

[0152] User equipment 1016 can typically correspond to... Figure 1The same type of user equipment, except that the downloaded software applications include a tagging engine instead of the backbone network 126, activity headers 128, 130, and 132, and feedback engine 134. The tagging engine can be used by the tagging user at user equipment 1016 (see [link to user equipment 1016]). Figure 10 A user device 1016 with a tagging engine can be referred to as a tagging device 1016. The tagging engine can be downloaded from app stores such as... Play or the of Server 1006 is operational. Figure 11 The method is to generate a feedback model based on labeled data from user device 1016.

[0153] Labeling users (not shown) can each operate user equipment 1016a to 1016c to label training data, including video sample data. User equipment 1016 communicates with server 1006 via a network. Users can send or receive training data, including video sample data and labeling data, to server 1006.

[0154] Network 1004 can be any network or network component capable of carrying data, including the Internet, Ethernet, fiber optic, satellite, mobile, wireless (e.g., Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network (LAN), wide area network (WAN), direct point-to-point connection, mobile data network (e.g., Universal Mobile Telecommunications System (UMTS), 3GPP Advanced Long Term Evolution (LTE Advanced), Microwave Access Global Interoperability (WiMAX), etc.) and others, including any combination of these.

[0155] Auxiliary device 1002 can be any two-way communication device capable of communicating with other devices, including mobile devices, such as those running... of Operating system or of Mobile devices with operating systems. Auxiliary device 1002 may allow the generation of management models at server 1006, and the delegation of training data, including video sample data, to user device 1016.

[0156] Each user device 1016 includes and executes software applications (such as a tagging engine) to participate in data tagging. The software application may be a web application for data tagging provided by server 1006, or it may be, for example, via an app store (such as...). of or App Applications installed on user equipment 1016.

[0157] As shown in the figure, user equipment 1016 is configured to communicate with server 1006 using network 1004. For example, server 1006 can provide web applications or application programming interfaces (APIs) for applications running on user equipment 1016.

[0158] Server 1006 is any networked computing device or system, including a processor and memory, and capable of communicating with a network (such as network 1004). Server 1006 may include one or more systems or devices communicatively coupled to each other. The computing device may be a personal computer, workstation, server, portable computer, or a combination thereof.

[0159] Server 1006 may include a database for storing video sample data and tag data received from the tagged user at user equipment 1016.

[0160] Databases can store user tagging information, video sample data, and other relevant information. Databases can be Structured Query Language (SQL) (such as PostgreSQL or MySQL), or more than just SQL (NoSQL) databases (such as MongoDB, or graph databases, etc.).

[0161] Next reference Figure 11 Figure 1100 illustrates a method for generating a feedback model according to one or more embodiments.

[0162] The generation of a feedback model may involve training a neural network. Training a neural network can use video clips labeled with activities or other information about the video content. For training, both "global" and "local" labels can be used. Global labels may contain information about multiple (or all) frames in the training video clip (e.g., activities taking place in the clip). Local labels may contain temporal information assigned to specific frames in the clip, such as the start or end of an activity.

[0163] 3D convolutions can be used in real-time applications (such as instruction). Each 3D convolution can become a "steppable" module during inference, where it can only be processed once per frame. During training, 3D convolutions can be applied in a "causal" manner. "Causal" means that information from the future cannot infiltrate the past in a convolutional neural network (see example...). Figure 13 (For more details). This may also involve training a discrete event detection head that needs to identify activities at precise time locations in a timely manner.

[0164] At 1102, multiple video samples are transmitted to multiple labeled users, each of the multiple video samples including video data, and each of the multiple labeled users receives at least two of the multiple video samples.

[0165] At 1104, multiple ranking responses are received from the multiple labeled users, each ranking response indicating a relative ranking selected by the corresponding labeled user from the at least two video samples transmitted to the corresponding labeled user based on ranking criteria.

[0166] In 1106, the ranking label for each video sample among the multiple video samples is determined based on the multiple ranking responses and the ranking criteria.

[0167] In 1108, multiple video samples are organized into multiple buckets based on the corresponding sorting label for each video sample.

[0168] In 1110, determine the classification label for each of the multiple buckets.

[0169] In 1112, a feedback model is generated based on the multiple buckets, the classification label of each corresponding bucket, and the video samples of each corresponding bucket.

[0170] In one or more embodiments, generating the feedback model may include applying gradient-based optimization to determine the feedback model.

[0171] In one or more embodiments, the feedback model may include at least one head network.

[0172] In one or more embodiments, each of the at least one head network may be a neural network.

[0173] In one or more embodiments, the method may further include determining that a sufficient number of ranking responses from the plurality of labeled users have been received.

[0174] In one or more embodiments, the ranking criteria may include at least one selected from a group of exercise speed, repetition, and range of motion.

[0175] In one or more embodiments, the ranking criteria may be associated with a specific type of physical exercise.

[0176] Method 1100 can be described as a pairwise labeling method. In many interactive applications, especially those related to guidance, training head recognition on labels corresponding to a linear order (or ranking) can be useful. For example, the network can provide outputs related to the speed of performing a workout. Another example is the recognition of range of motion during exercise. Similar to other types of labeling, labels corresponding to a linear order can be generated for a given video through manual labeling.

[0177] Pairwise tagging allows users to tag two videos (v1 and v2) at a time and provides only a relative judgment about their order. For example, in the case of velocity tagging, the tag might be equivalent to determining whether v1 > v2 (the velocity shown in the motion in video v1 is higher than the velocity shown in the motion in video v2), and vice versa. Given a sufficiently large number of such pairwise tags, sample datasets can be sized up. In practice, comparing each video with 10 other videos is often sufficient to produce rankings that are closely correlated with human judgment (see example...). Figure 16 Then, the individual video rankings can be grouped into any number of buckets, and a category label can be assigned to each bucket.

[0178] Next reference Figure 12 The diagram illustrates a model 1200 for determining feedback inference according to one or more embodiments. Model 1200 may be a neural network architecture and may receive two or more video frames 1202 as input from a video signal. Model 1200 has a backbone network 1204, which may preferably be a three-dimensional convolutional neural network generating motion features 1206, which are inputs to one or more detection heads, including a global activity detection head 1208, a discrete event detection head 1210, and a local activity detection head 1212.

[0179] Since most visual concepts in a video signal are related to each other, a shared neural network structure (such as shown in model 1200) can leverage commonalities through transfer learning and may include a shared backbone network 1204 and separate task-specific heads 1208, 1210, and 1212. Transfer learning may include determining motion features 1206 that can be used to extend the capabilities of model 1200, as the backbone network 1204 can be reused to process video signals upon receipt, and a new detection head can be further trained on top of it.

[0180] The backbone network 1204 receives at least one video frame 1202 from the video signal. The backbone network 1204 may be a shared backbone network on top of which multiple heads are jointly trained. The model 1200 may have an end-to-end trained architecture, with video frames including pixel data as input and activity tags as output (rather than using bounding boxes, pose estimation, or frame-by-frame analysis as intermediate representations). The backbone network 1204 may perform actions such as... Figure 13 The stepable convolution described in [the document].

[0181] Each head network 1208, 1210, and 1212 can be a neural network with one, two, or more fully connected layers.

[0182] The global activity detection head 1208 is connected to the layers of the backbone network 1204 and generates a fine-grained activity classification output 1214, which can be used to provide feedback 1220 to the user, including formal feedback inference, exercise score inference, and calorie estimation inference.

[0183] Feedback inference 1220 can be associated with a single output neuron of the global activity detection head 1208, and a threshold can be applied that triggers a corresponding formal feedback. In other cases, the softmax values ​​of multiple neurons can be summed to provide feedback.

[0184] Merging can occur when the classification output 1214 of the detection head 1208 is more granular than that required for a given feedback (in other words, when multiple neurons correspond to multiple different variants of the activity being performed).

[0185] One type of feedback inference 1220 is a training score. To fairly score users performing a specific exercise, the multi-class classification output 1214 of the feedback model 1208 can be calculated by computed softmax output vectors across each class (…). Figure 8 p in i ) and scalar reward value ( Figure 8 w i The inner product between the "reward" vector associated with each class is transformed into a single continuous value. More specifically, weights can be assigned to each activity tag associated with the exercise under consideration (see [link to relevant documentation]). Figure 8 Tag corresponding to the correct form (or higher strength) may receive a higher reward, while tag corresponding to the poor form may receive a lower reward. Thus, the inner product may be related to form, strength, etc.

[0186] refer to Figure 8 and Figure 9 The table illustrates this point in the context of scoring the formal accuracy and strength of "high knees," where w i Corresponding to the reward weight, and p i This corresponds to the classification output. Specifically, Figure 8 The explanation considers the overall reward in terms of form, speed, and intensity, while Figure 9 The explanation focuses on rewards that only consider the speed at which the exercise is performed.

[0187] Figure 8 and 9 The scoring method can be used to score metrics other than form, including metrics such as speed / intensity or instantaneous calorie expenditure rate.

[0188] The exercise score 1220 can further separate intensity and form scores (or scores for any other set of metrics) for multiple different aspects (such as form or intensity) of a user's fitness exercise performance. In this case, output neurons that are irrelevant to a particular aspect (such as form) can be removed from the softmax calculation (see example...). Figure 9 By doing so, the probability mass can be redistributed to other neurons related to the aspect under consideration, and the previously described fair scoring method can be used to obtain a score for the current particular aspect.

[0189] In another example of measurement, the calories burned by the user, 1220, can be estimated. The calorie estimate 1220 can be a special case of the scoring method described above, which can be used to dynamically estimate the calorie expenditure rate of a person exercising in front of a camera. In this case, each activity marker can be assigned a weight proportional to the task metabolic equivalent (MET) value of that activity (see references (4), (5)). Assuming the person's weight is known, this can be used to derive the instantaneous calorie expenditure rate.

[0190] The neural network head can be used to predict MET values ​​or calorie consumption from a given training dataset, where activities are labeled with this information. This allows the system to generalize to new activities during testing.

[0191] Return to reference Figure 12 In one or more embodiments, the at least one head network may include a discrete event detection head network 1210 for determining at least one event from the video signal based on layers of the backbone network, each of the at least one event including an event classification; and the feedback inference includes the at least one event.

[0192] The discrete event detection head 1210 can be used to perform event classification 1216 within a specific activity. For example, two such events might be the midpoint of an exercise (such as a push-up) and the end of a push-up repetition. Compared to the recognition head discussed above (which typically outputs a summary of activities performed consecutively over the last few seconds), the discrete event detection head can be trained to trigger a very short time interval (typically one frame) at the precise time location where an event occurs. This can be used to determine the temporal range of the action and, for example, to count the number of exercise repetitions performed so far 1222 in real time.

[0193] This also allows for behavioral policies that can execute a continuous sequence of actions in response to a sequence of observed inputs. An example application of behavioral policies is a gesture control system, where a video stream of gestures is converted into control signals, such as for controlling an entertainment system.

[0194] By combining discrete event counts with exercise scoring, this network can be used to provide users with repetition counts, where each count is weighted by an evaluation of the form / intensity / etc. of the repetitions performed. These weighted counts can be communicated to the user, for example, using a bar chart. Figure 5 The term is explained in the text. The metric derived from the combination of discrete event counts and exercise scores can be called the repetition score.

[0195] The local activity detection head 1212 can determine bounding boxes 1218 around the human body and face, and can predict activity markers 1224 for each bounding box, such as determining whether the face is "smiling" or "speaking" or whether the body is "jumping" or "dancing". The main motivation for this head is to allow systems and methods to perform perceptible interactions with multiple users simultaneously.

[0196] When multiple users are present in video frame 1202, it may be useful to perform spatial localization for each activity performed in the input video, rather than performing a single global activity prediction 1220. Performing spatial localization for each activity performed in the input video can also be used as an auxiliary task to make the global action classifier more robust to anomalous background conditions and user localization. Predicting bounding boxes 1218 to localize objects is a known image understanding task. Compared to image understanding, activity understanding in video can use 3D bounding boxes that are extended in both space and time. For training, the 3D bounding boxes can represent the localization as information as well as activity markers.

[0197] In addition to global activity predictions generated by the activity recognition head, localized heads can be used as separate heads in an action classifier architecture to generate local activity predictions from intermediate features. One approach to generating the 3D bounding boxes required for training is to apply existing object localizers of the images frame-by-frame to the training video. Annotations can be inferred without any further labeling of those videos known to show a single person performing an action. In this case, the known global action labels of the video can also be the activity labels for the bounding boxes.

[0198] Activity tags can be segmented by body parts (e.g., face, body, etc.) and can be attached to the corresponding bounding boxes (e.g., “smiling” and “jumping” tags will be attached to the face and body bounding boxes, respectively).

[0199] Let's refer to the following. Figure 12 and 13A stepable convolution diagram 1300 is shown for a model 1200 according to one or more embodiments, which is used to determine feedback inference. The stepable convolution diagram 1300 illustrates an output sequence and an input sequence. The input sequence may include inputs from various timestamps associated with received video frames. For example, frame 1306 shows the network inferring output 1302 based on the input at time t 1304, the input at time t-1 1308, and the input at time t-2 1310. Output 1302 is based on the stepable convolution of inputs 1310, 1308, and 1304. The input and output layers shown in the stepable convolution diagram 1300 may correspond to layers in a backbone network, or at least one detection head (see...). Figure 12 ).

[0200] Stepped convolutions can be used with model 1200 (see...) Figure 12 This is used to process video signals, such as streaming (real-time) video signals. In the case of receiving streaming video from a user device's video input device, the model can continuously update its predictions as new video frames are received. Compared to stateless conventional 3D convolutions, straddle convolutions can maintain internal state, which stores past information from the input video signal sequence (such as intermediate video frame representations, or input representations of the video frames themselves) for subsequent inference steps. For a kernel of size K (K in... Figure 13 The middle value is 3, i.e., the inference at time t 1302), and the last K-1 input elements (K-1 in the middle) Figure 13 The inputs at time steps t-1 (1308) and t-2 (1310) are required to perform the next inference step and therefore must be stored internally. Thus, the network's inputs represent the inputs included at the beginning. Once the new output is computed, the internal state needs to be updated to prepare for the next inference step. In the example below, this means storing the two inputs at time steps t-1 (1308) and t-2 (1304) in the internal state. The internal state can be buffer 124 (see [link to buffer 124]). Figure 1 ).

[0201] Various neural network architectures and layers can be used. 3D convolution can help ensure that motion patterns and other temporal aspects of the input video are processed efficiently. Decomposing 3D and / or 2D convolution into "outer products" and performing element-wise operations can help reduce the computational footprint.

[0202] In addition, other aspects of network architecture can be incorporated into Model 1200 (see Figure 12Other architectures can include those used for image (not video) processing, such as those described in references (6) and (10). For this purpose, the two-dimensional convolution can be “inflated” by adding a temporal dimension (see, for example, reference (7)). Finally, temporal and / or spatial strides can be used to reduce the computational footprint.

[0203] Next reference Figure 14 Figure 1400 illustrates a user interface for time stamping to generate a feedback model according to one or more embodiments.

[0204] User interface diagram 1400 provides an example view of user 1420 completing a physical exercise. The exercise can be yoga, Pilates, weight training, bodyweight exercises, or other physical activities. Figure 14 The example shown is an example of push-up exercises.

[0205] User 1420 can operate a software application that includes time stamps for generating a feedback model. The user equipment captures video signals processed by the feedback model to generate time stamps based on the movement and position of user 1420. The time stamps can be overlaid on video frames and output back to user 1420.

[0206] refer to Figure 14 In the example shown, the first video frame 1402 includes a user 1420 in a push-up position. A time-stamping interface can be used to assign event labels 1424, 1426, and 1428 to specific video frames. Event labels 1424, 1426, and 1428 can be assigned based on the movement and position of user 1420. The first video frame 1402 shows user 1420 at a position identified by the time-stamping interface as the "background" label 1424. The "background" label 1424 can be a default label provided to video frames where the time-stamping interface has not yet identified a specific event.

[0207] The time stamp interface in video frame 1404 has determined that user 1420 has completed the push-up repetition. The "High" label 1426 has been identified as an event marker for video frame 1404.

[0208] The time stamp interface in video frame 1410 has determined that user 1420 is at the midpoint of the push-up repetition. The "low bit" label 1428 has been identified as the event marker for video frame 1404.

[0209] Event classifier 1422 can be displayed on the user interface as a suggestion for marking upcoming events, which will be identified based on the user 1420's movement and position. Event classifier 1422 can improve over time as user 1420 provides more video signal input to the software application.

[0210] Figure 14 An example embodiment is shown, in which user 1420 performs a push-up exercise. In other embodiments, user 1420 may perform other exercises as mentioned above. In these other embodiments, the event markers for each video frame may correspond to the movement and body position of user 1420.

[0211] Time annotations that identify frame-by-frame events enable the learning of specific online behavioral strategies. In the context of fitness use cases, an example of an online behavioral strategy might be repetition counting, which could involve precisely identifying the start and end of an action. Tagging videos to obtain frame-by-frame labels can be time-consuming, as it requires examining each frame to find the presence of specific events. As shown in user interface 1400, the tagging process can be made more efficient by displaying suggested tagging procedures based on predictions from a neural network (which is iteratively trained to identify specific events). This interface can be used to quickly discover frames of interest in video samples.

[0212] Next reference Figure 15 Figure 1500 illustrates a user interface for pairwise tagging to generate a feedback model according to one or more embodiments.

[0213] Multiple video signals 1510 can be output to one or more tagging users via a tagging user interface 1502. Tagging users can compare the multiple video signals 1510 to provide multiple ranked responses based on specified criteria. Ranked responses can be transmitted from the tagging user's user device to a server. Specified criteria may include the speed at which the user is performing the exercise, the form of the exercise performed, the number of repetitions performed, the user's range of motion, or other criteria.

[0214] exist Figure 15 In the example shown, a tag user can compare two video signals 1510 and select a user based on a specified criterion. A tag user can indicate a relative ranking by selecting a first indicator 1508 or a second indicator 1512 using the tag user interface 1502, where each indicator corresponds to a specific user.

[0215] After indicating the relative ranking based on specified criteria, the tagging user can indicate that they have completed the requested task by selecting "Next" 1518. The tagging user may be required to provide ranking responses for any predetermined number of users. Figure 15In the illustrated embodiment, 25 ranking responses are required from the tagging user. The tagging user interface 1502 can provide an indication of the response number 1516 that the tagging user is currently completing and the percentage of ranking responses completed 1504. The tagging user can view and / or update previously completed ranking responses by selecting "Prev" 1514. Once the tagging user has completed the required number of ranking responses, the tagging user can select "Submit" 1506.

[0216] Next reference Figure 17 Figure 1700 illustrates a user interface including a virtual avatar for real-time interaction and guidance, according to one or more embodiments.

[0217] User equipment captures, for example Figure 12 The feedback model described herein processes the video signal to generate a virtual avatar. For the reasons mentioned above, the virtual avatar can be output to the user. As mentioned earlier, the virtual avatar can further provide feedback to the user.

[0218] The user interface can provide users with a view of the virtual avatar and a time dimension. The time dimension can be used to notify users of the remaining time of the workout, the total remaining time of the workout, the percentage of the workout completed, the percentage of the total workout completed, or other information related to workout timing.

[0219] The invention has been described herein by way of example only. Various modifications and variations may be made to these exemplary embodiments without departing from the spirit and scope of the invention, which is limited only by the appended claims.

[0220] References:

[0221] (1) Towards Situated Visual AI via End-to-End Learning on Video Clips, https: / / medium.com / twentybn / towards-situated-visual-ai-via-end-to-end-learning-on-video-clips-2832bd9d519f

[0222] (2) How We Construct a Virtual Being's Brain with Deep Learning, https: / / towardsdatascience.com / how-we-construct-a-virtual-beings-brain-with-deep-learning-8f8e5eafe3a9

[0223] (3) Putting the skeleton back in the closet, https: / / medium.com / twentybn / putting-the-skeleton-back-in-the-closet-1e57a677c865

[0224] (4) Metabolic equivalent of task, https: / / en.wikipedia.org / wiki / Metabolic_equivalent_of_task

[0225] (5) The Compendium of Physical Activities Tracking Guide, http: / / prevention.sph.sc.edu / tools / docs / documents_compendium.pdf

[0226] (6) Higher accuracy on vision models with EfficientNet-Lite, https: / / blog.tensorflow.org / 2020 / 03 / higher-accuracy-on-vision-models-with-efficientnet-lite.html

[0227] (7) Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, https: / / arxiv.org / abs / 1705.07750

[0228] (8) You Only Look Once: Unified, Real-Time Object Detection, https: / / arxiv.org / abs / 1506.02640

[0229] (9) YOLOv3: An Incremental Improvement, https: / / arxiv.org / abs / 1804.02767

[0230] (10) MobileNetV2: Inverted Residuals and Linear Bottlenecks, https: / / arxiv.org / abs / 1801.04381

[0231] (11) Depthwise separable convolutions for machine learning, https: / / eli.thegreenplace.net / 2018 / depthwise-separable-convolutions-for-machine-learning /

[0232] (12) TSM: Temporal Shift Module for Efficient Video Understanding, https: / / arxiv.org / abs / 1811.08383

[0233] (13) Jasper: An End-to-End Convolutional Neural Acoustic Model, https: / / arxiv.org / abs / 1904.03288.

Claims

1. A method for providing activity feedback to a user at a user device, the method comprising: A trained feedback model is provided, which has a neural network architecture and is generated at the server in the following way: Multiple video samples are transmitted to multiple annotation users, each of the multiple video samples including video data, and each of the multiple annotation users receives at least two video samples from the multiple video samples; Multiple ranking responses are received from the multiple labeled users, each ranking response indicating a relative ranking selected by the corresponding labeled user from the at least two video samples transmitted to the corresponding labeled user based on ranking criteria; The ranking label for each video sample among the plurality of video samples is determined based on the plurality of ranking responses and the ranking criteria; The multiple video samples are organized into multiple buckets based on the corresponding sorting tag of each video sample; Determine the classification label for each of the plurality of buckets; as well as The feedback model is generated based on the plurality of buckets, the classification label of each corresponding bucket, and the video sample of each corresponding bucket; A video signal is received at the user equipment, the video signal comprising at least two video frames, wherein a first video frame of the at least two video frames is captured before a second video frame of the at least two video frames; Generate the input layer of the feedback model that includes the at least two video frames; The feedback inference associated with the second video frame among the at least two video frames is determined based on the feedback model and the input layer; as well as While the user is performing an activity, the user device's output device is used to output the feedback inference to the user, wherein the output is in the form of an audio prompt to the user, a message from a virtual assistant or avatar, or subtitles superimposed on the video signal.

2. The method of claim 1, wherein the feedback model comprises a backbone network and at least one head network.

3. The method of claim 2, wherein the backbone network is a three-dimensional convolutional neural network.

4. The method of claim 3, wherein each of the at least one head network is a neural network.

5. The method of claim 4, wherein: The at least one head network includes a global activity detection head network, which is used to determine the activity classification of the video signal based on the layers of the backbone network; and The feedback inference includes the activity classification.

6. The method of claim 5, wherein the activity classification includes at least one selected from the group of exercise scores, calorie estimates, and exercise form feedback.

7. The method of claim 5, wherein: The feedback inference includes a repetition score, which is determined based on the activity classification and the training repetition count received from the discrete event detection head; and The activity classification includes exercise scores.

8. The method of claim 6, wherein the training score is a continuous value determined based on the inner product between the softmax output vector of a plurality of activity markers across the global activity detection head network and the scalar reward value vector across the plurality of activity markers.

9. The method of claim 4, wherein: The at least one head network includes a discrete event detection head network, which is used to determine at least one event from the video signal based on layers of the backbone network, each of the at least one event including an event classification; and The feedback inference includes at least one of the events.

10. The method of claim 9, wherein: Each of the at least one event further includes a timestamp corresponding to the video signal; and The at least one event corresponds to a portion of the user's exercise repetition.

11. The method of claim 10, wherein the feedback inference includes exercise repetition counting.

12. The method of claim 4, wherein: The at least one head network includes a local activity detection head network, which is used to determine at least one bounding box from the video signal based on the layers of the backbone network and to determine an activity classification corresponding to each bounding box in the at least one bounding box; and The feedback inference includes the at least one bounding box and the activity classification corresponding to each bounding box in the at least one bounding box.

13. The method of claim 12, wherein the feedback inference includes activity classification for one or more users, the at least one bounding box corresponding to the one or more users.

14. The method of claim 1, wherein the video signal is a video stream received from the video capture device of the user equipment, and the feedback inference is provided in near real-time upon receiving the video stream.

15. The method of claim 1, wherein the output device is at least one selected from the group consisting of audio output devices and display devices.

16. A system for providing activity feedback to a user at a user equipment, the system comprising: A server for generating a trained feedback model, the feedback model having a neural network architecture, wherein the server is configured to: Multiple video samples are transmitted to multiple annotation users, each of the multiple video samples including video data, and each of the multiple annotation users receives at least two video samples from the multiple video samples; Multiple ranking responses are received from the multiple labeled users, each ranking response indicating a relative ranking selected by the corresponding labeled user from the at least two video samples transmitted to the corresponding labeled user based on ranking criteria; The ranking label for each video sample among the plurality of video samples is determined based on the plurality of ranking responses and the ranking criteria; The multiple video samples are organized into multiple buckets based on the corresponding sorting tag of each video sample; Determine the classification label for each of the plurality of buckets; as well as The feedback model is generated based on the plurality of buckets, the classification label of each corresponding bucket, and the video sample of each corresponding bucket; The system further includes user equipment, the user equipment comprising: A memory, the memory including the trained feedback model; Output devices; A processor that communicates with the memory and the output device, wherein the processor is configured to; A video signal is received at the user equipment, the video signal comprising at least two video frames, wherein a first video frame of the at least two video frames is captured before a second video frame of the at least two video frames; Generate the input layer of the feedback model that includes the at least two video frames; Based on the feedback model and the input layer, a feedback inference is determined that is associated with the second video frame among the at least two video frames; and While the user is performing an activity, the output device is used to output the feedback inference to the user, wherein the output is in the form of an audio prompt to the user, a message from a virtual assistant or avatar, or subtitles superimposed on the video signal.

17. The system of claim 16, wherein: The feedback model includes a backbone network and at least one head network; The backbone network is a three-dimensional convolutional neural network; The at least one head network includes a global activity detection head network, which is used to determine the activity classification of the video signal based on the layers of the backbone network, wherein each of the at least one head network is a neural network; and The feedback inference includes the activity classification.

18. The system of claim 17, wherein the activity classification includes at least one selected from the group of exercise scores, calorie estimates, and exercise form feedback.

19. The system of claim 17, wherein: The feedback inference includes a repetition score, which is determined based on the activity classification and the training repetition count received from the discrete event detection head; and The activity classification includes exercise scores.

20. The system of claim 18, wherein the training score is a continuous value determined based on the inner product between the softmax output vector of a plurality of activity markers across the global activity detection head network and the scalar reward value vector across the plurality of activity markers.

21. The system of claim 17, wherein: The at least one head network includes a discrete event detection head network, which is used to determine at least one event from the video signal based on layers of the backbone network, each of the at least one event including an event classification; and The feedback inference includes at least one of the events.

22. The system of claim 20, wherein: Each of the at least one event further includes a timestamp corresponding to the video signal; and The at least one event corresponds to a portion of the user's exercise repetition.

23. The system of claim 21, wherein the feedback inference includes exercise repetition counting.

24. The system of claim 17, wherein: The at least one head network includes a local activity detection head network, which is used to determine at least one bounding box from the video signal based on the layers of the backbone network and to determine an activity classification corresponding to each bounding box in the at least one bounding box; and The feedback inference includes the at least one bounding box and the activity classification corresponding to each bounding box in the at least one bounding box.

25. The system of claim 23, wherein the feedback inference includes activity classification for one or more users, the at least one bounding box corresponding to the one or more users.

26. The system of claim 16, wherein the output device is at least one selected from the group consisting of audio output devices and display devices.

27. The system of claim 16, wherein the video signal is a video stream received from the video capture device of the user equipment, and the feedback inference is provided in near real-time upon receiving the video stream.

28. The system of claim 27, wherein the output device is an audio output device, and the feedback inference is an audio prompt to the user.

29. The system of claim 27, wherein the output device is a display device, and the feedback inference is provided as subtitles superimposed on the video signal.

Images