VIDEO ANNOTATION DEVICES, SYSTEMS AND METHODS

Abstract

DEVICES, SYSTEMS, AND METHODS FOR GAMING VIDEO ANNOTATION The invention relates to devices, systems, and methods for gamifying the video annotation process to maintain engagement in the annotation process and increase the quality and quantity of annotated dataset. Methods for gamifying the annotation of gestures in a video include determining gamified feedback based on input and presenting the gamified feedback to an operator to keep the operator engaged in the annotation process. Methods for gamifying the annotation of self-perceived gestures in a video by a subject are capable of being performed by the subject without requiring an operator, in which case the method captures an aspect of the subject's experience, and the gamification maintains or increases the subject's engagement in the annotation process.Annotated datasets are used to train artificial intelligence and machine learning systems for the automated detection and characterization of subjects' gestures and self-perception gestures. Figure for the abstract: none.

Description

Title of the invention: VIDEO ANNOTATION DEVICES, SYSTEMS AND METHODS FOR GAMING SUMMARY

[0001] This summary is proposed to present a selection of concepts in a simplified form, which are described in greater detail below in the detailed description. This summary is not intended to identify key features of the claimed subject matter, nor to be used as an aid in determining the scope of the claimed subject matter.

[0002] In some embodiments, a method for gamifying the annotation of gestures in a video of a subject performing a gesture is proposed. A video playback device displays a video of a subject performing a gesture. A processing device receives input from an operator corresponding to the gesture and annotates the gesture in the video based on the input. The processing device determines the gamified feedback based on the input. The processing device provides the gamified feedback to the operator to maintain or increase the operator's engagement in the annotation process.

[0003] In some embodiments, a method for gamifying the annotation of self-perception gestures in a video is proposed. A computing device represents a video of a subject that includes an image of a self-perception gesture by the subject and stores an annotation corresponding to the self-perception gesture in the video. The computing device determines the gamified feedback based on the self-perception gesture or the annotation and provides the gamified feedback to the subject to maintain or increase the subject's engagement.

[0004] In some embodiments, a computing device for gamifying video annotation is proposed. The computing device includes circuitry for representing a video of a subject performing a gesture; circuitry for receiving input that corresponds to the gesture and annotating the gesture in the video based on the input; circuitry for determining gamified feedback based on the input; and circuitry for providing the gamified feedback to maintain or increase engagement with the annotation. Description of the drawings

[0005] The foregoing aspects and many related advantages of this invention will be more readily appreciated as they are better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, in which:

[0006] [Fig-1] [Fig.1] shows a non-limiting example of an embodiment of a view of an overall video annotation process that includes a subject as observed by an operator, according to various aspects of this disclosure.

[0007] [Fig.2] Fig.2 shows a non-limiting example of an embodiment use of an AI / AA system for detecting the presence of a cotton wipe in a video that can be used by a subject to remove makeup in the video, according to various aspects of this disclosure.

[0008] [Fig. 3] [Fig. 3] shows a non-limiting example of an embodiment of a series gestures that are annotated by an operator and detected by an AI / AA system, as well as objective metrics that are used to characterize subjects, products and subject-product combinations, according to various aspects of this disclosure.

[0009] [Fig. 4] Figure 4 shows a non-limiting example of an embodiment of system and a method for gamifying video annotation, according to various aspects of this disclosure.

[0010] [Fig. 5] [Fig. 5] shows a non-limiting example of an embodiment of a graph and multi-gesture tracking table as performed by an AI / AA system trained with an annotated video dataset, according to various aspects of this disclosure.

[0011] [Fig. 6] Figure 6 shows a non-limiting example of an embodiment of method of gamifying gesture annotation in a video with an operator, according to various aspects of this disclosure.

[0012] [Fig.7] Fig.7 shows a non-limiting example of an embodiment of method of gamification of annotation of self-perception gestures in a video with a subject, according to various aspects of this disclosure.

[0013] [Fig. 8A] Figure 8A shows a non-limiting example of an embodiment of use of a trained AI / AA system to detect gestures in a video and produce results for automated video analysis, according to various aspects of this disclosure.

[0014] [Fig.8B] [Fig.8B] shows a non-limiting example of an embodiment of using a trained AI / AA system to predict outcomes for a subject, a product, or both, according to various aspects of this disclosure.

[0015] [Fig.9] Figure [Fig.9] shows a functional diagram which illustrates a non-limiting example of an embodiment of a computing device suitable for use as a computing device with embodiments of the disclosure, according to various aspects of this disclosure. Detailed description

[0016] Skin care product trials constitute an important part of the product development cycle. During product trials, a product is used by the Test subjects are used, and this usage is observed by operators. Live or recorded video footage of subjects trying the product is captured and analyzed by operators to determine if the product is satisfactory for the subject. The results of the footage are combined with feedback from the subjects in the form of questionnaires or question-and-answer surveys to understand how the product is received. This information can be used to guide a new phase of product development, in which the product is modified to try to improve its reception with a subject during a subsequent trial.

[0017] The review of video sequences by operators may seem monotonous or repetitive to them, and consequently, they may lose focus on the review process. In cases where the operator is responsible for annotating a video for the purpose of compiling digital information related to product use, and / or for the purpose of training an AI / AA model, this loss of attention or interest can negatively impact the quality of the test results and the AI / AA model, thus limiting the usefulness of the study. Accordingly, the disclosure proposes approaches to maintain or increase an operator's attention during the video review and annotation of a subject's video.

[0018] In general terms, the disclosure proposes devices, systems, and methods for gamifying the video annotation process. Specific features of the disclosure promote sustained engagement in the annotation process and provide an increase in the quality and quantity of annotated dataset and, consequently, an increase in the quality of an artificial intelligence and / or machine learning (AI / ML) system trained with the dataset. The gamification methods may include use by an operator who needs to be kept engaged in the annotation process, and in addition to, or as an alternative to, the operator, the methods may include use by a subject who is using or testing a product.The approaches disclosed here improve the efficiency, productivity, and robustness of video tracking and annotation by keeping the operator (and / or subject) alert and motivated, thereby reducing errors and fatigue, and improving workflow with game-like feedback mechanisms.

[0019] In some embodiments, the approaches disclosed herein generate multiple training datasets for training AI / AA models to model self-perception gestures or to generate new image data to link to existing gesture and self-perception gesture data, so that the trained AI / AA systems automatically detect subjects' gestures and self-perception gestures. In some embodiments, an AI / AA system is used to predict trial outcomes for a given product-subject combination. For example, whether the subject is expected to like or dislike the product's characteristics, the subject's data, and the product's properties. In some embodiments, the predicted results are validated against the actual results, and any differences are used to further train the AI / AA system. In some embodiments, the AI / AA system is used to accelerate product trials in a commercial environment.

[0020] Product testing scenario operators are sometimes responsible for viewing and annotating videos of subjects using products. These operators may lose focus or attention while reviewing and annotating videos. Consequently, in general terms, the disclosure proposes a method for gamifying the annotation of gestures in a video to maintain or increase the operator's attention. The gestures played out in the video may include a subject using a product or interacting with themselves during the use, application, or removal of a product, such as a skincare product.

[0021] Referring now to [Fig. 1], an overview of a video annotation process that includes a subject as observed by an operator is shown. In a laboratory environment, a subject 11 sits in front of a housing 12 having a one-way mirror 13 and a camera that records gestures made by the subject 11. The one-way mirror 13 is, for example, a mirror that provides the perception of one-way transmission when one side of the mirror is brightly lit (i.e., the side of the mirror with the subject) and the other side is dark (i.e., the side of the mirror with the camera; within the housing 12). This allows viewing from the darkened side, but not vice versa. The operating principle of one-way mirrors is known in art.When the subject 11 performs gestures (e.g., applying makeup, applying mascara, and the like), the camera records or provides a live video feed to a computing device configured to display the video or live video feed of the subject within a user interface 14 for presentation to the operator. In some embodiments, the user interface 14 includes, among other features, a video of the subject and instructions for pressing keys or providing other forms of input when the subject performs corresponding gestures in the video or video feed. In some embodiments, the annotated video is evaluated for objective metrics associated with product use, including but not limited to total time, application time, and / or removal time.In some embodiments, the annotated video is used to train an AI / ML model for automated gesture detection to complement or replace the operator in annotating subject videos. Automated gesture detection can be implemented within a system. AI / AA trained and configured for visual observation and gesture detection.

[0022] A method includes displaying, with a video playback device (for example, a computing device), a video of a subject performing a gesture and receiving, with a computing device, input from an operator that corresponds to the gesture, and annotating the gesture in the video based on the input. The computing device determines gamified feedback based on the input and provides this gamified feedback to the operator to maintain or increase the operator's engagement in the annotation process. In some embodiments, the video playback device is the same device as the computing device, or alternatively, these devices are separate or distinct.

[0023] The disclosure also provides for a video or video stream of the operator when the operator views the subject's video, and the operator's video or video stream is analyzed by an AI / ML system to identify trends or correlations between operator characteristics and the quality and / or quality of annotated data produced by the operators. These trends or correlations can be used for various purposes, for example, to identify potential new operators or to evaluate, reward, or provide feedback to existing operators.As a non-limiting example, if a strong correlation is found between an observable characteristic of an operator, an observable characteristic of a subject, and the quality or quantity of data produced for that operator-subject combination, future operator recruitment efforts can then focus on matching potential operators with potential subjects based on one or more of these observable characteristics. In this way, the research process is managed more efficiently, and the quality and quantity of annotated datasets increase.

[0024] In some embodiments, the subjects perform any gesture or action in the video or video stream that is directly or indirectly related to a cosmetic product, including but not limited to makeup, skincare products such as cleansers, devices such as skin massagers and scrapers, and haircare products. In some embodiments, the subject applies a cosmetic product to the subject's body, removes the cosmetic product from the subject's body, or both. In some embodiments, the operator's input is particular or specific to the gesture performed by the subject.For example, the operator can press a first key when the subject performs a first gesture for a first type of annotation (e.g., a hand movement to apply makeup), and the operator can press a second key when the subject performs a second gesture for a second type of annotation (e.g., a hand movement to remove makeup with a cotton ball). The annotation of the... gesture in video can include the association, with the computing device, of the input to a portion of the video (e.g., image(s), time(s), frame(s) at which the gesture is performed.

[0025] In some embodiments, gamified feedback may take any of various forms, whether visual, audio, audiovisual, text, graphic, sound, image, video, or other media or content, in any format, or other feedback to the operator that provides a game-like experience. In some embodiments, gamified feedback includes, for example, the representation of time remaining in the video; the representation of a high score for the operator and / or a plurality of operators; and / or the representation of the operator leveling up or leveling down.

[0026] In some embodiments, gamified feedback can be determined by any of several methods. In some embodiments, gamified feedback is determined on the basis of operator performance; for example, in some embodiments, good performance provides the basis for positive feedback (e.g., a smiling face emoji, text displayed on the screen, "GOOD JOB!"), while poor performance provides the basis for negative feedback (e.g., a frowning face emoji, text displayed on the screen, "TRY AGAIN!"). Good performance may include performance that is timely and accurate with respect to the annotations, while poor performance may include performance that is not timely or is inaccurate with respect to the annotations.These determinations can be made against a set of annotations for a control video, for example, or can be determined for a test video as part of a comparison between the operator's performance and that of one or more other operators with the same video. While such a "good or bad" division of performance can be beneficial in many cases, in at least some embodiments, more complex or nuanced feedback can be implemented.

[0027] In some embodiments, gamification includes a plurality of operators competing with each other for a high score, for example. In some embodiments, determining gamified feedback involves comparing the input, or lack thereof, with input received from a second operator for comparison and representing positive or negative feedback for the operator based on the comparison. In some embodiments, a high scoreboard of historical and unprecedented high scores is implemented as gamified feedback to motivate an operator before they annotate a video and contribute to the historical dataset.

[0028] In some embodiments, gamification may include an operator competing against their own previous performance. In some embodiments, determining the gamified feedback includes comparing the input with previous input received from the operator during a previous viewing of the video for comparison and representing positive or negative feedback for the operator based on the comparison.

[0029] In general, gamification provides a game-like experience to the operator which increases the operator's engagement in the annotation process and increases the quality and quantity of annotated data to be used in AI / AA model training and product testing.

[0030] Subjects in product testing scenarios use products and provide feedback regarding their product experience. This feedback is typically limited to answers provided in response to questions posed by the operator. Subjects may lose interest in the product testing process or unintentionally provide incomplete answers, which can impact the test results and the quality of the conclusions that can be reliably drawn from the test.There is a need for subjective feedback from subjects that is not limited to a question-and-answer format and that can better reflect the subjects' actual experience of the product, optionally in a non-laboratory setting, for the participatory production of an annotation of subjects' gestures (e.g., hand, lip, eye, and facial movements, for example, during the application or removal of makeup) and self-perception gestures (e.g., facial expressions or outward expressions of thoughts, emotions, feelings, or opinions, for example, smiling). A "gesture" includes any gesture, while a "self-perception gesture" includes any gesture associated with a subject's self-perception.

[0031] A subject's experience with a makeup product may be motivated by their satisfaction with how they look when wearing the product, or by their satisfaction with how others express their perception of the subject when the subject wears the product. For example, the subject may smile or laugh if their experience with the makeup product is positive, and may frown or contort their face if their experience is negative or ambivalent. These self-perceived gestures are difficult to capture and quantify in a controlled laboratory environment, but they hold significant value for understanding what makes a product successful.Although these and other self-perception gestures can be performed in the laboratory or in the individual's daily life or in a home environment, there is a need for effective means to establish, maintain and increase the subject's engagement with the products. skin care for facilitated annotation for subject videos for training AI / AA models. This disclosure addresses this unmet need and proposes devices, systems, and methods that are performed by subjects anywhere as part of a multi-subject collaborative product testing process, optionally remotely from a controlled environment or specific laboratory.

[0032] Accordingly, in another aspect, the disclosure proposes a method for gamifying the annotation of self-perception gestures in a video. The method comprises displaying, with a computing device, a video of a subject performing a gesture, receiving, with the computing device, an image of the subject's self-perception gesture, and annotating the self-perception gesture in the video based on the image. The method further comprises determining, with the computing device, gamified feedback based on the self-perception gesture and providing, with the computing device, the gamified feedback to the subject to maintain or increase the subject's engagement in the annotation process.Subjects who perform well in the annotation process, for example those who achieve a high score, may be eligible to receive a gift or promotional reward as a token of appreciation for their contributions.

[0033] The actions performed by the subjects may include applying a cosmetic product to the subject's body, removing the cosmetic product from the subject's body, or both. In some embodiments, the gamified feedback is determined by any one of various processes and may take any of various forms, whether visual, audio, audiovisual, or otherwise, for the operator providing a game-like experience. In some embodiments, the gamified feedback includes, for example, feedback that is perceived as positive by the subject when the subject displays a positive self-perception gesture, such as a smile (for example, the subject smiles or annotates a smile, and the text on the screen displays "YOU LOOK GREAT!").Similarly, in some embodiments, gamified feedback includes feedback that is perceived by the subject as negative or encouraging when the subject exhibits a negative self-perception gesture, such as frowning (e.g., the subject frowns or notes a frown, and the on-screen text displays "TRY AGAIN!"). In at least some embodiments, gamified feedback can be provided by the subject's peers or through social connections, for example, using a social media platform or a virtual meeting space.

[0034] In certain embodiments, the self-perception gesture includes a facial expression, a facial contortion, a smile, a frown, a Facial movement, a remark, a vocalization, or a body movement. For example, the subject might generate a video recording or a live video stream of themselves applying makeup, and during this process, they might smile due to the effect the makeup application has on their attitude or self-perception. The video recording or live video stream could be generated in a home or remote environment, outside of a laboratory setting, or it could be generated within the laboratory setting.

[0035] In some embodiments, the method further comprises receiving, with the computing device, input from the subject corresponding to the self-perception gesture and annotating the self-perception gesture in the video based on the input. For example, in some embodiments, the subject presses a button on a smartphone screen that signals to the smartphone that the subject is happy or about to smile, and the subject may smile. The subject's smile can be annotated in the video or live stream following the button-press input. As another example, the subject may generate a more complex or nuanced facial expression, and the smartphone may prompt the subject to insert input to explain their feelings associated with this facial expression, and the device can thus annotate the facial expression based on the explanation or another input from the subject.In some embodiments, the subject's input is specific to the self-perception gesture and / or the annotation of the gesture in the video includes the association, using the computing device, of the input to a portion of the video at which the gesture is performed, resulting in an annotation associated with the video that indicates the time at which a particular gesture is performed.

[0036] In some embodiments, annotated datasets are used to train AI / AA models for the automated detection and characterization of subjects' gestures. Referring now to [Fig. 2], an application 21 of an AI / AA system is shown for detecting the presence of a cotton wipe in a video or video stream that can be used by a subject to remove makeup in the video or video stream. In some embodiments, an AI / AA system comprising an AI / AA model is used to detect the presence 22 of a cotton wipe when it is present 23 and / or to detect the absence 24 of the cotton wipe when it is absent 25. Automated gesture detection can replace or supplement operator gesture detection in the video or video stream, thereby improving the quality and quantity of experimental results obtained from the product testing process.

[0037] As shown in [Fig. 3], in certain embodiments, a variety of gestures are suitable for being annotated by an operator and detected by an AI / AA system, and objective metrics are used to characterize subjects, products, and subject-product combinations. Examples of gestures 31 include, but are not limited to, the dema Makeup removal and cleaning (e.g., with a cotton wipe), application to the eyes (e.g., with a mascara brush, eyelash curler, mascara pump, eyebrow product, pencil, eyeliner, eyeshadow, and similar), application to the lips (e.g., with a tapered tip, a soft applicator, a thick pencil, a thin pencil, and / or a lip gloss tube), application of skin cream and sunscreen (e.g., with a cream, lotion, finger application, and / or spray application), application of foundation (e.g., with a setting spray, powder, lotion, highlighter stick, and / or sculpting techniques), and interactions with new packaging and instructions (e.g., pumping, shaking, left-right symmetry, and similar).From these gestures, 31 a series of objective measures 32 can be deduced which are useful for objectively quantifying the subject's interactions with the product. Examples of metrics 32 include, but are not limited to, duration, frequency count, order of experience (e.g., which gestures were performed within a sequence of gestures), and style and symmetry (e.g., which gestures were performed on which side and the characteristics or metrics of these).

[0038] An example of a system 41 and a method for gamifying video annotation is shown in [Fig. 4]. In some embodiments, the system 41 includes a plurality of computing devices, for example, one or more smartphones, one or more desktop computers, one or more tablets, or one or more other types of computing devices (see also [Fig. 9] and the associated description). In some embodiments, the system 41 includes a software application or "app" 42, in the form of processor-executable instructions stored on a non-transient, computer-readable storage medium, or in the form of a circuit or processor circuitry configured for logical operations. The app 42 causes the system 41 to perform all or part of a method for gamifying video annotation when it is executed.In some embodiments, the app 42 includes, as a component thereof, or can interact with, the user interface 44, which can be viewed by the operator 49 during video annotation. In some embodiments, the subject 48 interacts with the app 42 indirectly through the operator 49 or directly through a computing device accessible to the subject 48, for example, a smartphone or tablet. In some embodiments, the app 42 executes any process disclosed herein, in whole or in part, in any order or sequence of steps. The system 41 organizes the training data 45 for the creation and / or training of an AI / AA model 46. In some embodiments, the AI / AA model 46 is managed and validated according to procedures known in the art and trained on training data and updated. the test on test data. In some embodiments, the predictions 43 made by the system 41 are compared with study results 47 to determine whether the predictions 43 are sufficiently accurate or whether the AI / AA model needs to be retrained or further trained.

[0039] An example of a graph and table 51 of multi-gesture tracking as performed by an AI / AA system trained with an annotated video dataset is shown in [Fig. 5]. In some embodiments, a graph includes time as the control (x) axis and a binary result as the result (y) axis, as shown. In this way, an annotation can indicate the presence or absence of one or more gestures during the video or video stream. This data can form the basis for metrics such as gesture durations.

[0040] An example of a method 61 for gamifying gesture annotation in a video with an operator is shown in [Fig. 6]. In block 62, a video playback device displays a video of a subject performing a gesture while using a product. In block 63, an operator provides input to a computing device that corresponds to the gesture. In block 64, the computing device annotates the action in the video based on the input to produce an annotated video dataset. In block 65, the computing device determines gamified feedback based on the input and / or the annotated video dataset, and in block 66, the computing device provides the gamified feedback to the operator. In block 67, when the operator views, hears, receives, or otherwise interacts with the gamified feedback, the operator increases engagement with the computing device (and in the annotation process).

[0041] An example of method 71 for gamifying the annotation of self-perception gestures in a video with a subject is shown in [Fig. 7]. In block 72, a video playback device showing a subject a recorded video or a live video stream of themselves, i.e., a self-video playback device (e.g., a computing device, a smartphone, a tablet, and the like), shows the subject's self-video to the subject while the subject uses a product. In some embodiments, the self-video playback device includes, for example, a smartphone or tablet in "photographic self-portrait" mode, in which a live video stream of the subject is generated while the live video stream is visible to the subject. In block 73, the subject provides input to a computing device, such as the self-video playback device, which corresponds to the action.In block 74, the computing device annotates the action in the video based on the input to produce an annotated video dataset. In block 75, the computing device determines gamified feedback based on the subject's input and / or the annotated video dataset, and in block 76 provides the gamified feedback to the subject. In block 77, when the subject views, hears, receives, or otherwise interacts with the gamified feedback, the subject's ability increases. engagement with the computing device (and in the annotation process).

[0042] Referring now to Figures 8A and 8B, an example of using a trained AI / AA system to detect gestures in a video and produce results for automated video analysis is shown (Fig. 8A), and an example of using a trained AI / AA system to predict outcomes for a subject, a product, or both (Fig. 8B). As shown in Fig. 8A, a computing device trains an AI / AA model with the annotated dataset, and the trained model detects an action performed by a subject using a product in a video to produce an outcome. Although any suitable type of machine learning model can be used, an artificial neural network (ANN) can be used in one embodiment. Furthermore, although any training technique can be used, gradient descent can be used in one embodiment.The computing device then correlates the result with the subject and / or the product to produce an outcome dataset that maps subject characteristics to product characteristics. As shown in [Fig. 8B], a computing device can train an AI / MA model with the outcome dataset to generate predictions for subject-product combinations based on subject and product characteristics. These predictions can better guide product development and testing by allowing R&D teams to focus on products predicted to be more likely to be successful with a given subject.

[0043] In yet another aspect, the disclosure proposes computing devices and systems configured to perform one or more disclosure processes, in whole or in part, in any order or sequence. In some embodiments, a computing device for gamifying a video annotation includes a processor and a non-transient, computer-readable storage medium on which instructions are stored that, when executed by the processor, configure the processor to perform a process. The device may also include a circuit configured to perform the process.The process performed by the computing device may include the representation of a video of a subject performing a gesture; the reception of an input that corresponds to the gesture and the annotation of the gesture in the video based on the input; the determination, with the computing device, of gamified feedback based on the input; and the provision, with the computing device, of the gamified feedback to maintain or increase engagement in the process.

[0044] Referring now to [Fig. 9], a functional diagram is shown, which illustrates an example of an embodiment of a computing device 91 suitable for use as a computing device or computing system with embodiments of the disclosure.

[0045] As used herein, the "computing system" means one or more computing devices that are configured to perform all or part of any disclosure process, in any order or sequence of steps, optionally in combination with one or more other computing devices that are configured to perform all or part of any disclosure process, in any order or sequence of steps. In at least some cases, a process may be performed by two or more computing devices that together form at least part of a computing system, and in such cases, the steps performed by a first computing device may be complementary to the steps performed by a second computing device. In other cases, a process may be performed by a computing device that forms at least part of a computing system.

[0046] As used herein, the "computing device" means a physical hardware computing device that is configured to perform all or part of any disclosure process, in any order or sequence of steps, optionally with human input.

[0047] Although many different types of computing devices have been discussed above, computing device example 91 describes various features common to many different types of computing devices. Although [Fig. 9] is described with reference to a computing device that is implemented as a device on a network, the description below is applicable to servers, personal computers, mobile phones, smartphones, tablets, embedded computing devices, and other devices that can be used to implement portions of the embodiments of this disclosure. Some embodiments of a computing device may be implemented in or include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other custom device.Furthermore, people with ordinary skills in art and other fields will recognize that the calculating device 91 can be any of the devices currently available or yet to be developed.

[0048] In its most basic configuration, the computing device 91 includes at least one processor 93 and a system memory 92 connected by a communication bus 96. Depending on the exact configuration and type of device, the system memory 92 may be volatile or non-volatile memory, such as read-only memory (“ROM”), random-access memory (“RAM”), EEPROM, flash memory, or similar memory technology. Those with ordinary knowledge of the art and others will recognize that the system memory 92 typically stores data and / or program modules that are immediately accessible to the processor 93. and / or being operated by the latter. In this respect, the processor 93 can serve as the computing center of the computing device 91 by taking charge of the execution of instructions.

[0049] A non-limiting example of instructions is a software program, such as software written and compiled as a MATLAB-specific executable file that reads a video file as input. The software example allows the operator to define each key corresponding to each gesture; for example, an operator can define the "a" key as being associated with the subject applying mascara, the "r" key as being associated with the subject removing mascara, and the "p" key as being associated with counting the number of cotton pads used during a makeup removal experiment.The software example can respond in real time to the pressing and releasing of computer keyboard keys in terms of the appearance and modification of text or graphic symbols, providing the result of keeping the operator alert and interested in the task to be performed in order to provide useful data for further video monitoring and annotation (e.g. time remaining, high score, and similar).

[0050] The performance of the sample software provides gamified real-time video annotation that can generate a single dataset containing multiple dimensions of a subject-product experience [for example, three dimensions such as 1) time spent applying mascara, 2) time spent removing mascara, 3) number of cotton pads]. The disclosed embodiments also provide more efficient operation of video product studies, as study operators have less work to do during the study. For example, it is no longer necessary to store and count cotton pads during makeup removal, as the cotton pad count can be done by observing the subject's video after the study is completed.In some embodiments, the software also provides more efficient study operation, meaning that more subjects are suitable for study in a workday, thus reducing costs for fewer study days, and it also provides the ability to increase the amount of data collected in the same time period. The software also provides a dramatic increase in the amount of structured, analyzable, and easily processed training data that can be extracted from videos of "subject experiences," for example, cleaning, applying makeup, removing makeup, brushing hair, and the like.

[0051] Although an example of software is provided as being written in MATLAB executable code, in some embodiments the software can be written in any programming language and executed accordingly on any appropriately configured computing device, as known in art.

[0052] As illustrated in more detail in [Fig. 9], the computing device 91 may include a network interface 95 comprising one or more components for communicating with other devices on a network. Embodiments of this disclosure may access basic services that employ the network interface 95 to perform communications using common network protocols. The network interface 95 may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, 2G, 3G, LTE, WiMAX, Bluetooth, Bluetooth Low Energy, and / or similar protocols. As will be appreciated by a person with ordinary skills in the art, the network interface 95 illustrated in [Fig. 9] may include a network interface 95 comprising one or more components for communicating with other devices on a network.9] may represent one or more wireless interfaces or physical communication interfaces described and illustrated above with respect to particular components of the computing device 91. .

[0053] In the example embodiment shown in [Fig. 9], the computing device 91 also includes a storage medium 94. However, it is possible to access services using a computing device that does not include means for data persistence on a local storage medium. Therefore, the storage medium 94 shown in [Fig. 9] is represented by a dashed line to indicate that the storage medium 94 is optional. In any case, the storage medium 94 may be volatile or non-volatile, removable or non-removable, implemented using any technology capable of storing information such as, but not limited to, a hard disk drive, an SSD, a CD-ROM, a DVD or any other disk storage medium, magnetic cassettes, magnetic tape, magnetic disk storage medium and / or the like.

[0054] Suitable implementations of computing devices that include a processor 93, system memory 92, a communication bus 96, a storage medium 94, and a network interface 95 are known and commercially available. For ease of illustration and because it is not important to understand the claimed subject matter, [Fig. 9] does not show some of the typical components of many computing devices. In this regard, the computing device 91 may include input devices, such as a keyboard, numeric keypad, mouse, microphone, touch input device, touchscreen, tablet, and / or the like. These input devices may be coupled to the computing device 91 by wired or wireless connections, including RF, infrared, serial, parallel, Bluetooth, Bluetooth Low Energy, USB, or other suitable connection protocols using wireless or physical connections.Similarly, the computing device 91 may also include output devices such as a display, speakers, a printer, and the like. Since these devices are well-known in the arts, they are not illustrated or described in further detail here. NON-LIMITING METHODS OF IMPLEMENTATION

[0055] Although general features of the disclosure are described and shown, and specific features of the disclosure are presented in the claims, the following non-limiting embodiments relate to features and combinations of features that are explicitly contemplated as forming part of the disclosure. The following non-limiting embodiments are modular and can be combined with each other in any number, order, or combination to form a new non-limiting embodiment, which can itself be combined with other non-limiting embodiments. For example, embodiment 1 can be combined with embodiment 2 and / or embodiment 3, which can be combined with embodiment 4, and so on.

[0056] Embodiment 1. A method for gamifying gesture annotation in a video of a subject performing a gesture, the method comprising: receiving, by a computing device, input from an operator viewing the video of the subject which corresponds to the gesture; storing, by the computing device, an annotation corresponding to the gesture in the video on the basis of the input; determining, by the computing device, gamified feedback on the basis of the input; and providing, by means of the computing device, gamified feedback to the operator to maintain or increase the operator's engagement.

[0057] Embodiment 2. Method of any other embodiment, in which the gesture includes at least one of the following: the application of a cosmetic product, the removal of a cosmetic product.

[0058] Embodiment 3. Method of any other embodiment, wherein the operator input identifies the gesture, and wherein the storage of the annotation corresponding to the gesture in the video on the basis of the input includes the association, with the computing device, of the input to a portion of the video at the level of which the gesture is performed.

[0059] Embodiment 4. Method of any other embodiment, further comprising at least one of: the representation of time remaining in the video; the representation of a high score for the operator and / or a plurality of operators; or the representation of a passage to a higher level or a passage to a lower level for the operator.

[0060] Embodiment 5. A method for any other embodiment, further comprising: comparing the input with other inputs for comparison and displaying positive or negative feedback to the operator based on the comparison; comparing the input with an input received from a second operator for comparison and displaying positive or negative feedback to the operator based on the comparison; or comparing input with a previous input received from the operator during a previous viewing of the video for comparison and the representation of positive or negative feedback for the operator based on the comparison.

[0061] Embodiment 6. Method for gamifying self-perception annotation in a video, the method comprising: the representation, by a computing device, of a video of a subject which includes an image of a self-perception gesture by the subject; the storage, by the computing device, of an annotation corresponding to the self-perception gesture in the video; the determination, with the computing device, of gamified feedback based on the self-perception gesture or the annotation; and the provision, with the computing device, of the gamified feedback to the subject to maintain or increase the subject's engagement.

[0062] Embodiment 7. A method of any other embodiment, wherein the subject applies a cosmetic product to a portion of the subject's body, removes the cosmetic product from the portion of the subject's body, or both, in the subject's video.

[0063] Embodiment 8. A method of any other embodiment, wherein the self-perception gesture includes at least one of a facial expression, a facial contortion, a smile, a frown, a facial movement, a remark, a vocalization or a body movement.

[0064] Embodiment 9. A method of any other embodiment, further comprising: receiving, by the computing device, an input from the subject which corresponds to the self-perception gesture; and generating, by the computing device, the annotation on the basis of the input, wherein the annotation identifies the self-perception gesture in the video.

[0065] Embodiment 10. Method of any other embodiment, in which the generation of the annotation identifying the self-perception gesture in the video on the basis of the input includes associating, with the computing device, the input with a portion of the video at the level of which the gesture is performed.

[0066] Embodiment 11. Computing device for gamifying a video annotation, the computing device comprising circuitry configured to perform a method, the method comprising: the representation of a video of a subject performing a gesture; the reception of an input that corresponds to the gesture and the annotation of the gesture in the video on the basis of the input; the determination, with the computing device, of gamified feedback on the basis of the input; and the provision, with the computing device, of the gamified feedback to maintain or increase engagement with the annotation.

[0067] Embodiment 12. Calculation device of any other embodiment, wherein the action includes the application of a cosmetic product to a portion of the subject's body, the removal of the cosmetic product from the portion of the subject's body, or both.

[0068] Embodiment 13. Computing device of any other embodiment, wherein the input is received from the subject or an operator and gamified feedback is provided to the subject or the operator.

[0069] Embodiment 14. Calculation device of any other embodiment, in which the method further comprises: the representation of a time remaining in the video; the representation of a high score for the operator and / or a plurality of operators; the representation of a passage to a higher level or a passage to a lower level for the operator; the comparison of the input, or a lack of input, with an input received from a second operator for a comparison and the representation of positive or negative feedback for the operator on the basis of the comparison; or the comparison of the input with a previous input received from the operator during a previous viewing of the video for a comparison and the representation of positive or negative feedback for the operator on the basis of the comparison.

[0070] Embodiment 15. Computing device of any other embodiment, wherein the method further comprises: receiving, with the computing device, an image of a self-perception gesture of the subject and annotating the self-perception gesture in the video on the basis of the image; wherein the self-perception gesture includes a facial expression, a facial contortion, a smile, a frown, a facial movement, a remark, a vocalization or a body movement.

[0071] Although illustrative embodiments have been shown and described, it will be appreciated that various changes can be made to them without departing from the spirit and scope of the invention.

Claims

Claims

1. A method of gamifying gesture annotation in a video of a subject performing a gesture, the method comprising: - receiving, by a computing device, an input from an operator viewing the video of the subject that corresponds to the gesture; - storing, by the computing device, an annotation corresponding to the gesture in the video based on the input; - determining, by the computing device, gamified feedback based on the input; and - providing, by the computing device, the gamified feedback to the operator to maintain or increase operator engagement.

2. The method of claim 1, wherein the action comprises at least one of applying a cosmetic product, removing a cosmetic product.

3. The method of claim 1, wherein the operator input identifies the gesture, and wherein storing the annotation corresponding to the gesture in the video based on the input comprises associating, with the computing device, the input with a portion of the video at which the gesture is performed.

4. The method of claim 1, further comprising at least one of: - representing the time remaining in the video; - representing a high score for the operator and / or a plurality of operators; or - representing a level-up or a level-down for the operator.

5. The method of claim 1, further comprising: - comparing the input with other inputs for comparison and representing positive or negative feedback to the operator based on the comparison; - comparing the input with an input received from a second operator for a comparison and representing positive or negative feedback to the operator based on the comparison; or - comparing the input with a previous input received from the operator during a previous viewing of the video for a comparison and representing positive or negative feedback to the operator based on the comparison.

6. A method of gamifying annotation of self-perception gestures in a video, the method comprising: - representing, by a computing device, a video of a subject that includes an image of a self-perception gesture by the subject; - storing, by the computing device, an annotation corresponding to the self-perception gesture in the video; - determining, with the computing device, gamified feedback based on the self-perception gesture or the annotation; and - providing, with the computing device, the gamified feedback to the subject to maintain or increase engagement of the subject.

7. The method of claim 6, wherein the subject applies a cosmetic product to a portion of the subject's body, removes the cosmetic product from the portion of the subject's body, or both, in the video of the subject.

8. The method of claim 6, wherein the self-perceiving gesture comprises at least one of a facial expression, a facial contortion, a smile, a frown, a facial movement, a remark, a vocalization, or a body movement.

9. The method of claim 6, further comprising: - receiving, by the computing device, an input from the subject that corresponds to the self-perceiving gesture; and - generating, by the computing device, the annotation based on the input, wherein the annotation identifies the gesture of self-perception in the video.

10. The method of claim 9, wherein generating the annotation identifying the self-perceived gesture in the video based on the input comprises associating, with the computing device, the input with a portion of the video at which the gesture is performed.

Images