Method and system for analyzing participation degree of subject in video

Abstract

The invention relates to a computer-implemented method and system for analyzing video. The method comprises the steps of: receiving, via a receiving module, data comprising the video displaying a series of images of the subject; extracting, via an extraction module, features related to the subject in the video, the extracting step comprising: extracting, via a first extraction module, a first feature comprising one or more visual characteristics from one or more images of the series of images constituting the video; and extracting, via a second extraction module, a second feature comprising one or more spatial and / or temporal characteristics from the video; and processing, via an analysis module, the extracted features to determine a score that is a scalar representative of the degree of participation of the subject.

Description

Technical Field

[0001] The present invention relates to a method and system for analyzing a video, and in particular, but not exclusively, the present invention relates to a method and system for analyzing the involvement level of a subject in a video. Background Art

[0002] In recent years, online video communications have become increasingly popular due to their convenience. In particular, the outbreak of COVID-19 has further increased the demand and application of online video communications, such as online conferencing, learning, and business meetings. However, the effectiveness of online meetings (such as online courses) is often questioned compared to traditional face-to-face courses, as it is believed that students generally have lower attention during online classes. In face-to-face classes, students benefit more directly from the interaction of physical classes, and instructors can more easily observe students' reactions, such as students' emotions, facial expressions, and body language to determine their level of engagement. These elements are either lacking or become more difficult in online classes, because many times, students may be muted and interactive learning is hindered. Therefore, detecting students' level of engagement during online courses can help improve learning efficiency.

[0003] Various engagement level identification systems have been developed, aiming to improve the user experience during online meetings and learning. However, it is well known that traditional systems have limitations, such as, but not limited to, poor label quality and data imbalance in machine training datasets, as well as inter-class and intra-class differences in the resulting engagement levels. For example, datasets such as DAiSEE (Gupta, A.; D'Cunha, A.; Awasthi, K.; and Balasubramanian, V. 2016. Daisee: Towards user engagement recognition in the wild. arXiv preprint arXiv: 1609.01885) and EngageWild (Kaur, A.; Mustafa, A.; Mehta, L.; and Dhall, A. 30 2018. Prediction and localization of student engagement in the wild. In 2018 Digital Image 2 Computing: Techniques and Applications (DICTA), 1–8. IEEE) classify engagement levels into four categories, namely highly disengaged (HD), disengaged (DE), engaged (EG), and highly engaged (HE). Category labels are ineffective because they only indicate the user's engagement level within four set ranges or categories. Students labeled with the same engagement level category still show different levels of engagement. For example, some students labeled as engaged (EG), when observed carefully, may actually be highly engaged (HE) compared to other students in the same category. Therefore, existing systems lack accuracy because the true level of engagement of students is inevitably generalized by the four categories.

[0004] Selecting features to analyze from videos is also crucial to determining students’ engagement levels during online courses. However, most existing engagement detection systems are limited to high-level features, such as students’ head and body positions. The benefit of analyzing high-level visual features is that it ignores background noise and / or other apparently irrelevant information from the video. However, focusing only on high-level features may result in the loss of many other useful information, such as students’ motion and behavior information, which cannot be gleaned from high-level features.

[0005] On the other hand, image and video analysis based on artificial intelligence (AI) has been used for engagement prediction. For example, Liao, Liang, and Pan (Liao, J.; Liang, Y.; and Pan, J. 2021. Deep facial spatiotemporal network for engagement prediction in online learning. Applied Intelligence, 51: 6609–6621) disclosed combining SENet and LSTM with a global attention layer to extract facial spatial features and temporal characteristics. Mehta et al. (Mehta, NK; Prasad, SS; Saurav, S.; Saini, R.; and Singh, S. 2022. Three-dimensional DenseNet self-attention neural network for automatic detection of student's engagement. Applied Intelligence, 52 (12): 13803–13823) combined 3D DenseNet and 3D self-attention modules to capture the global relationship between features. Selim, Elkabani, and Abdou (Selim, T.; Elkabani, I.; and Abdou, MA2022. Students engagement level detection in online 25e-learning using hybrid efficientnetb7 together withtcn,lstm,and bi-lstm.IEEE Access, 10:99573–99583) further utilized EfficientNetB7 and different temporal networks, such as LSTM, Bi-LSTM, and TCN. However, all these studies failed to demonstrate actual improvements in determining user engagement levels in the settings of online meetings or online learning.

[0006] Purpose of the Invention

[0007] It is an object of the present invention to alleviate or eliminate to some extent one or more problems associated with known systems and methods for engagement detection.

[0008] Another object of the present invention is to provide an improved system and method for analyzing the engagement level of a subject in a video.

[0009] Another object of the present invention is to provide an improved system and method for analyzing the engagement level of participants in an online course or conference.

[0010] The above objects are achieved by the combination of features of the independent claims; the dependent claims disclose further advantageous embodiments of the invention.

[0011] Those skilled in the art will derive other objects of the present invention from the following description. Therefore, the above-mentioned objects are not exhaustive and are only used to illustrate some of the many objects of the present invention. Summary of the invention

[0012] The present invention generally relates to a method and system for analyzing the engagement level of one or more subjects in a video based on an artificial intelligence (AI) computer implementation. The analysis can be performed in real time during a live video conference or online course. The video can also be a pre-recorded video showing an online conference or online course with participants. In particular, the present invention is suitable for analyzing multimodal features of one or more subjects or participants in a video. Features, such as high-level visual features, spatiotemporal characteristics, and audio features associated with the video can be extracted and analyzed based on one or more artificial intelligence machine learning algorithms. More preferably, other non-verbal communication data, such as text information including instant messages and chat room information, user reactions and responses, such as digital graphic icons entered by users during video conferences, can also be extracted and analyzed. The system of the present invention analyzes multimodal features via an artificial intelligence machine learning model to generate a score that reflects the engagement level of the participants in scalar form. Suggestions or prompts corresponding to the scores can also be generated and displayed to the user via the computer device of the user running the system. The prompt can be provided in the form of advice to help improve the user's engagement level, for example, "Please place yourself in the center of the screen!". Therefore, the present invention provides a useful and effective system and method for determining the user's engagement level in a video. The application of the present invention can also be extended to facilitate users to give speeches or demonstrations, wherein the demonstration video is captured or recorded in real time for system analysis.

[0013] In a first main aspect, the present invention provides a computer-implemented method for analyzing the engagement level of a subject in a video. The method includes receiving data of a video including a series of images showing a subject via a receiving module; extracting features related to the subject in the video via an extraction module, the extraction step including: extracting first features including one or more visual characteristics from one or more images in the series of images constituting the video via a first extraction module; and extracting second features including one or more spatial and / or temporal characteristics from the video via a second extraction module; and processing the extracted features via an analysis module to determine a score as a scalar representation of the subject's engagement level.

[0014] In a second main aspect, the present invention provides a system for analyzing the engagement level of a subject in a video implementing the method described in the first main aspect. The system comprises: a receiving module configured to receive data of a video including a series of images of a subject; an extraction module configured to extract features related to the subject shown in the video, the extraction module comprising: a first extraction module for extracting first features including one or more visual characteristics from one or more images in the series of images constituting the video; and a second extraction module for extracting second features including one or more spatial and / or temporal characteristics from the video; and a processing module adapted to process the extracted features to determine a score as a scalar representation of the engagement level of the subject.

[0015] In a third main aspect, the invention provides a non-transitory computer readable medium for storing machine readable instructions which, when executed on a processor, implement the method of the first main aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features of the present invention will be apparent from the following description of preferred embodiments provided by way of example only in conjunction with the accompanying drawings, which include:

[0017] Figure 1 A technical implementation of a system for analyzing the engagement level of a subject in a video according to an embodiment of the present invention is shown;

[0018] Figure 2 It is shown Figure 1 A schematic block diagram of an exemplary implementation of a system;

[0019] Figure 3 It is shown Figure 1 A flowchart of the data processing implemented; and

[0020] Figure 4 It is shown Figure 1 Flow chart of data processing of another implementation. DETAILED DESCRIPTION

[0021] The following description describes preferred embodiments by way of example only and is not limited to the combination of features necessary to implement the invention.

[0022] In this specification, references to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in the specification does not necessarily all refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive of other embodiments. Moreover, various features are described that may be exhibited by some embodiments but not others. Similarly, various requirements are described that may be requirements of some embodiments but not others.

[0023] It should be understood that the elements shown in the drawings can be implemented in various forms of hardware, software or a combination thereof. These elements can be implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, a memory and an input / output interface.

[0024] This specification illustrates the principles of the present invention. It will therefore be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the present invention and are included within its spirit and scope.

[0025] In addition, all statements citing the principles, aspects and embodiments of the present invention and specific examples thereof herein are intended to cover equivalents in structure and function. In addition, it is intended that such equivalents include currently known equivalents and equivalents developed in the future, that is, any element of the performance of the same function developed, regardless of structure.

[0026] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of systems and devices embodying the principles of the invention.

[0027] The functionality of the various elements shown in the drawings may be provided through the use of dedicated hardware as well as hardware capable of processing software in association with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, a single shared processor, or multiple individual processors (some of which may be shared). Moreover, the explicit use of the term "processor" should not be interpreted as referring exclusively to hardware capable of executing software, and may implicitly include, but is not limited to, digital signal processor ("DSP") hardware, read-only memory ("ROM") for storing software, random access memory ("RAM"), and non-volatile memory.

[0028] In the claims herein, any element expressed as a means for performing a particular function is intended to encompass any manner of performing that function, including, for example, a) a combination of circuit elements that perform that function, or b) any form of software, thus including firmware, microcode, or the like, in combination with appropriate circuitry to execute that software to perform that function. The invention as defined by such claims resides in the fact that the functions provided by the various described means are combined together in the manner required by the claims. It is therefore considered that any means that can provide those functions are equivalent to the means shown herein.

[0029] The present invention generally relates to an artificial intelligence (AI) based, computer-implemented method and system for analyzing the engagement of one or more subjects in a video. The analysis can be performed in real time during a live video conference or based on a pre-recorded video showing the subject. The video conference can be conducted via an existing software development kit (SDK) or platform for video communication. Specifically, the present invention analyzes multimodal features of the subject displayed in the video. Features, such as high-level visual features, spatiotemporal features, and audio features associated with the video can also be extracted and analyzed based on one or more artificial intelligence machine learning algorithms. Other non-verbal communication data, such as text messages including instant messages and chat room messages, user reactions and responses, such as digital graphic icons entered by users during the video conference, can also be extracted and analyzed. Based on the multimodal analysis, the system generates an engagement score to reflect the engagement of the participants in a scalar form. Suggestions, tips, or advice corresponding to the score can also be generated and displayed to the user via the operating system of the computer device to help improve the subject's engagement in the meeting.

[0030] Reference Figure 1, a simplified flow chart illustrating the technical implementation of the system 100 according to an embodiment of the present invention is shown. The system 100 can be implemented in any computer device, such as a desktop computer, a laptop computer, a tablet computer and / or a mobile phone with a program or application software installed (e.g., a software development kit (SDK) providing a video communication platform for online video conferencing). Preferably, the video communication platform allows users to create and conduct online video conferences or courses. Preferably, the video communication platform is configured with a function that supports real-time extraction of real-time data of participants. For example, when operating in collaboration with the video communication platform, the system 100 is capable of capturing real-time data from each participant, which may include video data, audio data, chat room messages and / or reactions of participants, etc. The system 100 may include or may be connected to a back-end server that processes an artificial intelligence (AI) algorithm based on the extracted data to analyze the user's participation level as a score in the form of a scalar. The analysis results will then be sent to be displayed on the user's device, showing the participation level as a score and optionally suggestions corresponding to the score for improving the user's participation level during the online meeting or online course.

[0031] Figure 2 Further illustrated is a system 100 implementing the concepts of the present invention. In this embodiment, the system 100 may include or be implemented at a computer processing device, such as a smart phone, tablet computer, laptop computer, personal computer (PC), or similar communication device, although any suitable data or signal processing device or system may also be utilized. The system 100 may be connected to a server and / or database 106 or the like via a communication network, although this is not required for implementation of the present invention. The communication network may include a wireless network, such as a wireless cellular network, a wired network, the Internet, or any combination of the foregoing networks.

[0032] The system 100 may include a plurality of functional blocks for performing its various functions. For example, the system 100 may include a receiving module 110 configured to receive data (e.g., video data) including a series of images or frames showing one or more subject persons. The receiving module 110 may include various data receivers, control elements, user interfaces, etc. Although the receiving module 110 is described as a receiver of signals or data, it should be understood that the functional block may be implemented as a transceiver that provides transmission and reception of signals or data, such as via an antenna 105 and / or another type of system input, although it should be understood that video data in the form of recorded online courses, meetings, and / or other video clips may be received or input to the receiving module 110 of the system 100 by appropriate means familiar to those skilled in the art. In a particular embodiment, the receiving module 110 may be configured to receive raw real-time video data in any known video stream format (e.g., YUV420 format). The receiving module 110 may then encode the video stream format into a video codec, such as an H.264 codec using an off-the-shelf encoding framework available for computer devices. The receiving module 110 can also receive raw real-time audio data in any commercial format (e.g., PCM format), and then encode it into an audio codec (e.g., AAC) using an off-the-shelf encoding framework available for computer devices. Preferably, the receiving module 110 can also receive real-time reaction data supported by the SDK, such as participant reactions, chat room messages, and / or other actions provided by participants.

[0033] The system 100 may further include an extraction module 120 for extracting features related to the subject person from the video. Preferably, the extraction module 120 includes a first extraction module 122 and a second extraction module 124. The first extraction module 122 is configured to extract a first feature including one or more visual characteristics from one or more images in a series of images constituting the video; and the second extraction module 124 is configured to extract a second feature including one or more spatial and / or temporal characteristics from the video.

[0034] More preferably, the system 100 may include an analysis module 150 adapted to process the extracted features and to determine a score as a scalar representation of the subject's determined engagement level. In the context of the present invention, the term "scalar" relates to an exact quantity described by magnitude, and when used in relation to the subject's engagement level, it refers to an engagement level score provided in an exact quantity. This is in contrast to the prior art, in which the results from the prediction of the subject's engagement level are specified as predetermined categories or classes, which means a generalized, diverse range of engagement levels. The precisely determined engagement level score in the form of a scalar of the present invention may preferably be output or displayed by the display module 180, and in one embodiment, the display module 180 may also be configured to output or display a suggestion or advice corresponding to the determined score to improve the subject's engagement level, for example by presenting a prompt sentence "Please place yourself in the center of the screen!" as a push notification placed at the device display screen. In one embodiment, the extraction module 120 and / or the analysis module 150 may be arranged at a server 106, such as a cloud server, a backend server, or the like.

[0035] although Figure 2The receiving module 110, the extraction module 120, the analysis module 150, and the display module 160 in the embodiment are deployed as functional components of the system 100, but according to the concept of the present invention, it is not limited to such a deployment configuration. For example, one or more of these modules can be deployed as individual functional blocks that are different from the system 100 but connected to the system 100. One or more of the receiving module 110, the extraction module 120, the analysis module 150, and the display module 160 can be implemented separately using logic circuits and / or executable code / machine-readable instructions stored in the memory 140 of the system 100 for performing the functions described herein. For example, the executable code / machine-readable instructions can be stored in one or more memories 140, such as random access memory (RAM), read-only memory (ROM), flash memory, magnetic storage, optical storage, or the like, suitable for storing one or more instruction sets, such as application software, firmware, operating system, applet and / or the like, data such as configuration parameters, operating parameters and / or thresholds, collected data, processed data and / or the like, etc. The one or more memories 140 may include processor-readable memory for use with respect to the one or more processors 160, which may be operable to execute code segments of any one or more of the functional modules described above, and / or utilize data provided thereby to perform the functions of the system 100 as described herein. Additionally or alternatively, one or more of the receiving module 110, the extracting module 120, the analyzing module 150, and the display module 160 may include one or more special-purpose processors configured to perform the functions of the system 100 as described herein, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), or the like.

[0036] Preferably, the first features extracted by the first extraction module 122 may include or relate to high-level visual features or characteristics of the subject person during the online video conference. The high-level visual features may include, but are not limited to, visual characteristics that can be derived from one or more two-dimensional images showing the subject. For example, the first features may include visual characteristics of one or more facial and / or body characteristics of the subject displayed in one or more images or frames constituting the video. In one embodiment, the one or more facial and / or body characteristics may include facial movements, head position, and / or gaze direction of the subject. In one embodiment, facial movements may include facial movements based on the Facial Action Coding System (FACS) ( CH (1969). Man's Face And Mimic Language; Ekman P, Friesen WV, Hager JC 15 (2002) Facial Action Coding System: The Manual on CD ROM. Salt Lake City: A Human Face) One or more facial action units. In a specific embodiment, high-level features can be extracted by OpenFace 2.2.0 (Baltrusaitis, T.; Zadeh, A.; Lim, YC; and Morency, L.-P. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), 59–66. IEEE). High-level features are useful because they provide general but important information for predicting the level of engagement. Features of facial movements, head positions and / or gaze directions may also include other engagement-related features that help determine the level of engagement, such as nodding, yawning, lowering the head, etc. Another benefit of utilizing high-level features in engagement analysis is that it can reduce the interference of noisy data, such as the background and clothes of the participants shown in the video.

[0037] The second feature extracted by the second extraction module 124 may include one or more spatial and / or temporal characteristics obtained from the video. Specifically, the spatial and / or temporal characteristics of the second feature may include, for example, features learned by a neural network during action classification training (Carreira, J.; and Zisserman, A. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings 30 of the IEEE Conference on Computer Vision and Pattern Recognition, 6299–6308). In one embodiment, the spatial and / or temporal characteristics of the second feature may be represented by vector data, and more preferably, by an expanded three-dimensional (I3D) vector.

[0038] Although high-level features can capture the state of a single frame, the spatiotemporal information of the participants will not be derived from the high-level data. Therefore, if the system relies solely on extracting and analyzing high-level features from the video, important information related to the subject's level of participation may be missed. In addition, high-level features may not fully capture facial expressions. Therefore, the present invention utilizes an expanded 3D network (I3D) (Carreira, J.; and Zisserman, A. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6299–6308) to compensate for the ignored information and extract further spatial and / or temporal features from the video. I3D is a three-dimensional (3D) video classification network that contains a 3D convolutional network and optical flow to extract spatiotemporal information.

[0039] In another embodiment, the extraction module 120 may also include a third extraction module 126 for extracting a third feature including one or more audio features associated with the data of the video. The audio features of the third feature may include one or more of the volume, pitch, speech length, speech duration, speech content and / or emotion of the speech represented by the audio feature or the audio feature. For example, Parselmouth (Jadoul, Y.; Thompson, B.; and DeBoer, B. 2018. Introducing parselmouth: A python interface to praat. 20 Journal of Phonetics, 71: 1–15); McFee, Brian, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. "librosa: Audio and music signal analysis in python." In Proceedings of the 14th python in science conference, pp. 18-25. 2015 can be used to extract audio features. Zoom Live Transcript can extract speech content as input.

[0040] The extraction module 120 may also include a fourth extraction module 128 for extracting a fourth feature including input from a user. The user input may include non-verbal feedback, such as text data such as instant messages and / or chat room messages, graphic data and / or image data such as conference reactions and / or digital graphic icons in the form of "Emojis" sent between different users participating in the online conference. The fourth feature may also include one or more of the frequency of input from the user, the duration between individual inputs from the user, the reaction of one user to other users, and / or the degree of responsiveness of the user. In a specific embodiment, the user input of the fourth feature can be extracted by an existing video communication software or platform that supports user reaction extraction functions, such as but not limited to Zoom. TM , Google Meet TM , Microsoft Teams TM , Skype TM wait.

[0041] Figure 3 A flow chart showing a display processing model according to an embodiment of the present invention is shown.

[0042] Preferably, the multimodal features extracted by the extraction module 120 will be processed by the analysis module 150. For example, the extracted first features including high-level visual features from one or more images or frames forming a video will be analyzed by the first analysis module 152 of the analysis module 150; and the spatial and / or temporal features of the second features will be analyzed by the second analysis module 154 of the analysis module 150. More preferably, the analysis of the first and / or second analysis modules 152, 154 is based on one or more artificial intelligence machine learning algorithms. In a specific embodiment, the first analysis module 152 is preferably adapted to process data from the first features based on a temporal convolutional network (TCN); the second analysis module 154 is preferably adapted to process data from the second features based on a multi-layer perceptron neural network (MLP).

[0043] In one embodiment, the high-level first features extracted from the video can be analyzed by segmenting the video into T segments of equal length. For videos less than 10 seconds in length, the video is played repeatedly until it contains more than 250 frames. In order to obtain the first high-level features from each segment, the minimum, maximum, and variance of the high-level features are derived according to the following formulas:

[0044] s∈R D×T

[0045] Then, they are combined into a feature vector, represented as

[0046] ps∈R 3D×T

[0047] Then, we can use the temporal convolutional network (TCN) (Bai, S.; Kolter, JZ; and Koltun, V. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271) to capture the spatial and / or temporal patterns from the feature vector ps. The output of TCN can be expressed as

[0048] X TCN ∈R C×T’

[0049] It is found that TCN outperforms other commonly used engagement prediction models and, in particular, the known limitations of data imbalance can be significantly improved.

[0050] To solve X TCN The different discrimination problems at each time step in X are solved by using the attention mechanism to filter TCN The attention weights may be generated based on the processed data from the first feature and the processed data from the second feature. The generated attention weights are further processed using the data from the first feature to determine a score that is a scalar representation of the subject's level of engagement.

[0051] We can use the features X extracted from the I3D module I3D ∈R d To calculate the attention score, the feature contains important spatial and / or temporal characteristics that help the filtering process. This operation can be described by the following equation:

[0052] X attn = Softmax ( MLP1 (X I3D ) × X TCN ) (1)

[0053] X HL = X TCN × X attn (2)

[0054] Where MLP1 consists of two fully connected (FC) layers and a dropout layer. The last FC layer has C hidden units, X attn ∈R T Act as X TCN and X HL ∈R C The attention score.

[0055] As mentioned before, the features extracted by the I3D module can include information that is ignored in high-level features. Therefore, in addition to the high-level features X HL In addition, we also compare them with X I3D Combining to create combined feature representations for downstream predictions. This combination is implemented via merging, resulting in:

[0056] X = CONCAT ( MLP2 (X I3D ), X HL ) (3)

[0057] Here, MLP2 consists of two fully connected (FC) layers with a rectified linear unit (ReLU) layer in between. The last FC layer has C hidden units. By merging the output of MLP2 with XHL, the combined feature representation X∈R 2C . Subsequently, the model prediction based on X can be formulated as:

[0058] s = MLP3 (NORM(X)) (4)

[0059] MLP3 is implemented as a normalized FC layer containing an unbiased normalized weight vector. When X is normalized, s is a scalar in the range [-1,1]. Higher values ​​indicate a high degree of engagement of the subject. Subsequently, a threshold (-0.5, 0, 0.5) can be used to classify the data into one of four levels of engagement, such as but not limited to highly disengaged (HD), disengaged (DE), engaged (EG), and highly engaged (HE).

[0060] Considering that engagement levels are not limited to independent categories but exist as precise units within a range, it is preferred to predict engagement levels as scalars, which allows for continuous representation of engagement levels. It enables the model to capture subtle variations and nuances in engagement levels that may vary within the same engagement level category. This emphasis on ordinal relationships and continuous representation allows the present invention to better handle the complexity of engagement level prediction and provide more informative and nuanced results.

[0061] The analysis module 150 may further include a third analysis module 156 configured to analyze the extracted third audio features. In one embodiment, the pre-trained BERT model (Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 20arXiv: 1810.0480) may be used to extract audio features from the speech sp. In one embodiment, the metadata of volume and pitch are selected and processed by the following operations:

[0062] X sp =FC(BERT(sp)) (5)

[0063] X aud =[L,Hv,Lv,Hp,Lp,std v ,std p ] (6)

[0064] X′=CONCAT(X,X sp ,X aud ) (7)

[0065] s=MLP3(Norm(X′)) (8)

[0066] Where X sp ∈R 768 , X aud ∈R 7 Including voice length L, high volume percentage Hv, low volume percentage Lv, high pitch percentage Hp, low pitch percentage Lp and volume std v and tone std p These multimodal features (denoted as X′) are then fed into the downstream head of the model, which is the same as when only visual features are considered.

[0067] The analysis module 150 may also include a fourth analysis module 158 for processing and analyzing the extracted fourth features based on the user input. For example, for text data that can be extracted from a user chat room, features such as sentiment and text length. The "sentiment" feature is a sentiment polarity score, such as a positive value for positive sentiment and a negative value for negative sentiment, which can be obtained using the textblob library ( https: / / textblob.readthedocs.io / en / dev / ). The “text length” feature is the number of words entered. Other non-text interactions, such as the use of emojis, reactions, and sharing of visuals, are represented using a binary feature vector whose length is the number of interactions that can be performed. Each vector element is placed with a “1” if the corresponding interaction is performed and a “0” otherwise.

[0068] As mentioned earlier, subjects within the same category may have different behaviors, and therefore they may show similar but not identical levels of engagement. In order to effectively account for intra-category differences when designing the loss criterion, it is crucial to avoid imposing common ground facts for each category. However, to train the model using mean squared loss (MSE Loss), a common ground fact needs to be set for each category. The specific design of the present invention is used to handle the complexity of intra-category variations and ordinal relationships, thereby providing more accurate and robust learning for engagement prediction.

[0069] Also refer to Figure 4 , the present invention introduces edge ranking loss as a means of training the model using relative evaluation rather than relying solely on absolute evaluation of individual data points. This approach makes it easier to learn from comparisons between different data combinations, which can promote better representative learning of minority categories in the case of data imbalance. The framework of the present invention includes two main components: a sampling process, which involves a momentum encoder and a score pool, and an edge ranking loss, which uses the results obtained from the sampling process to calculate the final loss. Specifically, the present invention utilizes the sampling mechanism of the MoCo method (He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR)) to generate engagement scores for the edge ranking loss function. The idea behind this approach is to maintain a score pool of pre-predicted engagement scores generated by momentum encoders of different categories as a reference point for evaluating the applicability of the predicted scores to other data. The momentum encoder has the same structure as the trained model and is initialized with the same weights. In addition, the momentum encoder is updated at each iteration by retaining 99.9% of the current weights and incorporating 0.1% of the model weights. This gradual update of the momentum encoder ensures that the generated scores remain relatively consistent and do not fluctuate excessively after each iteration.

[0070] In each training iteration, the momentum encoder processes a mini-batch of data, whose size is denoted as |B|, generating scores for each data point. These scores, along with the feature embeddings before the prediction head and their respective ground truth labels, are combined into triplets. These triplets are then stored in a score pool that operates on a first-in, first-out (FIFO) principle. The score pool has a predetermined length of |P|, and it is initially filled with data triplets from four different levels of participation, randomly shuffled to ensure a diverse mix of examples. Similar to the basic principle behind the momentum encoder update, the present invention updates the score pool as a queue by removing the most |B| outdated data triplets during each iteration to ensure a more stable transition.

[0071] The edge ranking loss uses the scores S generated by the trained model t , combined with the score pool P, to calculate the loss of updating the training model weights. The marginal ranking loss relative to a batch of training samples can be formulated as:

[0072]

[0073]

[0074]

[0075] Where B is represented as a training batch and data d1, whose label l1 represents a training sample of B, and P is a score pool, where each element is a triplet of the ground truth label, the predicted score and the corresponding high-dimensional feature (l2, s2, e2). dif f is determined by considering two factors: the difference between the labels of two data points and the cosine similarity between their embeddings. e1 is the high-dimensional feature of d1 before training the prediction head of the model, while e2 is the representative of d2 previously obtained from the momentum encoder and saved in the score pool. A larger margin is imposed when the label difference between two data points is large or the feature vectors of two data points from different classes are closer to each other. For example, assuming d1 is HE and d2 is DE, the loss will be calculated as

[0076] M2-(model(d1)-s2),

[0077] Among them, if M2>(model(d1)-s2), the model will be penalized by the loss.

[0078] The basic concept behind the margin ranking loss is that the engagement of a subject is expected to be predicted by a score higher than that of a less engaged subject and lower than that of a more engaged subject, where the score difference should be larger than the difference suggested by the label difference. In addition, cosine similarity is used to flexibly calculate the threshold. This method can increase the penalty when two data from different categories have similar representatives, and reduce the penalty when the correlation of the scores of the two data is well predicted.

[0079] The efficacy of the present invention based on training models of various combinations of different features has been evaluated. In the experiments, a combination of first features including high-level features, second features of spatiotemporal features, and third features of audio features was used. Only visual segments were used to make predictions for video segments without speech, while the full model was used to predict video segments containing speech.

[0080]

[0081] Table 1: Accuracy using different feature combinations, where Openface is used to extract and analyze the first feature, I3D is used to extract and analyze the second feature, and Audio refers to the third audio feature.

[0082]

[0083] Table 2: Comparison of models with and without audio features, where the first feature is extracted and analyzed using Openface, the second feature is extracted and analyzed using I3D, and Audio refers to the third audio feature.

[0084] It is revealed that the first feature including high-level characteristics and the second feature represented by the I3D vector together improve the performance of the present invention compared to the prior art that only extracts single modality features. It is further demonstrated that the method of combining the I3D vector with the high-level features of the present invention exhibits superior performance compared to the prior art by simply merging the two features. Features at different levels are found to be complementary and provide useful information for participation prediction.

[0085] In addition, the results show that multimodal analysis is beneficial for engagement prediction. In Table 2, for data containing speech, the models trained on visual and audio features show significantly better performance in terms of accuracy and average accuracy.

[0086]

[0087] Table 3: Comparison of different methods of combining the first high-level features and the second spatiotemporal features

[0088] In another aspect, the present invention also relates to a non-transitory computer-readable medium for storing machine-readable instructions which, when executed on a processor, implement the steps of the method as described above.

[0089] The above-mentioned system and various modules can be implemented at least in part by software. It will be appreciated by those skilled in the art that the above-mentioned various modules can be implemented at least in part by using general-purpose computer equipment or using customized equipment.

[0090] Here, the various aspects of the system and method described herein can be executed on any device or apparatus including the system. The program aspects of the technology can be considered as a "product" or "article" usually in the form of executable code and / or related data, which is carried or embodied in a type of machine-readable medium. "Storage" type media include any or all of the memories of mobile stations, computers, processors, etc. or their related modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide storage at the following locations: software programming is performed at any time. All or part of the software can sometimes communicate through the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another computer or processor. Therefore, another type of medium that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as through wired and optical landline networks and through various air links. Used on the physical interface between local devices. The physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered as the medium that carries the software. As used herein, unless restricted to tangible, non-transitory storage media, terms such as computer or machine readable media refer to any medium that participates in providing instructions to a processor for execution.

[0091] Although the present invention has been shown and described in detail in the drawings and the foregoing description, the present invention should be considered as exemplary rather than restrictive, and it should be understood that only exemplary embodiments are shown and described, and the scope of the present invention is not limited in any way. It is understood that any feature described herein can be used with any embodiment. The illustrative embodiments do not exclude each other, nor do they exclude other embodiments not listed herein. Therefore, the present invention also provides an embodiment including a combination of one or more of the illustrative embodiments described above. Without departing from the spirit and scope of the present invention, the present invention can be modified and varied, and therefore, only the limitations shown in the appended claims should be applied.

[0092] In the appended claims and the preceding description of the invention, unless the context requires otherwise due to express language or necessary implication, the word "comprise" or variations such as "include" or "comprising" are used in an inclusive sense, that is, specifying the presence of the stated features but not excluding the presence or addition of other features, in various embodiments of the invention.

[0093] It will be appreciated that, if any prior art publication is referred to herein, this reference does not constitute an admission that the publication forms part of the common general knowledge in the art.

Claims

1. A computer-implemented method for analyzing the engagement level of a subject in a video, comprising the following steps: receiving, via a receiving module, data of the video including a series of images showing the subject; Extracting features related to the subject in the video via an extraction module, the extraction step comprising: extracting, via a first extraction module, first features comprising one or more visual characteristics from one or more images in the series of images constituting the video; and extracting, via a second extraction module, second features comprising one or more spatial and / or temporal characteristics from the video; and The extracted features are processed via an analysis module to determine a score that is a scalar representation of the subject's level of engagement.

2. The method according to claim 1, wherein: The visual characteristics of the first feature include one or more facial and / or body characteristics of the subject displayed in the one or more images comprising the video.

3. The method according to claim 2, wherein: The one or more facial and / or body characteristics include facial movements, head position, and / or gaze direction of the subject shown in the one or more images comprising the video.

4. The method according to claim 1, wherein: The extracting step also includes extracting, via a third extraction module, third features including one or more audio characteristics associated with the data of the video.

5. The method according to claim 4, wherein: The audio characteristics of the third feature include one or more of volume, pitch, speech length, speech duration, speech content and / or emotion of the speech of or represented by the audio characteristics.

6. The method according to claim 4, wherein: The extracting step also includes extracting a fourth feature via a fourth extraction module, wherein the fourth feature includes input from a user, wherein the input includes one or more of text data, graphic data and / or image data input by one or more subjects during the shooting of the video.

7. The method according to claim 6, wherein: The fourth characteristic also includes one or more of a frequency of input from the user, a duration between respective inputs from the user, a reaction of the user to one or more other users, and / or a responsiveness of the user.

8. The method according to claim 1, wherein: The spatial and / or temporal characteristics of the second feature are represented by vector data; wherein the vector data includes an inflated three-dimensional (I3D) vector.

9. The method according to claim 1, wherein: The processing steps include: Analyzing the extracted first feature via a first analysis module; and The extracted second features are analyzed via a second analysis module; wherein the analysis steps of the first and / or the second analysis module are based on one or more machine learning algorithms.

10. The method according to claim 9, wherein: The first analysis module is adapted to process data from the first feature based on a temporal convolutional network (TCN); and wherein the second analysis module is adapted to process data from the second feature based on a multi-layer perceptron neural network (MLP).

11. The method according to claim 9, wherein: The processing step also includes generating attention weights based on processed data from the first feature and processed data from the second feature; and further processing the generated attention weights using the data from the first feature to determine a score that is a scalar representation of the subject's level of engagement.

12. The method according to claim 6, wherein: The processing step further includes one or more steps of analyzing the extracted third features via a third analysis module and analyzing the extracted fourth features via a fourth analysis module.

13. A system for analyzing the participation level of a subject in a video by implementing the steps of claim 1, the system comprising: a receiving module configured to receive data of the video including a series of images of the subject; An extraction module, the extraction module is configured to extract features related to the subject displayed in the video, the extraction module comprising: a first extraction module configured to extract first features including one or more visual characteristics from one or more images in the series of images constituting the video; and A second extraction module, the second extraction module is used to extract a second feature including one or more spatial and / or temporal characteristics from the video; and An analysis module is adapted to process the extracted features to determine a score that is a scalar representation of the subject's level of engagement.

14. The system according to claim 13, wherein: The analysis module includes a first analysis module for analyzing the extracted first feature, and a second analysis module for analyzing the extracted second feature; wherein the analysis step includes processing based on one or more machine learning algorithms; wherein the first analysis module is configured to process data from the first feature based on a temporal convolutional network (TCN); and the second analysis module is configured to process data from the second feature under a multilayer perceptron neural network (MLP).

15. The system of claim 13, wherein: The extraction module also includes at least one of a third extraction module for extracting a third feature including audio characteristics associated with the video, and a fourth extraction module for extracting a fourth feature including input from the user; wherein the analysis module also includes at least one of a third analysis module for processing the third feature and a fourth analysis module for processing the fourth feature.

Images