Learning state monitoring method and device applied to camera shooting set top box

Abstract

The invention provides a learning state monitoring method and device applied to a camera shooting set top box, and relates to the field of learning state monitoring. The method comprises: in response to a learning state analysis operation for a target user, obtaining a first image corresponding to the target user, the first image being a face image of the target user; acquiring a facial feature group corresponding to a target user through the first image; obtaining a second image corresponding to the target user, wherein the second image is a learning posture image of the target user; acquiring a learning action feature group corresponding to a target user through the second image; performing feature fusion on the facial feature group and the action feature group, and obtaining a comprehensive learning state feature group after feature fusion; and outputting a learning state analysis result corresponding to the target user through a learning state analysis model according to the comprehensive learning state feature group. According to the method, the problems that a simple learning model lacks comprehensive evaluation on learning features, so that the accuracy of a final judgment result is obviously insufficient, and the requirement of real-time and detailed state feedback is difficult to meet are solved.

Description

Technical Field

[0001] The present application relates to the field of learning status monitoring, and in particular to a learning status monitoring method and device applied to a camera top box. Background Art

[0002] In the modern educational learning machine system, most learning machines lack the function of supervised learning, and basically only provide video viewing and learning. Parents have no idea whether the learning state is healthy and focused.

[0003] At present, in order to solve the problem that the learning machine system lacks the supervised learning function, the existing methods usually build a simple learning model in the camera top box and judge whether the student's learning status is focused based on the model. The learning model will collect some learning characteristics of the target students, such as reading frequency, learning time, sight area, page browsing order and other indicators, and then analyze their concentration through these characteristics. However, this method has certain limitations: because the model only uses some single learning features, it is difficult to capture the complex correlation between multi-dimensional features when judging the student's learning status, and lacks a comprehensive evaluation of learning features. Therefore, the final judgment result is obviously insufficient in accuracy, and it is difficult to meet the needs of real-time and detailed status feedback.

[0004] Therefore, there is an urgent need for a learning state monitoring method and device applied to a camera top box. Summary of the invention

[0005] The present application provides a learning status monitoring method and device for use in a camera top box, which solves the problem that a simple learning model lacks a comprehensive evaluation of learning features, and therefore the final judgment result has obvious deficiencies in accuracy and is difficult to meet the needs of real-time and detailed status feedback.

[0006] In a first aspect of the present application, a learning status monitoring method applied to a camera top box is provided, the method comprising: in response to a learning status analysis operation for a target user, obtaining a first image corresponding to the target user, the first image being a facial image of the target user; obtaining a facial feature group corresponding to the target user through the first image; obtaining a second image corresponding to the target user, the second image being a learning posture image of the target user; obtaining a learning action feature group corresponding to the target user through the second image; performing feature fusion on the facial feature group and the action feature group, and obtaining a comprehensive learning status feature group after feature fusion; and outputting a learning status analysis result corresponding to the target user through a learning status analysis model according to the comprehensive learning status feature group.

[0007] Optionally, obtaining a facial feature group and a learning action feature group specifically includes: obtaining a micro-expression feature group and a blinking frequency feature group in the first image; using the micro-expression feature group and the blinking frequency feature group as the facial feature group; obtaining a head posture feature group and a limb movement feature group in the second image; using the head posture feature group and the limb movement feature group as the action feature group.

[0008] Optionally, feature fusion is performed on the facial feature group and the action feature group, and a comprehensive learning state feature group after feature fusion is obtained, specifically including: obtaining a target high-dimensional embedding space, and mapping the micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group to the target high-dimensional embedding space through standardization; performing pairwise feature interaction on the standardized micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group, to obtain multiple interactive feature groups; performing feature splicing through the multiple interactive feature groups, and obtaining a high-order feature matrix after high-order feature splicing; performing comprehensive feature extraction on the high-order feature matrix through a preset number of neural network connection layers, and obtaining a comprehensive learning state feature group after the comprehensive feature extraction.

[0009] Optionally, obtaining a comprehensive learning state feature group after comprehensive feature extraction specifically includes: obtaining the comprehensive learning state feature group according to the following formula:

[0010]

[0011] Among them, E ME is the micro-expression feature group after feature mapping, E BF is the eye blink frequency feature group after feature mapping, E HP is the head posture feature group after feature mapping, E GA To perform feature mapping on the body movement feature group, To perform pairwise feature interaction, F high is a high-order feature matrix, concat is feature concatenation, F LS is a comprehensive learning state feature group, w1, w2, w3 are the weight matrices corresponding to each neural network connection layer, b1, b2, b3 are the bias vectors corresponding to each neural network connection layer, and σ is the activation function.

[0012] Optionally, based on the comprehensive learning state feature group, the learning state analysis result corresponding to the target user is output through the learning state analysis model, specifically including: based on the comprehensive learning state feature group, respectively calculating the target user's emotion fluctuation evaluation value, fatigue evaluation value, involvement evaluation value and interactivity evaluation value through the learning state analysis model; calculating the comprehensive learning state value through the learning state analysis model based on the emotion fluctuation evaluation value, fatigue evaluation value, involvement evaluation value and interactivity evaluation value; based on the comprehensive learning state value, outputting the learning state analysis result corresponding to the target user.

[0013] Optionally, calculating the comprehensive learning state value through the learning state analysis model specifically includes: calculating the comprehensive learning state value according to the following formula:

[0014] S learning =tanh(α1*V emotion +α2*V fatigue +α3*V engagement +α4*

[0015] V interact );

[0016] Among them, S learning is the comprehensive learning state value, V emotion is the emotional fluctuation evaluation value, α1 is the evaluation weight corresponding to the emotional fluctuation evaluation value, V fatigue is the fatigue evaluation value, α2 is the evaluation weight corresponding to the fatigue evaluation value, V engagement is the evaluation value of the degree of involvement, α3 is the evaluation weight corresponding to the evaluation value of the degree of involvement, V interact is the interaction evaluation value, and α4 is the evaluation weight corresponding to the interaction evaluation value.

[0017] Optionally, based on the comprehensive learning state value, the learning state analysis result corresponding to the target user is output, specifically including: obtaining the target value interval of the comprehensive learning state value, the target value interval including a first target value interval, a second target value interval and a third target value interval; if it is judged that the target value interval is in the first target value interval, confirming that the learning state analysis result is a concentration state; if it is judged that the target value interval is in the second target value interval, confirming that the learning state analysis result is a fatigue state; if it is judged that the target value interval is in the third target value interval, confirming that the learning state analysis result is a distraction state.

[0018] In a second aspect of the present application, a learning status monitoring device for use in a camera top box is provided, the device comprising an acquisition module and a processing module, wherein:

[0019] An acquisition module is used to acquire a first image corresponding to the target user in response to a learning status analysis operation for the target user, the first image being a facial image of the target user; acquiring a facial feature group corresponding to the target user through the first image; acquiring a second image corresponding to the target user, the second image being a learning posture image of the target user; and acquiring a learning action feature group corresponding to the target user through the second image.

[0020] The processing module is used to perform feature fusion on the facial feature group and the action feature group, and obtain the comprehensive learning state feature group after the feature fusion; according to the comprehensive learning state feature group, the learning state analysis result corresponding to the target user is output through the learning state analysis model.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to perform any of the above methods.

[0023] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0024] 1. When performing the learning status analysis operation on the target user, obtain the facial image corresponding to the target user, and obtain the facial feature group corresponding to the target user through the first image, and obtain the learning posture image corresponding to the target user, and obtain the learning action feature group corresponding to the target user through the second image, so as to perform multi-dimensional feature fusion on the facial feature group and the action feature group, and finally output the learning status analysis result corresponding to the target user through the learning status analysis model based on the comprehensive learning status feature group, thereby ensuring the accuracy of the learning status analysis result and meeting the real-time and detailed learning status feedback requirements.

[0025] 2. Obtain the target high-dimensional embedding space, and through standardization, feature map the micro-expression feature group, blinking frequency feature group, head posture feature group and limb movement feature group to the target high-dimensional embedding space respectively, so as to perform pairwise feature interaction on the standardized micro-expression feature group, blinking frequency feature group, head posture feature group and limb movement feature group respectively, so as to obtain multiple interactive feature groups, thereby performing feature splicing through multiple interactive feature groups, and obtaining a high-order feature matrix after high-order feature splicing, so as to perform comprehensive feature extraction on the high-order feature matrix through a preset number of neural network connection layers, and obtain a comprehensive learning state feature group after comprehensive feature extraction, thereby not only realizing the efficient fusion of multimodal learning features, but also capturing the complex correlation between each feature group, being able to more accurately evaluate the learning state, and ultimately improving the real-time monitoring and analysis accuracy of the learning state.

[0026] 3. By confirming the target value interval of the comprehensive learning state value and outputting the learning state analysis result corresponding to the target user according to the target value interval, a refined classification of the learning state analysis result is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a learning state monitoring method applied to a camera top box provided in an embodiment of the present application;

[0028] Figure 2 It is a module schematic diagram of a learning status monitoring device applied to a camera top box provided in an embodiment of the present application;

[0029] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0030] Explanation of the reference numerals: 21, acquisition module; 22, processing module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0031] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0032] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0033] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0034] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0035] Please refer to Figure 1 , which shows a flow chart of a learning status monitoring method applied to a camera top box provided in an embodiment of the present application, the flow chart mainly includes the following steps: S101 to S106.

[0036] Step S101 , in response to a learning status analysis operation for a target user, a first image corresponding to the target user is acquired, where the first image is a facial image of the target user.

[0037] Specifically, in an embodiment of the present application, when the target user is correctly located in front of the set-top box, the set-top box can obtain an image of the target user through a camera installed inside, and analyze the captured image to analyze the learning status of the target user in real time. In an embodiment of the present application, the target user can be any child, and the image of the target user first includes his or her facial image. The set-top box will screen the captured facial images to be confirmed, and select the clearest one or more images as the confirmed facial image of the target user.

[0038] Step S102: obtaining a facial feature group corresponding to the target user through the first image.

[0039] Specifically, by performing a preprocessing operation on the first image and obtaining multiple facial expression features of the target user as a facial feature group, the steps are as follows: converting the first image into a grayscale image to reduce the processing complexity and remove the interference of color information, normalizing the grayscale image, and adjusting the pixel value to a uniform range, which helps to improve the accuracy of subsequent feature extraction. Applying methods such as Gaussian filtering to remove noise in the image makes the subsequent facial key point detection more accurate. Performing key point detection on the cropped facial area to identify the positions of key parts such as eyes, eyebrows, nose, and mouth, using a 68-point or 98-point facial key point detection model to ensure accurate capture of subtle facial movements, and smoothing the acquired key point coordinates to reduce possible jitter during the detection process. Based on the key point coordinates, calculate each expression feature (such as the angle of the corner of the mouth, the amplitude of the eyebrow lift, etc.). Use a deep learning model (such as a convolutional neural network combined with a long short-term memory network) to further extract facial micro-expression features, such as joy, confusion, sadness, surprise, etc. Each feature value in the feature group reflects the intensity or rate of change of a specific expression for subsequent calculation of the comprehensive learning state.

[0040] Step S103: acquiring a second image corresponding to the target user, where the second image is a learning posture image of the target user.

[0041] Specifically, the image of the target user first includes his learning posture image, that is, the second image. The learning posture image needs to include head posture features, eye direction features, sitting posture features, body inclination features, etc. The set-top box will screen the captured learning posture images to be confirmed, and select the clearest one or more images as the confirmed learning posture images of the target user.

[0042] Step S104: obtaining a learning action feature group corresponding to the target user through the second image.

[0043] Specifically, the second image is preprocessed and multiple learning action features of the target user are obtained as a learning action feature group. The steps are as follows: the second image is grayscaled, denoised and normalized to reduce background interference and improve the stability of feature extraction, and the head posture estimation algorithm is used to calculate the rotation and tilt angles of the head; the head orientation information is obtained to form head posture features, and the eye direction is detected to determine whether the user's line of sight is focused on the learning materials; the relative position of the hand is identified through a deep learning target detection algorithm to determine whether the hand is in contact with the learning tool and to obtain limb movement features; the posture information of the shoulders, back and other parts is identified to form an overall sitting posture feature, etc. The above-mentioned extracted head posture, eye direction, hand position, body posture and other features are integrated to form a learning action feature group.

[0044] In a possible implementation, step S104 also includes: obtaining a micro-expression feature group and a blinking frequency feature group in the first image; using the micro-expression feature group and the blinking frequency feature group as a facial feature group; obtaining a head posture feature group and a limb movement feature group in the second image; using the head posture feature group and the limb movement feature group as a motion feature group.

[0045] Specifically, the features in the facial feature group and the learning action feature group are classified into four major feature groups, namely, micro-expression feature group, blinking frequency feature group, head posture feature group and limb movement feature group, and further through the classified feature groups.

[0046] Step S105, performing feature fusion on the facial feature group and the action feature group, and obtaining a comprehensive learning state feature group after feature fusion.

[0047] Specifically, feature fusion is further performed on the feature groups classified in step S104, and a comprehensive learning state feature group after feature fusion is obtained.

[0048] In a possible implementation, step S105 also includes: obtaining a target high-dimensional embedding space, and mapping the micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group to the target high-dimensional embedding space through standardization processing; performing pairwise feature interaction on the standardized micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group to obtain multiple interactive feature groups; performing feature splicing through the multiple interactive feature groups, and obtaining a high-order feature matrix after the high-order feature splicing; performing comprehensive feature extraction on the high-order feature matrix through a preset number of neural network connection layers, and obtaining a comprehensive learning state feature group after the comprehensive feature extraction.

[0049] Specifically, by constructing a high-dimensional embedding space adapted to learning state analysis to accommodate the mapping results of various features, the high-dimensional space can better characterize the comprehensive learning state characteristics of the target user through multi-dimensional feature distribution modeling. For the four major feature groups, namely the micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group, standardization is performed separately to ensure that each feature group is unified in different dimensions, and the standardized feature groups are mapped to the target high-dimensional embedding space, thereby forming a standardized feature vector in the high-dimensional space, ensuring that the distribution in the embedding space is conducive to feature fusion. Let each feature group be represented as follows after standardization: Micro-expression feature group: F ME ; Blink frequency feature group: F BF ; Head posture feature group: F HP ; Body movement feature group: F GA Then use the embedding matrix W ME , WBF , W HP , W GA The above feature groups are respectively embedded in high dimensions, that is, the micro-expression feature group, blinking frequency feature group, head posture feature group and limb movement feature group are respectively mapped to the target high-dimensional embedding space, and the expression is as follows: E ME =F ME ·W ME 、E BF =F BF ·W BF 、E HP =F HP ·W HP 、E GA =F GA ·W GA After that, feature interaction is introduced to take into account the second-order mutual features between feature groups to better capture the correlation between facial expressions and action features, and all second-order interaction features are concatenated into a high-order feature matrix. The calculation formula of the high-order feature matrix is ​​as follows:

[0050]

[0051] Among them, F high is a high-order feature matrix, They are all second-order mutual features between feature groups. A multi-layer neural network is used to further process the high-order feature matrix to extract deep comprehensive features. Finally, the comprehensive learning state feature group is obtained through the following formula:

[0052] F LS =w3(σ(w2(σ(w1*F high +b1)+b2))+b3;

[0053] Among them, F LS is the comprehensive learning state feature group, and σ is the activation function. In the embodiment of the present application, in order to make the calculation process not complicated and to accurately obtain the comprehensive learning state feature group outputted at the end, a three-layer neural network connection layer is constructed, and w1, w2, and w3 are the weight matrices corresponding to each neural network connection layer, and b1, b2, and b3 are the bias vectors corresponding to each neural network connection layer.

[0054] Step S106: outputting the learning state analysis result corresponding to the target user through the learning state analysis model according to the comprehensive learning state feature group.

[0055] Specifically, according to the comprehensive learning state feature group, the target user's mood fluctuation evaluation value, fatigue evaluation value, engagement evaluation value and interaction evaluation value are calculated respectively through the learning state analysis model. The calculation methods of the mood fluctuation evaluation value, fatigue evaluation value, engagement evaluation value and interaction evaluation value are as follows. The mood fluctuation evaluation value is calculated by the following formula:

[0056]

[0057] Among them, V emotion is the emotional fluctuation evaluation value, is the i-th micro-expression feature in the micro-expression feature group, and w ME ,i is the feature weight corresponding to the i-th micro-expression feature, b ME , i is the bias term corresponding to the i-th micro-expression feature, k is the total number of micro-expression features, α emotion To adjust the coefficient of the comprehensive learning state characteristics on the fatigue evaluation. The emotional fluctuation evaluation value is based on the nonlinear combination of the micro-expression feature group and is used to measure the emotional fluctuation of the target user. The fatigue evaluation value is calculated by the following formula:

[0058]

[0059] Among them, V fatigue is the fatigue evaluation value, F BF is the blink frequency feature group, F HP is the head posture feature group, β1, β2, and β3 are all weight coefficients, which can be set according to actual calculation requirements. fatigue To adjust the weight of the comprehensive learning state feature on the fatigue evaluation value. The fatigue evaluation value reflects the fatigue state of the target user through a nonlinear combination of blink frequency features and head posture features.

[0060] The interaction rating value is calculated using the following formula:

[0061] V interact =δ1*relu(F GA )+δ2*(ln(1+|F GA ∣)+F GA 2 )+α interact *

[0062] sigmoid(F LS );

[0063] Among them, V interact is the interaction evaluation value, δ1 and δ2 are weight coefficients, α interactTo adjust the weight of the comprehensive learning state characteristics on the interaction evaluation value, you can set it according to the actual calculation needs. The interaction evaluation value is based on body movement characteristics and is used to measure the level of interaction between users and learning materials. The engagement evaluation value is calculated using the following formula:

[0064]

[0065] Among them, V engagement is the investment evaluation value, γ1-γ6 are adjustment weights, which can be set according to actual calculation requirements, α engagement To adjust the weight of comprehensive learning status characteristics in the evaluation of engagement. The functions of each item in the formula are explained as follows: is an exponential decay term, which reflects the negative impact of fatigue. Through exponential decay, the suppression of input is more significant when fatigue is high. tanh(V interact ) is a nonlinear compression, which tends to 1 when the interaction is high and close to 0 when it is low, and is used to amplify the contribution of high interaction; sigmoid (V emotion ) is the smoothing emotion fluctuation term. The Sigmoid function smoothes the emotion fluctuation to the range of 0 to 1, so that the extreme fluctuation of emotion will not have too large a jump effect on the engagement; (V interact *V emotion ) is a cross term, which reflects the interaction between user interaction and emotion, which can enhance or inhibit engagement; (1-V fatigue ) 2 is the square term of fatigue, which is used to further enhance the positive impact of low fatigue state on engagement, making the effect of improving engagement more obvious when fatigue is lower.

[0066] Furthermore, according to the emotion fluctuation evaluation value, fatigue evaluation value, engagement evaluation value and interaction evaluation value, the comprehensive learning state value is calculated through the learning state analysis model, and the comprehensive learning state value is calculated according to the following formula:

[0067] S learning =tanh(α1*V emotion +α2*V fatigue +α3*V engagement +α4*

[0068] V interact );

[0069] Among them, S learning is the comprehensive learning state value, V emotion is the emotional fluctuation evaluation value, α1 is the evaluation weight corresponding to the emotional fluctuation evaluation value, V fatigue is the fatigue evaluation value, α2 is the evaluation weight corresponding to the fatigue evaluation value, V engagementis the evaluation value of the degree of involvement, α3 is the evaluation weight corresponding to the evaluation value of the degree of involvement, V interact is the interaction evaluation value, and α4 is the evaluation weight corresponding to the interaction evaluation value.

[0070] According to the comprehensive learning state value, the learning state analysis result corresponding to the target user is output, and, in a possible implementation, step S106 also includes: obtaining the target value interval where the comprehensive learning state value is located, the target value interval includes a first target value interval, a second target value interval and a third target value interval; if it is judged that the target value interval is in the first target value interval, the learning state analysis result is confirmed to be a focused state; if it is judged that the target value interval is in the second target value interval, the learning state analysis result is confirmed to be a tired state; if it is judged that the target value interval is in the third target value interval, the learning state analysis result is confirmed to be a distracted state. For example, the first target value interval: defined as a relatively high comprehensive learning state value interval, usually corresponds to a highly focused state of the user. The second target value interval: defined as a medium comprehensive learning state value interval, usually indicates that the user shows signs of fatigue, but can still maintain a certain degree of focus. The third target value interval: defined as a relatively low comprehensive learning state value interval, usually reflects the user's distracted state or low involvement state. According to the target value interval where the comprehensive learning state value is located, the user's learning state analysis result is output, focused state: if S learning If the user is in the first target value range, the system confirms that the user is currently in a focused state, that is, the user maintains a high level of attention and involvement in the learning content. learning If the user is in the second target value range, the system determines that the user is in a state of fatigue. This state usually means that the user still has a certain degree of learning commitment, but due to fatigue, the attention has decreased. Distracted state: If S learning If the user is in the third target value interval, the system confirms that the user is in a distracted state, indicating that the user's commitment to learning is low and the user may be inattentive. It should be noted that in the embodiment provided by itself, the target value interval is divided into the first target value interval, the second target value interval and the third target value interval, which is only a division of the present application. According to different actual application requirements, a more detailed division can be performed, such as dividing the target value interval into four, five or six different intervals, and outputting the learning state analysis results corresponding to different target users according to the target value interval where the learning state value is located.

[0071] The present application adopts the above method, and when performing a learning status analysis operation on a target user, obtains a facial image corresponding to the target user, and obtains a facial feature group corresponding to the target user through a first image, and obtains a learning posture image corresponding to the target user, and obtains a learning action feature group corresponding to the target user through a second image, thereby performing multi-dimensional feature fusion of the facial feature group and the action feature group, and finally outputs a learning status analysis result corresponding to the target user through a learning status analysis model based on the comprehensive learning status feature group, thereby ensuring the accuracy of the learning status analysis result and being able to meet the needs of real-time and detailed learning status feedback.

[0072] Please refer to Figure 2 , which shows a module schematic diagram of a learning state monitoring device applied to a camera top box provided in an embodiment of the present application, the device includes an acquisition module 21 and a processing module 22, wherein:

[0073] The acquisition module 21 is used to obtain a first image corresponding to the target user in response to a learning status analysis operation for the target user, the first image being a facial image of the target user; through the first image, obtaining a facial feature group corresponding to the target user; obtaining a second image corresponding to the target user, the second image being a learning posture image of the target user; and through the second image, obtaining a learning action feature group corresponding to the target user.

[0074] The processing module 22 is used to perform feature fusion on the facial feature group and the action feature group, and obtain a comprehensive learning state feature group after feature fusion; according to the comprehensive learning state feature group, the learning state analysis result corresponding to the target user is output through the learning state analysis model.

[0075] In a possible implementation, the acquisition module 21 is used to acquire facial feature groups and learning action feature groups, specifically including: acquiring a micro-expression feature group and a blinking frequency feature group in the first image; using the micro-expression feature group and the blinking frequency feature group as the facial feature group; acquiring a head posture feature group and a limb movement feature group in the second image; using the head posture feature group and the limb movement feature group as the action feature group.

[0076] In one possible implementation, the processing module 22 is used to perform feature fusion on the facial feature group and the action feature group, and obtain a comprehensive learning state feature group after feature fusion, specifically including: obtaining a target high-dimensional embedding space, and mapping the micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group to the target high-dimensional embedding space through standardization processing; performing pairwise feature interaction on the micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group after standardization processing to obtain multiple interactive feature groups; performing feature splicing through multiple interactive feature groups, and obtaining a high-order feature matrix after high-order feature splicing; performing comprehensive feature extraction on the high-order feature matrix through a preset number of neural network connection layers, and obtaining a comprehensive learning state feature group after comprehensive feature extraction.

[0077] In a possible implementation manner, the processing module 22 is used to obtain the comprehensive learning state feature group after comprehensive feature extraction, specifically including: obtaining the comprehensive learning state feature group according to the following formula:

[0078]

[0079] Among them, E ME is the micro-expression feature group after feature mapping, E BF is the eye blink frequency feature group after feature mapping, E HP is the head posture feature group after feature mapping, E GA To perform feature mapping on the body movement feature group, To perform pairwise feature interaction, F high is a high-order feature matrix, concat is feature concatenation, F LS is a comprehensive learning state feature group, w1, w2, w3 are the weight matrices corresponding to each neural network connection layer, b1, b2, b3 are the bias vectors corresponding to each neural network connection layer, and σ is the activation function.

[0080] In one possible implementation, the processing module 22 is used to output the learning state analysis result corresponding to the target user through the learning state analysis model based on the comprehensive learning state feature group, specifically including: calculating the target user's emotion fluctuation evaluation value, fatigue evaluation value, involvement evaluation value and interactivity evaluation value through the learning state analysis model according to the comprehensive learning state feature group; calculating the comprehensive learning state value through the learning state analysis model according to the emotion fluctuation evaluation value, fatigue evaluation value, involvement evaluation value and interactivity evaluation value; and outputting the learning state analysis result corresponding to the target user according to the comprehensive learning state value.

[0081] In a possible implementation, the processing module 22 is used to calculate the comprehensive learning state value through the learning state analysis model, specifically including: calculating the comprehensive learning state value according to the following formula:

[0082] S learning =tanh(α1*V emotion +α2*V fatigue +α3*V engagement +α4*

[0083] V interact );

[0084] Among them, S learning is the comprehensive learning state value, V emotion is the emotional fluctuation evaluation value, α1 is the evaluation weight corresponding to the emotional fluctuation evaluation value, V fatigue is the fatigue evaluation value, α2 is the evaluation weight corresponding to the fatigue evaluation value, V engagement is the evaluation value of the degree of involvement, α3 is the evaluation weight corresponding to the evaluation value of the degree of involvement, V interact is the interaction evaluation value, and α4 is the evaluation weight corresponding to the interaction evaluation value.

[0085] In a possible implementation, the processing module 22 is used to output a learning status analysis result corresponding to the target user based on the comprehensive learning status value, specifically including: obtaining a target value interval in which the comprehensive learning status value is located, the target value interval including a first target value interval, a second target value interval and a third target value interval; if it is determined that the target value interval is in the first target value interval, confirming that the learning status analysis result is a concentration state; if it is determined that the target value interval is in the second target value interval, confirming that the learning status analysis result is a fatigue state; if it is determined that the target value interval is in the third target value interval, confirming that the learning status analysis result is a distraction state.

[0086] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0087] The present application also provides an electronic device. Figure 3 , Figure 3 30 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: at least one processor 301, at least one communication bus 302, a user interface 303, at least one network interface 304, and a memory 305.

[0088] The communication bus 302 is used to realize the connection and communication between these components.

[0089] The user interface 303 may include a display screen (Display) and a camera (Camera), and the optional user interface 303 may also include a standard wired interface and a wireless interface.

[0090] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0091] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0092] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 The memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a learning status monitoring application program applied to the camera top box.

[0093] exist Figure 3 In the electronic device shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the learning state monitoring application stored in the memory 305 and applied to the camera top box. When executed by one or more processors 301, the electronic device executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0094] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

[0095] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0097] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0100] The above are only exemplary embodiments disclosed in this application, and the scope of the disclosure of this application cannot be limited by this. That is, any equivalent changes and modifications made according to the teachings disclosed in this application are still within the scope of the disclosure of this application. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other implementation schemes disclosed in this application.

[0101] This application is intended to cover any modifications, uses or adaptive changes disclosed in this application, which follow the general principles disclosed in this application and include common knowledge or customary technical means in the technical field not recorded in this application.

Claims

1. A learning state monitoring method applied to a camera top box, characterized in that: The method comprises: In response to a learning status analysis operation for a target user, acquiring a first image corresponding to the target user, where the first image is a facial image of the target user; Obtaining a facial feature group corresponding to a target user through the first image; Acquire a second image corresponding to the target user, where the second image is a learning posture image of the target user; Obtaining a learning action feature group corresponding to the target user through the second image; Performing feature fusion on the facial feature group and the action feature group, and obtaining a comprehensive learning state feature group after feature fusion; According to the comprehensive learning state feature group, a learning state analysis result corresponding to the target user is outputted through a learning state analysis model.

2. The method according to claim 1, characterized in that Acquiring the facial feature group and the learning action feature group specifically includes: Acquire a micro-expression feature group and a blink frequency feature group in the first image; Using the micro-expression feature group and the blinking frequency feature group as the facial feature group; Acquire a head posture feature group and a limb movement feature group in the second image; The head posture feature group and the limb movement feature group are used as the movement feature group.

3. The method according to claim 2, characterized in that The step of fusing the facial feature group and the action feature group and obtaining a comprehensive learning state feature group after the feature fusion specifically includes: Acquire a target high-dimensional embedding space, and perform feature mapping on the micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group respectively with the target high-dimensional embedding space through standardization processing; Performing feature interactions on the standardized micro-expression feature group, the blinking frequency feature group, the head posture feature group, and the limb movement feature group, respectively, to obtain a plurality of interactive feature groups; Perform feature splicing through a plurality of the interactive feature groups, and obtain a high-order feature matrix after the high-order features are spliced; Comprehensive feature extraction is performed on the high-order feature matrix through a preset number of neural network connection layers, and the comprehensive learning state feature group after the comprehensive feature extraction is obtained.

4. The method according to claim 3, characterized in that The obtaining of the comprehensive learning state feature group after the comprehensive feature extraction specifically includes: The comprehensive learning state feature group is obtained according to the following formula: Among them, E ME is the micro-expression feature group after feature mapping, E BF is the eye blink frequency feature group after feature mapping, E HP is the head posture feature group after feature mapping, E GA The body movement feature group after feature mapping, To perform the pairwise feature interaction, F high is the high-order feature matrix, concat is the feature concatenation, F LS is the comprehensive learning state feature group, w1, w2, w3 are the weight matrices corresponding to each of the neural network connection layers, b1, b2, b3 are the bias vectors corresponding to each of the neural network connection layers, and σ is the activation function.

5. The method according to claim 1, characterized in that Outputting the learning state analysis result corresponding to the target user through the learning state analysis model according to the comprehensive learning state feature group specifically includes: According to the comprehensive learning state feature group, respectively calculating the target user's emotion fluctuation evaluation value, fatigue evaluation value, engagement evaluation value and interaction evaluation value through a learning state analysis model; Calculating a comprehensive learning state value through the learning state analysis model according to the emotion fluctuation evaluation value, the fatigue evaluation value, the engagement evaluation value, and the interaction evaluation value; According to the comprehensive learning status value, the learning status analysis result corresponding to the target user is output.

6. The method according to claim 5, characterized in that The calculating of the comprehensive learning state value by the learning state analysis model specifically includes: The comprehensive learning state value is calculated according to the following formula: S learning =tanh(α1*V emotion +α2*V fatigue +α3*V engagement +α4* V interact )； Among them, S learning is the comprehensive learning state value, V emotion is the emotion fluctuation evaluation value, α1 is the evaluation weight corresponding to the emotion fluctuation evaluation value, V fatigue is the fatigue evaluation value, α2 is the evaluation weight corresponding to the fatigue evaluation value, V engagement is the evaluation value of the degree of involvement, α3 is the evaluation weight corresponding to the evaluation value of the degree of involvement, V interact is the interaction evaluation value, and α4 is the evaluation weight corresponding to the interaction evaluation value.

7. The method according to claim 5, characterized in that Outputting the learning status analysis result corresponding to the target user according to the comprehensive learning status value specifically includes: Obtaining a target value interval where the comprehensive learning state value is located, the target value interval comprising a first target value interval, a second target value interval and a third target value interval; If it is judged that the target value interval is in the first target value interval, the learning status analysis result is confirmed to be a concentration state; if it is judged that the target value interval is in the second target value interval, the learning status analysis result is confirmed to be a fatigue state; if it is judged that the target value interval is in the third target value interval, the learning status analysis result is confirmed to be a distraction state.

8. A learning status monitoring device used in a camera top box, characterized in that: The device comprises an acquisition module and a processing module, wherein: The acquisition module is used to, in response to the learning state analysis operation for the target user, acquire a first image corresponding to the target user, the first image being a facial image of the target user; acquire a facial feature group corresponding to the target user through the first image; acquire a second image corresponding to the target user, the second image being a learning posture image of the target user; and acquire a learning action feature group corresponding to the target user through the second image; The processing module is used to perform feature fusion on the facial feature group and the action feature group, and obtain a comprehensive learning state feature group after feature fusion; based on the comprehensive learning state feature group, output the learning state analysis result corresponding to the target user through the learning state analysis model.

9. An electronic device, characterized in that: It includes a processor, a communication bus, a user interface, a network interface and a memory, wherein the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images