Method and device for evaluating learning concentration degree of user, electronic equipment and storage medium

Abstract

The invention discloses a method and a device for evaluating the learning concentration degree of a user, electronic equipment and a storage medium, relates to the technical field of image analysis and visual computing, realizes comprehensive and detailed concentration degree evaluation of the user, can adapt to diversified learning scenes while improving the accuracy and intelligence of learning concentration degree evaluation, and improves the learning experience of the user. And the risk of misjudgment is reduced. The method comprises the following steps: acquiring a video stream acquired when a user learns, and extracting a plurality of to-be-detected image frames from the video stream; based on the plurality of to-be-detected image frames, detecting eyes and behavior actions of the user during learning to obtain an action detection result; based on the plurality of to-be-detected image frames, detecting articles placed in a learning environment where the user is located during learning, and based on the video stream, carrying out environment noise detection to obtain an environment detection result; and evaluating the concentration degree of the user during learning by using the behavior action detection result and the learning environment detection result to obtain a concentration degree evaluation result.

Description

Technical Field

[0001] The present application relates to the field of image analysis and visual computing technology, and in particular to a method, device, electronic device and storage medium for evaluating a user's learning concentration. Background Art

[0002] With the rapid development of information technology and the continuous innovation of educational methods, distance learning has become an important trend in the field of modern education. Whether parents accompany their children to study at home or students receive distance education through online platforms, it has greatly broadened the space and time limits of learning. However, this learning model also brings new problems, namely how to effectively monitor and evaluate learners' concentration, especially in the absence of direct supervision and guidance.

[0003] In the related art, facial recognition technology is widely used to evaluate the user's concentration. This technology mainly evaluates whether the user is in a state of concentration by detecting whether the user's eyes are fixed on the screen. However, the applicant recognizes that sometimes the user's eyes are fixed on the screen, but in fact they are in a daze or distracted. This method cannot accurately identify the user's non-concentrated state; moreover, in self-study scenarios, such as taking notes, answering test questions or participating in online discussions, users may need to leave the screen frequently to write, think or communicate with other learners. It is obviously unreasonable to judge the user's concentration by whether the eyes are fixed on the screen. It has certain limitations, resulting in inaccurate assessment of learning concentration, prone to misjudgment, and poor intelligence. Summary of the invention

[0004] In view of this, the present application provides a method, device, electronic device and storage medium for evaluating a user's learning concentration, the main purpose of which is to solve the current problems of inaccurate evaluation of learning concentration, easy misjudgment and poor intelligence.

[0005] According to the first aspect of the present application, a method for evaluating a user's learning concentration is provided, the method comprising:

[0006] Acquire a video stream collected when the user is learning, and extract a plurality of image frames to be detected from the video stream;

[0007] Based on the multiple image frames to be detected, detecting the eyes and behavioral actions of the user during learning to obtain an action detection result;

[0008] Based on the multiple image frames to be detected, objects placed in the learning environment where the user is learning are detected, and environmental noise detection is performed based on the video stream to obtain an environmental detection result;

[0009] The user's concentration during learning is evaluated using the behavior detection result and the learning environment detection result to obtain a concentration evaluation result.

[0010] According to a second aspect of the present application, a device for evaluating a user's learning concentration is provided, the device comprising:

[0011] An extraction module, used to obtain a video stream collected when the user is learning, and extract a plurality of image frames to be detected from the video stream;

[0012] A detection module, configured to detect the eyes and behavioral actions of the user during learning based on the multiple image frames to be detected, and obtain an action detection result;

[0013] The detection module is further used to detect objects placed in the learning environment where the user is learning based on the multiple image frames to be detected, and to perform environmental noise detection based on the video stream to obtain an environmental detection result;

[0014] The evaluation module is used to evaluate the user's concentration during learning by using the behavior detection result and the learning environment detection result to obtain a concentration evaluation result.

[0015] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0016] According to a fourth aspect of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0017] By means of the above technical scheme, the present application provides a method, device, electronic device and storage medium for evaluating a user's learning concentration. The present application obtains a video stream collected when the user is learning, extracts a plurality of image frames to be detected from the video stream, detects the eyes and behavioral actions of the user when learning based on the plurality of image frames to be detected, and obtains an action detection result. Based on the plurality of image frames to be detected, the objects placed in the learning environment where the user is learning are detected, and the environmental noise is detected based on the video stream to obtain an environment detection result. The user's concentration when learning is evaluated by using the behavioral action detection result and the learning environment detection result to obtain a concentration evaluation result. The method can not only accurately capture the subtle changes in the user's movements during the learning process, but also analyze the influence of external factors such as the noise level and the placement of objects in the learning environment on the user's concentration, thereby realizing a comprehensive and detailed evaluation of the user's concentration. While improving the accuracy and intelligence of the evaluation of learning concentration, the method can adapt to diversified learning scenarios and reduce the risk of misjudgment, and has good intelligence.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0020] Figure 1 A schematic diagram of a method for evaluating a user's learning concentration provided by an embodiment of the present application is shown;

[0021] Figure 2A A schematic diagram of another method for evaluating the learning concentration of a user provided in an embodiment of the present application is shown;

[0022] Figure 2B A schematic diagram of a flow chart of another method for evaluating a user's learning concentration provided in an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of the structure of a device for evaluating the learning concentration of a user provided in an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0026] The present application embodiment provides a method for evaluating a user's learning concentration, such as Figure 1 As shown, the method includes:

[0027] 101. Obtain a video stream collected when the user is learning, and extract multiple image frames to be detected from the video stream.

[0028] The embodiment of the present application can be applied to an online education platform, which allows teachers to easily create, edit and publish a variety of course content, including graphics, audio, video and other forms, to meet the learning needs and interests of different students; students can freely choose and learn these courses according to their own learning progress and interests. In addition to the display of course content, the online education platform also emphasizes real-time interaction. Through built-in chat, Q&A, discussion area and other functions, students can ask questions to teachers in real time, exchange learning experiences with other students, and form a strong learning atmosphere. At the same time, the platform also supports live teaching functions. Teachers can explain the course content to students in real time, and students can interact with teachers in real time through bullet screens, comments and other methods. Moreover, by integrating cameras and intelligent algorithms, the online education platform can capture the video stream of students in the learning process in real time, and automatically extract multiple image frames to be detected from it, not only focusing on students' eye activities, such as blinking frequency and line of sight direction, to accurately judge whether their attention is focused, but also deeply analyzing students' behavioral actions, such as sitting posture, gestures, etc., to comprehensively evaluate the learning status. Furthermore, the online education platform uses object recognition technology to monitor the placement of objects in the learning environment, such as books and stationery, and combines audio processing technology to detect the environmental noise level in real time and identify whether there are interference sources in the students' learning atmosphere. These multi-dimensional data are input into the concentration assessment model, which comprehensively considers multiple factors such as eye movement, behavioral movements, and learning environment to quantify the students' concentration.

[0029] Among them, in order to evaluate the user's learning concentration, the online education platform will use cameras or intelligent monitoring equipment to collect real-time video streams in the space where the user conducts learning activities. The video stream records the user's learning process in the form of a continuous image sequence, including key information such as facial expressions and body postures. In order to effectively analyze the user's learning status, the online education platform uses image processing technology to extract multiple image frames to be detected from the continuous video stream evenly or according to specific event trigger conditions (such as changes in user movements) as the basis for subsequent analysis, ensuring the accuracy and timeliness of the analysis.

[0030] For example, a video stream collected when a user is taking an online course is obtained, and a series of image frames reflecting the user's learning dynamics are extracted from the video stream as image frames to be detected.

[0031] 102. Based on the multiple image frames to be detected, detect the eyes and behavioral actions of the user during learning to obtain an action detection result.

[0032] In the embodiment of the present application, the online education platform will detect the user's eye opening and closing state (such as the length of time staring at the screen, the length of time the eyes leave the screen) and behavioral actions (such as sitting posture, gestures) when learning based on multiple image frames to be detected. Eye detection helps to determine the user's concentration level, while behavioral action detection can also reflect the user's concentration level while also reflecting the user's learning attitude and physical comfort, which helps to promptly detect signs of learning distraction or fatigue and achieve accurate assessment of learning concentration.

[0033] For example, if it is detected that the user has not blinked for 10 consecutive seconds and his eyes are focused on the screen, it may indicate that he is in a state of concentration; while frequent changes in sitting posture or hand movements may mean that the user is uncomfortable or distracted.

[0034] 103. Based on the multiple image frames to be detected, objects placed in the learning environment where the user is learning are detected, and environmental noise detection is performed based on the video stream to obtain an environmental detection result.

[0035] In the embodiment of the present application, the online education platform further identifies items placed in the user's learning environment, such as books, stationery, electronic devices, etc., based on multiple image frames to be detected; at the same time, combined with audio processing technology, audio data is extracted from the video stream, the environmental noise level is analyzed, and its impact on the user's learning is evaluated. In this way, environmental detection provides an additional dimension for the comprehensive evaluation of the user's learning concentration, which helps to identify potential learning interference factors, such as a messy learning desk or excessive noise levels, so as to more accurately evaluate the user's learning concentration.

[0036] For example, if a large number of entertainment items not related to learning are identified in the image and the ambient noise exceeds 60 decibels, this indicates that the user's learning environment is not ideal and the user's learning concentration in subsequent evaluations will be relatively low.

[0037] 104. Use the behavior action detection results and the learning environment detection results to evaluate the user's concentration during learning and obtain a concentration evaluation result.

[0038] In the embodiment of the present application, the online education platform will use the behavioral action detection results and the learning environment detection results to quantify the user's concentration during the learning process. In the evaluation process, multiple factors such as eye activity, behavioral action, placement of items in the learning environment, and noise level will be considered, and the final concentration evaluation result will be obtained through a machine learning algorithm. In this way, through comprehensive evaluation, the actual state of the user in the learning process can be more accurately reflected, so that corresponding measures can be taken to improve learning efficiency and quality.

[0039] The method provided in the embodiment of the present application obtains a video stream collected when a user is learning, extracts multiple image frames to be detected from the video stream, detects the eyes and behavioral actions of the user when learning based on the multiple image frames to be detected, and obtains action detection results; based on the multiple image frames to be detected, detects objects placed in the learning environment where the user is learning; and detects environmental noise based on the video stream to obtain environment detection results; uses the behavioral action detection results and the learning environment detection results to evaluate the concentration of the user when learning to obtain a concentration evaluation result. The method can not only accurately capture the subtle changes in the user's movements during the learning process, but also analyze the impact of external factors such as the noise level and the placement of objects in the learning environment on the user's concentration, thereby achieving a comprehensive and detailed concentration evaluation of the user. While improving the accuracy and intelligence of the evaluation of learning concentration, it can adapt to diversified learning scenarios and reduce the risk of misjudgment, and has good intelligence.

[0040] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, the embodiment of the present application provides another method for evaluating the learning concentration of a user, such as Figure 2A As shown, the method includes:

[0041] 201. Receive a video stream captured by a terminal device held by a user when the user is learning, extract multiple original image frames from the video stream, perform a preprocessing operation on each of the multiple original image frames, and use the preprocessed multiple original image frames as multiple image frames to be detected.

[0042] The embodiment of the present application can be applied to an online education platform. The detailed information of the online education platform can be found in the description of step 101 in the above embodiment, which will not be repeated here.

[0043] Online education platforms provide front-end applications to the outside world, and users can download the front-end applications on their own terminal devices to use the functions of the online education platform. Among them, the terminals held by users, such as smart phones, tablets or special learning devices, are equipped with cameras. In virtual learning scenarios such as online classrooms, in order to achieve face-to-face teaching, the online education platform will request authorization from the user to turn on the camera, and with the user's authorization, obtain the video stream containing the user's learning status captured and transmitted in real time by the user's terminal based on the camera. On the one hand, the video stream can be displayed to the teaching teacher so that the teaching teacher and the user can communicate face to face. On the other hand, it will also serve as the basis for evaluating the user's learning concentration, so that the user can be reminded later based on the concentration evaluation results.

[0044] In order to analyze the user's behavior and learning environment in detail, in an embodiment of the present application, the online education platform will extract multiple original image frames from the video stream. In specific implementation, the online education platform will use corresponding video acquisition software to continuously capture image frames at a certain frame rate (such as 30 frames per second), thereby obtaining multiple original image frames. Multiple original image frames can record the user's learning status in real time and continuously, providing a rich data source for subsequent analysis.

[0045] Considering that the original image frames may be affected by factors such as environmental noise and lighting changes, in order to improve the accuracy of subsequent processing, the online education platform will perform preprocessing operations on each of the multiple original image frames, and use the preprocessed multiple original image frames as multiple image frames to be detected. Among them, the preprocessing operations include but are not limited to denoising operations and color correction operations, which aim to improve image quality and reduce interference factors. The denoising operation can remove random noise in the original image frame to make the image clearer; the color correction operation can ensure the authenticity of the color of the original image frame and avoid color distortion caused by lighting or equipment differences.

[0046] 202. Based on the multiple image frames to be detected, detect the eyes and behavioral actions of the user when learning to obtain an action detection result.

[0047] In an embodiment of the present application, the online education platform will perform detailed detection and analysis of the user's eye state and behavioral movements during learning based on multiple image frames to be detected to obtain comprehensive action detection results.

[0048] Specifically, first, the online education platform will use an eye detection algorithm to determine the user's eye coordinates in each image frame to be detected, and based on the determined eye coordinates, determine the user's eye position information in each image frame to be detected, so as to ensure the accuracy of subsequent eye status detection.

[0049] Next, the online education platform will detect the user's eye status in multiple image frames to be detected one by one according to the eye position information, record multiple time periods when the user's eyes are continuously open, and accumulate multiple time periods to obtain the user's eye gaze duration. Among them, the gaze duration includes the time of staring at the screen and the time of looking away from the screen. The longer the time of staring at the screen, the more focused the user's attention is, and the longer the time of looking away from the screen, it means that the user may be distracted or not focused enough.

[0050] In addition to gaze duration, the online education platform also pays attention to the user's sight trajectory. It will identify the user's sight direction vectors in multiple image frames to be detected one by one according to the eye position information, and connect the multiple sight direction vectors identified according to the time sequence of multiple image frames to be detected to form the user's sight trajectory. The sight trajectory not only reveals the movement of the user's visual focus during the learning process, but also provides important clues for the distribution of the user's interest points.

[0051] At the same time, the online education platform also uses behavioral detection algorithms to determine the user's head position in each image frame to be detected, and calculates the head posture parameters of the user's head based on the angle information of the head position in each image frame to be detected and the coordinates of the key points of the head. The head posture parameters reflect the tilt, rotation and other movements of the user's head, which can be used to deeply analyze the user's learning posture, and can also be combined with other action features (such as hand movements, eye tracking, etc.) in subsequent steps to jointly identify the user's sitting posture and prevent physical problems caused by maintaining a bad posture for a long time. In addition to being able to evaluate sitting posture, head posture parameters can also be used as part of the assessment of the user's learning concentration, because when the user's head posture changes frequently, such as frequently turning or lowering the head, it may mean that their attention is distracted, so combining head posture parameters can improve the accuracy of learning concentration assessment and effectively promote the user's learning efficiency.

[0052] Similarly, the online education platform will also use the behavior detection algorithm to determine the user's hand position in each image frame to be detected, and determine the hand movement characteristics of the user's hand based on the hand position in each image frame to be detected. The hand movement characteristics can carefully depict the user's gesture usage habits during the learning process, including the type of gestures, frequency, and amplitude of the movements. By analyzing the hand movement characteristics, the online education platform can identify whether the user has too many unnecessary gestures, which may distract their attention and affect their learning concentration. Subsequently, the online education platform will also add hand movement characteristics to the assessment of the user's learning concentration in order to further improve the accuracy of the assessment of learning concentration.

[0053] After obtaining the head posture parameters and hand movement features through the above process, the online education platform will detect the user's sitting posture in multiple image frames to be detected based on the head posture parameters and hand movement features to obtain the user's sitting posture information. Specifically, the online education platform will integrate head posture parameters, such as the tilt angle and rotation angle of the head, and hand movement features, such as the type, position, and amplitude of gestures, to form a complete set of user movement data sets. Then, the online education platform will analyze these data to identify the user's sitting posture in different image frames, such as whether it is straight, whether it is leaning forward or backward, whether it is shrugging, etc. On the one hand, the sitting posture information provides intuitive learning posture feedback, which can help discover and correct the user's bad sitting posture, and effectively prevent cervical and lumbar diseases caused by maintaining the same bad posture for a long time; on the other hand, the sitting posture information is also one of the important indicators for evaluating the user's learning concentration, which can reflect to a certain extent whether the user has studied seriously, and further improve the accuracy of the evaluation of learning concentration.

[0054] After completing the above analysis process, the online education platform will use gaze duration, eye track, head posture parameters, hand movement characteristics and sitting posture information as action detection results to facilitate further evaluation of the user's learning concentration.

[0055] 203. Based on the multiple image frames to be detected, objects placed in the learning environment where the user is learning are detected, and environmental noise detection is performed based on the video stream to obtain an environmental detection result.

[0056] In an embodiment of the present application, in addition to detecting the user itself, the online education platform also pays attention to the impact of the learning environment on the user's concentration. Therefore, the online education platform will detect the objects placed in the learning environment where the user is studying based on multiple image frames to be detected, and perform environmental noise detection based on the video stream to obtain environmental detection results.

[0057] Specifically, when detecting the learning environment in which the user is learning, the online education platform will use the object detection algorithm to identify at least one target object placed in the learning environment in which the user is learning in each image frame to be detected, and determine the object information of each target object to construct an object information set. Among them, the object information includes the placement position and object category of the target object, that is, the online education platform will identify various types of objects placed in the user's learning environment, such as books, stationery, electronic devices, etc., and record the placement position and category information of these objects in detail, and integrate them into an object information set.

[0058] At the same time, the online education platform will also extract audio data from the video stream, perform frame processing on the audio data to obtain multiple audio frames, and convert each audio frame into a spectrum representation to more intuitively observe the distribution of audio signals at different frequencies; then use the converted spectrum representation to calculate the audio amplitude of each audio frame, and based on the audio amplitude of each audio frame, construct a noise intensity index and determine the type of noise, such as human voice, mechanical sound, etc.

[0059] After completing the above analysis process, the online education platform will use the item information set, noise intensity index and noise type as the environmental detection results. In this way, on the one hand, through item detection, the online education platform can more clearly understand which items are necessary and which are interference factors in the user's learning environment, so as to evaluate whether the user is focused on learning. On the other hand, environmental noise detection can help the online education platform to detect noise interference in a timely manner, such as whether there is TV sound, chat sound, etc., so that while evaluating whether the user is focused on learning, it can also remind the user to close doors and windows, use noise-cancelling headphones and other measures to reduce noise interference.

[0060] 204. Obtain a concentration assessment model.

[0061] In an embodiment of the present application, in order to realize the evaluation of concentration, a concentration evaluation model is pre-constructed in the online education platform. Among them, the concentration evaluation model is constructed based on multiple evaluation rules, and the concentration evaluation model is used to evaluate the action detection results and the environmental detection results according to multiple evaluation rules to obtain the user's concentration probability. In the process of constructing the concentration evaluation model, a series of evaluation rules are designed to comprehensively measure the user's concentration state. The evaluation rules cover multiple dimensions such as eyes and sight, behavioral actions, desktop item placement, and noise detection. It aims to provide users with accurate concentration evaluation through comprehensive analysis of action detection results and environmental detection results. The evaluation rules are introduced below:

[0062] First, when training the concentration evaluation model, it is necessary to focus on the evaluation rules of eyes and sight. During the learning process, the user's sight should be focused on the screen most of the time, which is one of the key indicators for evaluating concentration. Specifically, the evaluation rules include the gaze rule. The gaze rule indicates that the screen gaze duration per minute should not be less than 45 seconds, and at least 80% of the time in any 10 minutes should be kept on the screen; further, the screen gaze duration also requires that the user's gaze time on the screen should not be less than 10 seconds at a time. The gaze rule also includes the time when the eyes leave the screen. If the user's eyes leave the screen for more than 2 minutes (whether continuous or intermittent accumulation), it is considered that the attention is not focused. At the same time, the sight track is also crucial. The gaze rule requires that the user's sight track should be mainly concentrated in the area directly related to learning, and avoid frequent inspections of the surrounding environment and objects. If the user's sight frequently patrols or switches focus (more than 3 times per minute and cumulatively more than 10 minutes / hour), it will also be recorded as inattention.

[0063] Next, we need to continue to pay attention to the behavioral action rules. The stability of head posture is one of the manifestations of focused learning, and users should avoid frequently swinging left and right or lowering their heads. If there is a significant forward leaning, backward leaning, or left and right tilting movement for a long time (more than 2 minutes), the user needs to be reminded to adjust the posture. At the same time, hand movements are also an important aspect of assessing concentration. Therefore, the behavioral action rules will stipulate that users should keep their hands in the work area for related learning activities. If they frequently leave the desktop or paper and pen area (more than 10 times / hour), it will be regarded as distraction. In addition, based on the behavioral action rules, the online education platform can also remind users to avoid meaningless hand movements, such as frequently touching the face and organizing items. Furthermore, the behavioral action rules also stipulate the sitting posture. Users should maintain an upright sitting posture with the back of the chair in contact with the back of the chair to prevent leaning forward or leaning for a long time. Every hour, users should try to avoid more than 5 significant changes in sitting posture.

[0064] In terms of the placement of items on the desk, evaluation rules, namely the rules for the placement of items, are also set. The rules for the placement of items indicate that the work area must be kept clean and empty, and no sundries should be placed except for necessary learning tools. At no time should items unrelated to learning, such as mobile phones, snacks, etc., appear in the work area. Furthermore, the rules for the placement of items also indicate that the number of stationery should be controlled within 10 pieces and concentrated in the stationery area; no more than 5 books of the same category, and they must be placed in the book area; the total number of sundries should not exceed 3 pieces, and they should be placed in the sundries area to avoid occupying the work area. In addition, the rules for the placement of items also indicate that the stacking height of books should not exceed 15 cm, and the stacking height of paper should not exceed 3 cm to ensure that the desktop is clean and orderly.

[0065] Finally, noise detection rules are also an integral part of assessing concentration. Noise detection rules indicate that if the ambient noise is too loud or there are other significant sources of interference (such as television, noisy conversations), it is necessary to mark the current user's learning environment as having excessive noise. In addition, the online education platform will assess the user's concentration and remind the user to avoid these situations to avoid affecting concentration.

[0066] In summary, these evaluation rules together constitute the basis of the concentration evaluation model. By comprehensively analyzing the user's eyes and sight, behavioral actions, desktop object placement, and noise detection results, the concentration evaluation model can provide users with accurate concentration evaluation and help them improve their learning efficiency and quality.

[0067] In an embodiment of the present application, after the setting of the evaluation rules is completed through the above process, the training of the concentration evaluation model can be started. Among them, in order to train the concentration evaluation model more accurately, user learning data is randomly captured, and 2000 hours of user learning behavior data are accumulated. For users in the test sample set, it is necessary to manually judge whether the user's learning behavior is focused one by one, and the result of manual judgment is used as the benchmark result of model judgment. Next, the embodiment of the present application uses a logistic regression model to detect user learning concentration, and uses the various contents in the above rule definition as specific features in this model, such as: gaze rule X1, gaze duration X2, etc. The training process of the concentration evaluation model is described below, which specifically includes the following steps one to four:

[0068] Step 1: Define the model function.

[0069] Logistic regression is a generalized linear model that is widely used in binary classification problems. In logistic regression, the Sigmoid function plays a key role. It converts the prediction results of the linear combination into a probability form, thereby making a probability prediction for the classification problem. Therefore, in the embodiment of the present application, the model function is defined as the Sigmoid function, and the assumed function form of logistic regression is as follows:

[0070] Formula 1: h θ (X) = g(θ T X)

[0071] Wherein, g(z) in Formula 1 is the Sigmoid function, and its expression is shown in the following Formula 2:

[0072] Formula 2:

[0073] Where z is the inner product of the feature vector X and the parameter vector θ, which is as shown in the following formula 3:

[0074] Formula 3: z=θ0+θ1X1+θ2X2+...+θ n X n

[0075] Therefore, the hypothesis function of logistic regression can be inferred as the following formula 4:

[0076] Formula 4:

[0077] In formula 4, h θ (X) represents the probability that the sample belongs to the positive class when the input X and parameter θ are given, that is, the probability that the user is focused; X1, X2, …, X n are the eigenvalues ​​of the input data; θ1, θ2, …, θ n are the weights or coefficients corresponding to these features; θ0 is the intercept term or bias term, when all eigenvalues ​​X1, X2, …, X n When both are 0, it gives the base level of the prediction results.

[0078] Step 2: Define the loss function.

[0079] In logistic regression, the embodiment of the present application uses the log-likelihood loss function to measure the difference between the probability distribution predicted by the focus assessment model and the actual label distribution. The log-likelihood loss function not only helps to evaluate the quality of the model, but its optimization process is equivalent to finding parameter estimates that maximize the likelihood of the observed data. For a binary classification problem, the goal of logistic regression is to determine the optimal parameter θ by minimizing the log-likelihood loss.

[0080] Among them, the log-likelihood loss function can be expressed as the following formula 5:

[0081] Formula 5:

[0082] In Formula 5, m is the number of samples; y (i) is the actual label of the ith sample, which can take the value of 0 or 1, 0 is the negative category (indicating that the user is not focused on learning), and 1 is the positive category (indicating that the user is focused on learning); x (i) represents the feature vector of the i-th sample; h θ (x (i) ) is the predicted output given by the logistic regression model, that is, given the feature vector x (i) The conditional probability of the i-th sample input being a positive class under the condition of θ and parameter θ.

[0083] Maximum likelihood estimation is a commonly used statistical method for estimating unknown parameters in probability models. Specifically, in logistic regression, it is hoped to find a set of parameters θ so that the likelihood function under a known training set reaches the maximum value. The likelihood function expresses the possibility of observing specific data given the parameters. Since directly processing the likelihood function may involve a large number of product operations of small probability values, which will lead to numerical instability problems, the present embodiment of the application turns to the log-likelihood function, that is, taking the natural logarithm of the original likelihood function, which does not change the position of the extreme point, and converts the continuous multiplication into continuous addition, thereby simplifying the calculation. In addition, for the convenience of optimization, the present embodiment of the application changes the target from maximizing the log-likelihood to minimizing the negative log-likelihood, so that there is a negative sign in the log-likelihood loss function. Minimizing the negative log-likelihood is actually looking for a set of parameters θ that can maximize the probability of the observed data. This method can ensure that the model reflects the pattern in the training data as accurately as possible.

[0084] In order to find the parameter θ that minimizes J(θ), the embodiments of the present application use a variety of optimization algorithms, such as gradient descent, Newton's method or other convex optimization techniques. The basic idea of ​​these optimization algorithms is to iteratively adjust the parameter value until the loss function is no longer significantly reduced. In each iteration, the gradient direction calculated according to the current parameters guides how to update the parameters in the next step to gradually approach the global optimal solution. It is worth noting that because the log-likelihood loss is a convex function about θ, as long as the appropriate step size and initial point are selected, it can be guaranteed to converge to the global optimal point in the end.

[0085] Thus, through the log-likelihood loss function, logistic regression not only provides an effective means of classification, but also provides a clear framework for parameter estimation and model optimization.

[0086] 3. Minimize the loss function.

[0087] In order to minimize the loss function and find the optimal model parameter θ, the embodiment of the present application adopts the gradient descent method as the optimization algorithm. The core idea of ​​the gradient descent method is to continuously adjust the parameter value in an iterative manner, and search along the direction of the negative gradient of the objective function (ie, the loss function) until it converges to a local or global minimum. Among them, the gradient is a vector, which indicates that the directional derivative of a function at this point achieves the maximum value along this direction. For a multivariate function f(θ), its gradient Δf(θ) is a vector pointing to the direction in which the function value grows fastest. Therefore, in order to find the minimum value of the function, the parameters should be updated in the opposite direction of the gradient. In an embodiment of the present application, in each iteration, it is necessary to calculate the gradient of the loss function with respect to the parameter according to the current parameter value, initialize θ, set θ to all 1, and then calculate the gradient in each step of the algorithm iteration, and move along the direction of the gradient to change the parameter θ until the fitting effect of θ reaches the required value or the number of iterations reaches the set value, that is, update the parameter θ according to the following formula 6:

[0088] Formula 6:

[0089] Where, in Formula 6, θ j represents the jth parameter; α is the learning rate (also called step size), which determines the distance along the gradient direction at each update. In practical applications, it is necessary to select a suitable learning rate. If the learning rate is too large, the minimum value may be missed, and if the learning rate is too small, the convergence speed is too slow; Represents the loss function J(θ) for parameter θ j The partial derivative of gives the direction in which the loss function changes most rapidly at the current point.

[0090] 4. Verify repeatedly.

[0091] In an embodiment of the present application, steps two and three need to be repeated until the loss function converges, or the concentration evaluation model is obtained as expected. In addition to relying on the change in the loss function to determine whether the model converges, another important consideration is the classification accuracy of the model. In an embodiment of the present application, the concentration evaluation model needs to reach an accuracy of at least 85% before it can be put into use, that is, in the testing phase, the rate at which the concentration evaluation model correctly identifies whether a user is a focused user should not be less than 85%. In order to evaluate this, the performance of the model can be tested on a set of data independent of the training set (i.e., the test set), and the corresponding accuracy, precision, recall and other related indicators can be calculated. Only when all key performance indicators meet business requirements can it be determined that the concentration evaluation model has reached the standard for deployment.

[0092] Furthermore, the focus model will output the user's focus probability for the input content, where the predicted focus probability is greater than or equal to the preset probability threshold of 0.6, and the user is judged as a focused user; otherwise, the user is considered a non-focused user. The setting of the preset probability threshold can refer to the needs of the specific application scenario and the cost of misclassification. For example, if the negative impact of mistakenly marking a non-focused user as a focused user is large, a higher preset probability threshold can be selected to reduce false positives; vice versa. Therefore, choosing an appropriate threshold is a process of balancing accuracy and other business goals.

[0093] In this way, through the above systematic and repeated verification process, it can be ensured that the trained focus evaluation model has both sufficient accuracy and good generalization ability, so as to be effectively used to distinguish between focused users and non-focused users.

[0094] 205. Calling a concentration evaluation model, and using a plurality of first evaluation rules among a plurality of evaluation rules that match the action detection result, to evaluate the action detection result, and obtain a first evaluation result.

[0095] In the embodiment of the present application, the online education platform calls the concentration evaluation model, uses multiple first evaluation rules that match the action detection results in multiple evaluation rules, evaluates the action detection results, and obtains a first evaluation result. The process of generating the first evaluation result is described below:

[0096] First, the online education platform obtains the gaze duration in the action detection result, and evaluates whether the gaze duration and the time when the user's eyes leave the screen indicated by the gaze duration are focused by calling the concentration evaluation model and based on the gaze rules in the multiple first evaluation rules, and determines whether the user's eyes are mainly focused on the screen and whether the time when the user's eyes leave the screen is too long. According to the evaluation results obtained, the online education platform will mark the gaze duration with a gaze duration category label, wherein, when the evaluation results obtained indicate that the user's attention is not focused, it is necessary to mark the gaze duration with a category label for indicating non-focused learning. Referring to the content in the above step 204, it can be known that when training the concentration evaluation model, the non-focused state is marked as 0, therefore, the gaze duration category label with a value of 0 can be used for marking here; on the contrary, when the evaluation results obtained indicate that the user's attention is focused, the gaze duration is marked with a category label for indicating focused learning, that is, the gaze duration category label with a value of 1 is used for marking.

[0097] At the same time, the online education platform also pays attention to the user's gaze trajectory, obtains the gaze trajectory in the action detection results, and evaluates whether the gaze trajectory is concentrated in the area related to learning by calling the concentration evaluation model and based on the gaze rules. According to the evaluation results obtained, the online education platform will label the gaze trajectory with a gaze trajectory category label, wherein, when the evaluation results obtained indicate that the user's gaze is not concentrated in the area related to learning, the gaze trajectory is labeled with a category label used to indicate non-focused learning, and when the evaluation results obtained indicate that the user's gaze is concentrated in the area related to learning, the gaze trajectory is labeled with a category label used to indicate focused learning. Similar to the gaze duration category label, it can also be assigned a value of 0 or 1 to indicate whether the user is focused, which will not be repeated here.

[0098] In addition, the online education platform will also conduct a comprehensive assessment of the user's head posture, hand movement characteristics and sitting posture information. The online education platform will obtain the head posture parameters in the action detection results, and evaluate whether the head posture parameters are stable and standard by calling the concentration evaluation model and based on the behavioral action rules in multiple first evaluation rules. According to the obtained evaluation results, the head posture parameters are labeled with head posture category labels. When the obtained evaluation results indicate that the user's head posture is unstable or non-standard, the head posture is labeled with a category label for indicating non-focused learning, and when the obtained evaluation results indicate that the user's head posture is stable and standard, the head posture is labeled with a category label for indicating focused learning. Similar to the gaze duration category label, it can also be assigned a value of 0 or 1 to indicate whether the user is focused, which will not be repeated here.

[0099] Next, the online education platform will continue to obtain hand motion features in the motion detection results, and evaluate whether the hand motion features indicate that the user's hands are performing activities related to learning based on the behavioral action rules. The hand motion features are labeled with hand motion category labels based on the evaluation results. When the evaluation results indicate that the user's hands are not performing activities related to learning, the hand motion features are labeled with category labels indicating non-focused learning. When the evaluation results indicate that the user's hands are performing activities related to learning, the hand motion features are labeled with category labels indicating focused learning. Similar to the gaze duration category label, it can also be assigned a value of 0 or 1 to indicate whether the user is focused, which will not be elaborated here.

[0100] Furthermore, the sitting posture information is continued to be obtained from the action detection results, and the frequency of changes in the user's sitting posture indicated by the sitting posture information is evaluated by calling the concentration evaluation model and based on the behavioral action rules, and the sitting posture information is labeled with a sitting posture category label according to the obtained evaluation result. When the obtained evaluation result indicates that the frequency of changes in the user's sitting posture is greater than or equal to the frequency threshold, the sitting posture information is labeled with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that the frequency of changes in the user's sitting posture is less than the frequency threshold, the sitting posture information is labeled with a category label for indicating focused learning. Similar to the gaze duration category label, it can also be assigned a value of 0 or 1 to indicate whether the user is focused, which will not be repeated here.

[0101] It should be noted that the concentration evaluation model will use the gaze duration, eye trajectory, head posture parameters, hand movement characteristics and sitting posture information that have completed the category labeling as the first evaluation result. Since the concentration evaluation model is trained based on multiple evaluation rules, after the user's behavior and action detection results are input into the concentration evaluation model, the concentration evaluation model will automatically evaluate the concentration and label the category label to prepare for the subsequent calculation of the concentration probability.

[0102] 206. Calling the concentration evaluation model, using multiple second evaluation rules among the multiple evaluation rules that match the environment detection result, evaluate the environment detection result to obtain a second evaluation result.

[0103] In the embodiment of the present application, the online education platform calls the concentration evaluation model, uses multiple second evaluation rules that match the environment detection results in multiple evaluation rules, evaluates the environment detection results, and obtains the second evaluation results. The process of generating the second evaluation results is described below:

[0104] First, the online education platform obtains the item information set in the environmental detection results, and evaluates whether there are target items unrelated to learning, such as toys, game consoles, etc., in the item information set by calling the concentration evaluation model and based on the item placement rules in the multiple second evaluation rules, and evaluates whether the placement of target items related to learning (such as books, stationery, etc.) in the item information set meets the standards indicated by the item placement rules. According to the obtained evaluation results, the item information set is labeled with item information category labels. Among them, when the obtained evaluation results indicate that there are target items unrelated to learning or the placement of target items related to learning does not meet the standards indicated by the item placement rules, the item information set is labeled with a category label for indicating non-focused learning, and when the obtained evaluation results indicate that there are no target items unrelated to learning and the placement of target items related to learning meets the standards indicated by the item placement rules, the item information set is labeled with a category label for indicating focused learning. Similar to the gaze duration category label in the above step 205, it can also be assigned a value of 0 or 1 to indicate whether the user is focused, which will not be repeated here.

[0105] In addition, noise intensity index and noise type are also important components of the environmental detection results. Noise not only affects the user's attention, but may also have a negative impact on the user's emotions and learning efficiency. Therefore, the online education platform will obtain the noise intensity index in the environmental detection results, by calling the concentration evaluation model, and based on the noise detection rules in the multiple second evaluation rules, evaluate whether the noise in the user's learning environment exceeds the standard, and mark the noise intensity category label for the noise intensity index according to the obtained evaluation results. Among them, when the obtained evaluation results indicate that the noise in the user's learning environment exceeds the standard, the noise intensity index is marked with a category label for indicating non-focused learning, and when the obtained evaluation results indicate that the noise in the user's learning environment does not exceed the standard, the noise intensity index is marked with a category label for indicating focused learning. Similar to the gaze duration category label in the above step 205, it can also be assigned with 0 or 1 to indicate whether the user is focused, which will not be repeated here.

[0106] At the same time, the online education platform will also analyze the noise type to determine whether there are potential interference sources, such as phone ringing, TV sound, chat sound, etc. That is, the online trading platform will obtain the noise type in the environmental detection results, call the concentration evaluation model based on the noise detection rules, evaluate whether there are interference sources in the user's learning environment, and mark the noise type with the interference source category label according to the obtained evaluation results. Among them, when the obtained evaluation results indicate that there are interference sources in the user's learning environment, the noise type is marked with a category label for indicating non-focused learning, and when the obtained evaluation results indicate that there are no interference sources in the user's learning environment, the noise type is marked with a category label for indicating focused learning. Similar to the gaze duration category label in the above step 205, it can also be assigned a value of 0 or 1 to indicate whether the user is focused, which will not be repeated here.

[0107] It should be noted that the concentration evaluation model will use the item information set, noise intensity index and noise type that have completed the category labeling as the second evaluation result. Since the concentration evaluation model is trained based on multiple evaluation rules, after the user's learning environment detection results are input into the concentration evaluation model, the concentration evaluation model will automatically evaluate the concentration and label the category labels to prepare for the subsequent calculation of the concentration probability.

[0108] 207. Call the concentration evaluation model, use the concentration probability calculation formula to calculate the first evaluation result and the second evaluation result, predict the user's concentration probability when learning, determine the preset probability threshold, compare the concentration probability with the preset probability threshold, and obtain a probability comparison result.

[0109] In the embodiment of the present application, the online education platform will use the concentration probability calculation formula to predict the user's concentration probability when learning by calling the concentration evaluation model, combining the first evaluation result and the second evaluation result. Among them, the online education platform will use the concentration evaluation model to comprehensively analyze the user's behavior characteristics, and finally determine whether the user is a focused user based on a preset probability threshold.

[0110] First, the online education platform will obtain the first evaluation results and the second evaluation results. The two evaluation results together constitute a comprehensive description of the user's behavior and environment during learning.

[0111] Next, the online education platform will input the first and second evaluation results into the concentration evaluation model. The concentration evaluation model will calculate a probability value reflecting the user's current concentration level based on the internal learning mechanism and weight distribution, that is, the concentration probability.

[0112] After obtaining the concentration probability, the online education platform will compare it with the preset probability threshold. If the calculated probability is greater than or equal to the preset probability threshold, it is considered that the user has shown sufficient concentration during the learning process and can be judged as a focused user, and then the probability comparison result is generated; otherwise, if the probability is lower than the preset probability threshold, it means that the user's attention is not focused enough and should be regarded as a non-focused user, and then the probability comparison result is generated.

[0113] In this way, through the deep integration of the first evaluation results and the second evaluation results, and with the help of the analytical capabilities of the concentration assessment model, an accurate quantitative evaluation of students' learning status can be achieved, ensuring that the assessment of students' concentration is more accurate.

[0114] 208. The concentration probability and probability comparison result are used as the concentration evaluation result.

[0115] In the embodiment of the present application, after obtaining the concentration probability and probability comparison results, the online education platform will use the concentration probability and probability comparison results as the concentration evaluation results, thereby providing an objective and quantitative concentration evaluation standard. In the process of actual application, through regular or real-time evaluation, the online education platform can clearly understand the user's concentration state, so as to adjust the learning strategy in a targeted manner and improve learning efficiency. At the same time, teachers or parents can also use this evaluation result to more accurately understand and intervene in the student's learning status, and provide students with more personalized learning support and guidance.

[0116] For example, suppose a student is studying mathematics on his own. The online education platform calculates his concentration probability as 85% through real-time monitoring and analysis, while the preset concentration standard is 90%. At this time, the probability comparison result shows that the student's concentration does not meet the preset standard. Based on this evaluation result, the online education platform can remind students that there may be some distraction during self-study, so that students can adjust their learning status, such as turning off their mobile phones, adjusting their learning environment, etc., to improve their concentration. At the same time, teachers can also provide students with more targeted learning suggestions or guidance based on the results of their concentration evaluation.

[0117] In this way, through the concentration evaluation model, the embodiment of the present application can handle and identify diversified learning scenarios, including self-study, doing exercises and attending classes, effectively improving the comprehensiveness and accuracy of identification. Combined with image processing technology, the online education platform can intelligently identify the user's learning behavior, realize the automation and intelligence of reminding students to focus on learning, greatly simplify and reduce the teaching cost of teachers and improve the learning efficiency of students. In summary, the logical process in this application is summarized as follows:

[0118] like Figure 2BAs shown in the figure, the online education platform will obtain and preprocess the video stream collected when the user is learning to obtain multiple image frames to be detected. Then, the multiple image frames to be detected are detected, and eye detection and gaze analysis, behavior and sitting posture detection, desktop object recognition and environmental noise analysis are performed based on the multiple image frames to be detected. Then, these detection results are integrated, and the concentration evaluation model is used to perform a comprehensive concentration evaluation on these detection results to obtain the concentration evaluation result and end the process.

[0119] The method provided in the embodiment of the present application obtains a video stream collected when a user is learning, extracts multiple image frames to be detected from the video stream, detects the eyes and behavioral actions of the user when learning based on the multiple image frames to be detected, and obtains action detection results; based on the multiple image frames to be detected, detects objects placed in the learning environment where the user is learning; and detects environmental noise based on the video stream to obtain environment detection results; uses the behavioral action detection results and the learning environment detection results to evaluate the concentration of the user when learning to obtain a concentration evaluation result. The method can not only accurately capture the subtle changes in the user's movements during the learning process, but also analyze the impact of external factors such as the noise level and the placement of objects in the learning environment on the user's concentration, thereby achieving a comprehensive and detailed concentration evaluation of the user. While improving the accuracy and intelligence of the evaluation of learning concentration, it can adapt to diversified learning scenarios and reduce the risk of misjudgment, and has good intelligence.

[0120] Further, as Figure 1 In a specific implementation of the method, the present application provides a device for evaluating a user's learning concentration, such as Figure 3 As shown, the device includes: an extraction module 301, a detection module 302 and an evaluation module 303.

[0121] The extraction module 301 is used to obtain a video stream collected when the user is learning, and extract a plurality of image frames to be detected from the video stream;

[0122] The detection module 302 is used to detect the eyes and behavioral actions of the user when learning based on the multiple image frames to be detected to obtain an action detection result;

[0123] The detection module 302 is further used to detect objects placed in the learning environment where the user is learning based on the multiple image frames to be detected, and to perform environmental noise detection based on the video stream to obtain an environmental detection result;

[0124] The evaluation module 303 is used to evaluate the user's concentration during learning by using the behavior detection result and the learning environment detection result to obtain a concentration evaluation result.

[0125] In a specific application scenario, the extraction module 301 is used to receive the video stream captured by the terminal device held by the user when the user is learning, and extract multiple original image frames from the video stream; perform a preprocessing operation on each of the multiple original image frames, and use the preprocessed multiple original image frames as the multiple image frames to be detected, wherein the preprocessing operation includes but is not limited to a denoising operation and a color correction operation.

[0126] In a specific application scenario, the detection module 302 is used to determine the eye coordinates of the user in each of the image frames to be detected by using an eye detection algorithm, and determine the eye position information of the user in each of the image frames to be detected based on the determined eye coordinates; detect the eye states of the user's eyes in the multiple image frames to be detected one by one according to the eye position information, record multiple time periods in which the user's eyes are continuously open, and accumulate the multiple time periods to obtain the gaze duration of the user's eyes; identify the sight direction vectors of the user's eyes in the multiple image frames to be detected one by one according to the eye position information, and connect the identified multiple sight direction vectors according to the temporal sequence of the multiple image frames to be detected to form a sight track of the user's eyes; Using a behavior detection algorithm, the head position of the user is determined in each of the image frames to be detected, and the head posture parameters of the user's head are calculated based on the angle information of the head position in each of the image frames to be detected and the coordinates of the head key points; using the behavior detection algorithm, the hand position of the user is determined in each of the image frames to be detected, and the hand movement characteristics of the user's hand are determined based on the hand key point coordinates and hand posture of the hand position in each of the image frames to be detected; based on the head posture parameters and the hand movement characteristics, the sitting posture of the user in the multiple image frames to be detected is detected to obtain the sitting posture information of the user; the gaze duration, the line of sight trajectory, the head posture parameters, the hand movement characteristics and the sitting posture information are used as the action detection result.

[0127] In a specific application scenario, the detection module 302 is used to use an object detection algorithm to identify at least one target object placed in the learning environment where the user is learning in each image frame to be detected, and determine the object information of each target object to construct an object information set, wherein the object information includes the placement position and object category of the target object; extract audio data from the video stream, perform frame processing on the audio data to obtain multiple audio frames, and convert each of the audio frames into a spectrum representation; calculate the audio amplitude of each audio frame using the spectrum representation obtained by the conversion, construct a noise intensity index based on the audio amplitude of each audio frame, and determine the noise type; use the object information set, the noise intensity index and the noise type as the environment detection result.

[0128] In a specific application scenario, the evaluation module 303 is used to obtain a concentration evaluation model, wherein the concentration evaluation model is constructed based on multiple evaluation rules, and the concentration evaluation model is used to evaluate the action detection result and the environment detection result according to the multiple evaluation rules to obtain the user's concentration probability; the concentration evaluation model is called, and the action detection result is evaluated using multiple first evaluation rules in the multiple evaluation rules that match the action detection result to obtain a first evaluation result; the concentration evaluation model is called, and the environment detection result is evaluated using multiple second evaluation rules in the multiple evaluation rules that match the environment detection result to obtain a second evaluation result; the concentration evaluation model is called, and the first evaluation result and the second evaluation result are calculated using the concentration probability calculation formula to predict the user's concentration probability when learning; a preset probability threshold is determined, and the concentration probability is compared with the preset probability threshold to obtain a probability comparison result, and the probability comparison result is used to indicate whether the user is judged as a focused user when learning; the concentration probability and the probability comparison result are used as the concentration evaluation result.

[0129] In a specific application scenario, the evaluation module 303 is used to obtain the gaze duration in the action detection result, call the concentration evaluation model to evaluate whether the gaze duration indicated by the gaze duration and the duration of the gaze away from the screen are focused based on the gaze rule in the multiple first evaluation rules, and mark the gaze duration with a gaze duration category label according to the obtained evaluation result, wherein, when the obtained evaluation result indicates that the user's attention is not focused, the gaze duration is marked with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that the user's attention is focused, the gaze duration is marked with a category label for indicating non-focused learning. Marking a category label for indicating focused learning; obtaining a gaze trajectory from the action detection result, calling the concentration evaluation model to evaluate whether the gaze trajectory is concentrated in an area related to learning based on the gaze rule, and marking the gaze trajectory with a gaze trajectory category label according to the obtained evaluation result, wherein when the obtained evaluation result indicates that the user's gaze is not concentrated in an area related to learning, the gaze trajectory is marked with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that the user's gaze is concentrated in an area related to learning, the gaze trajectory is marked with a category label for indicating focused learning; obtaining head posture parameters from the action detection result, calling the concentration evaluation model to evaluate whether the head posture parameters are stable and standard based on the behavioral action rules in the multiple first evaluation rules, and marking the head posture category label for the head posture parameters according to the obtained evaluation result, wherein when the obtained evaluation result indicates that the user's head posture is unstable or non-standard, the head posture is marked with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that the user's head posture is stable and standard, the head posture is marked with a category label for indicating focused learning; in the action detection result, Obtain hand movement features from the detection results, call the concentration evaluation model to evaluate whether the hand movement features indicate that the user's hands are performing activities related to learning based on the behavioral action rules, and label the hand movement features with hand movement category labels based on the obtained evaluation results, wherein when the obtained evaluation results indicate that the user's hands are not performing activities related to learning, the hand movement features are labeled with category labels for indicating non-focused learning, and when the obtained evaluation results indicate that the user's hands are performing activities related to learning, the hand movement features are labeled with category labels for indicating focused learning;Acquire sitting posture information from the action detection result, call the concentration evaluation model to evaluate the frequency of changes in the user's sitting posture indicated by the sitting posture information based on the behavioral action rule, and mark the sitting posture information with a sitting posture category label according to the obtained evaluation result, wherein when the obtained evaluation result indicates that the frequency of changes in the user's sitting posture is greater than or equal to the frequency threshold, mark the sitting posture information with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that the frequency of changes in the user's sitting posture is less than the frequency threshold, mark the sitting posture information with a category label for indicating focused learning; use the gaze duration, the sight line trajectory, the head posture parameter, the hand movement feature and the sitting posture information marked with the category label as the first evaluation result. ;

[0130] In a specific application scenario, the evaluation module 303 is used to obtain an item information set from the environmental detection result, call the concentration evaluation model based on the item placement rules in the multiple second evaluation rules, evaluate whether there are target items unrelated to learning in the item information set, and evaluate whether the placement of target items related to learning in the item information set meets the standards indicated by the item placement rules, and mark the item information set with item information category labels according to the obtained evaluation results, wherein, when the obtained evaluation results indicate that there are target items unrelated to learning or the placement of target items related to learning does not meet the standards indicated by the item placement rules, the item information set is marked with a category label for indicating non-focused learning, and when the obtained evaluation results indicate that there are no target items unrelated to learning and the placement of target items related to learning meets the standards indicated by the item placement rules, the item information set is marked with a category label for indicating focused learning; obtain a noise intensity index from the environmental detection result, call the concentration evaluation model based on the noise detection rules in the multiple second evaluation rules, and evaluate the user's learning status. Whether the noise in the environment exceeds the standard, and according to the obtained evaluation result, the noise intensity category label is marked for the noise intensity index, wherein when the obtained evaluation result indicates that the noise in the learning environment of the user exceeds the standard, the noise intensity index is marked with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that the noise in the learning environment of the user does not exceed the standard, the noise intensity index is marked with a category label for indicating focused learning; the noise type is obtained from the environmental detection result, and the concentration evaluation model is called to evaluate whether there is an interference source in the learning environment of the user based on the noise detection rule, and according to the obtained evaluation result, the interference source category label is marked for the noise type, wherein when the obtained evaluation result indicates that there is an interference source in the learning environment of the user, the noise type is marked with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that there is no interference source in the learning environment of the user, the noise type is marked with a category label for indicating focused learning; the object information set, the noise intensity index and the noise type with completed category label marking are used as the second evaluation result.

[0131] The device provided in the embodiment of the present application obtains a video stream collected when a user is learning, extracts multiple image frames to be detected from the video stream, detects the eyes and behavioral actions of the user when learning based on the multiple image frames to be detected, and obtains action detection results; based on the multiple image frames to be detected, detects objects placed in the learning environment where the user is learning; and detects environmental noise based on the video stream to obtain environment detection results; uses the behavioral action detection results and the learning environment detection results to evaluate the concentration of the user when learning, and obtains a concentration evaluation result. The device can not only accurately capture the subtle changes in the user's movements during the learning process, but also analyze the impact of external factors such as the noise level and the placement of objects in the learning environment on the user's concentration, thereby achieving a comprehensive and detailed concentration evaluation of the user. While improving the accuracy and intelligence of the evaluation of learning concentration, it can adapt to diversified learning scenarios and reduce the risk of misjudgment, and has good intelligence.

[0132] It should be noted that for other corresponding descriptions of the functional units involved in the device for evaluating the learning concentration of a user provided in the embodiment of the present application, reference can be made to Figure 1 and FIG. 2A to FIG. 2B The corresponding description in will not be repeated here.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0134] The above embodiments and the technical features in the embodiments can be arbitrarily combined with each other. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0136] In an exemplary embodiment, see Figure 4, and also provides a computer device, which includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device, wherein each functional unit can communicate with each other through the bus. The memory stores a computer program, and the processor is used to execute the program stored in the memory and execute the user's learning concentration evaluation method in the above embodiment.

[0137] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for evaluating the learning concentration of a user.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0139] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present application.

[0140] Those skilled in the art will appreciate that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more devices different from the implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0141] The above application serial numbers are for description only and do not represent the advantages or disadvantages of the implementation scenarios.

[0142] The above disclosure only discloses several specific implementation scenarios of the present application. However, the present application is not limited thereto, and any changes that can be conceived by technicians in this field should fall within the scope of protection of the present application.

Claims

1. A method for evaluating a user's learning concentration, characterized in that: include: Acquire a video stream collected when the user is learning, and extract a plurality of image frames to be detected from the video stream; Based on the multiple image frames to be detected, detecting the eyes and behavioral actions of the user during learning to obtain an action detection result; Based on the multiple image frames to be detected, objects placed in the learning environment where the user is learning are detected, and environmental noise detection is performed based on the video stream to obtain an environmental detection result; The user's concentration during learning is evaluated using the behavior detection result and the learning environment detection result to obtain a concentration evaluation result.

2. The method according to claim 1, characterized in that The step of acquiring a video stream collected when the user is learning and extracting a plurality of image frames to be detected from the video stream includes: Receiving the video stream collected by the terminal device held by the user when the user is learning, and extracting multiple original image frames from the video stream; A preprocessing operation is performed on each of the multiple original image frames, and the multiple original image frames after preprocessing are used as the multiple image frames to be detected, wherein the preprocessing operation includes but is not limited to a denoising operation and a color correction operation.

3. The method according to claim 1, characterized in that The detecting of the eyes and behavioral actions of the user during learning based on the multiple image frames to be detected to obtain an action detection result includes: Determine eye coordinates of the user in each of the image frames to be detected by using an eye detection algorithm, and determine eye position information of the user in each of the image frames to be detected based on the determined eye coordinates; According to the eye position information, detect the eye states of the user's eyes in the multiple image frames to be detected one by one, record multiple time periods in which the user's eyes are continuously open, and accumulate the multiple time periods to obtain the gaze duration of the user's eyes; According to the eye position information, the sight line direction vectors of the user's eyes in the multiple image frames to be detected are identified one by one, and according to the temporal sequence of the multiple image frames to be detected, the identified multiple sight line direction vectors are connected to form a sight line track of the user's eyes; Determine the head position of the user in each of the image frames to be detected by using a behavior detection algorithm, and calculate the head posture parameters of the user's head according to the angle information of the head position in each of the image frames to be detected and the coordinates of the key points of the head; Determine the hand position of the user in each of the image frames to be detected by using the behavior detection algorithm, and determine the hand motion features of the user's hand according to the hand key point coordinates and hand posture of the hand position in each of the image frames to be detected; Based on the head posture parameter and the hand movement feature, detecting the sitting posture of the user in the multiple image frames to be detected to obtain the sitting posture information of the user; The gaze duration, the sight line trajectory, the head posture parameter, the hand movement feature and the sitting posture information are used as the movement detection result.

4. The method according to claim 1, characterized in that The detecting of objects placed in the learning environment where the user is learning based on the multiple image frames to be detected, and the detecting of environmental noise based on the video stream to obtain an environmental detection result include: Using an object detection algorithm, identifying at least one target object placed in the learning environment where the user is learning in each of the image frames to be detected, and determining the object information of each of the target objects to construct an object information set, wherein the object information includes the placement position and object category of the target object; Extracting audio data from the video stream, performing frame processing on the audio data to obtain a plurality of audio frames, and converting each of the audio frames into a frequency spectrum representation; Calculating the audio amplitude of each of the audio frames using the spectrum representation obtained by the conversion, constructing a noise intensity index according to the audio amplitude of each of the audio frames, and determining the noise type; The object information set, the noise intensity index and the noise type are used as the environment detection result.

5. The method according to claim 1, characterized in that The step of evaluating the user's concentration during learning by using the behavior detection result and the learning environment detection result to obtain a concentration evaluation result includes: Acquire a concentration evaluation model, wherein the concentration evaluation model is constructed based on a plurality of evaluation rules, and the concentration evaluation model is used to evaluate the action detection result and the environment detection result according to the plurality of evaluation rules to obtain the concentration probability of the user; Calling the concentration evaluation model, and using a plurality of first evaluation rules among the plurality of evaluation rules that match the action detection result to evaluate the action detection result, to obtain a first evaluation result; Calling the concentration evaluation model, and using a plurality of second evaluation rules among the plurality of evaluation rules that match the environment detection result to evaluate the environment detection result, to obtain a second evaluation result; Calling the concentration evaluation model, calculating the first evaluation result and the second evaluation result using a concentration probability calculation formula, and predicting the concentration probability of the user when studying; Determine a preset probability threshold, compare the concentration probability with the preset probability threshold, and obtain a probability comparison result, wherein the probability comparison result is used to indicate whether the user is judged as a focused user when learning; The concentration probability and the probability comparison result are used as the concentration evaluation result.

6. The method according to claim 5, characterized in that The calling of the concentration evaluation model, using a plurality of first evaluation rules among the plurality of evaluation rules that match the action detection result, to evaluate the action detection result, and obtaining a first evaluation result, comprises: Obtaining the gaze duration from the action detection result, calling the concentration evaluation model to evaluate whether the gaze duration indicated by the gaze duration and the duration of the gaze away from the screen are focused based on the gaze rule in the multiple first evaluation rules, and marking the gaze duration with a gaze duration category label according to the obtained evaluation result, wherein when the obtained evaluation result indicates that the user's attention is not focused, marking the gaze duration with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that the user's attention is focused, marking the gaze duration with a category label for indicating focused learning; Obtaining a gaze trajectory from the action detection result, calling the concentration evaluation model to evaluate whether the gaze trajectory is concentrated in an area related to learning based on the gaze rule, and labeling the gaze trajectory with a gaze trajectory category label according to the obtained evaluation result, wherein when the obtained evaluation result indicates that the user's gaze is not concentrated in an area related to learning, labeling the gaze trajectory with a category label for indicating non-concentrated learning, and when the obtained evaluation result indicates that the user's gaze is concentrated in an area related to learning, labeling the gaze trajectory with a category label for indicating focused learning; Obtaining head posture parameters from the action detection result, calling the concentration evaluation model to evaluate whether the head posture parameters are stable and standard based on the behavioral action rules in the multiple first evaluation rules, and marking the head posture parameters with head posture category labels according to the obtained evaluation results, wherein when the obtained evaluation results indicate that the user's head posture is unstable or non-standard, the head posture is marked with a category label for indicating non-focused learning, and when the obtained evaluation results indicate that the user's head posture is stable and standard, the head posture is marked with a category label for indicating focused learning; Obtain hand motion features from the motion detection results, call the concentration evaluation model to evaluate whether the hand motion features indicate that the user's hands are performing activities related to learning based on the behavioral action rules, and label the hand motion features with hand motion category labels based on the obtained evaluation results, wherein when the obtained evaluation results indicate that the user's hands are not performing activities related to learning, the hand motion features are labeled with category labels for indicating non-focused learning, and when the obtained evaluation results indicate that the user's hands are performing activities related to learning, the hand motion features are labeled with category labels for indicating focused learning; Acquire sitting posture information from the action detection result, call the concentration evaluation model to evaluate the frequency of changes in the user's sitting posture indicated by the sitting posture information based on the behavioral action rule, and mark the sitting posture information with a sitting posture category label according to the obtained evaluation result, wherein when the obtained evaluation result indicates that the frequency of changes in the user's sitting posture is greater than or equal to a frequency threshold, mark the sitting posture information with a category label for indicating non-concentrated learning, and when the obtained evaluation result indicates that the frequency of changes in the user's sitting posture is less than the frequency threshold, mark the sitting posture information with a category label for indicating focused learning; The gaze duration, the sight line trajectory, the head posture parameters, the hand movement characteristics and the sitting posture information that have completed the category labeling are used as the first evaluation result.

7. The method according to claim 5, characterized in that The calling of the concentration evaluation model and using a plurality of second evaluation rules among the plurality of evaluation rules that match the environment detection result to evaluate the environment detection result to obtain a second evaluation result include: Obtain an item information set from the environmental detection result, call the concentration evaluation model to evaluate whether there are target items unrelated to learning in the item information set based on the item placement rules in the multiple second evaluation rules, and evaluate whether the placement of target items related to learning in the item information set meets the standards indicated by the item placement rules, and mark the item information set with item information category labels according to the obtained evaluation results, wherein, when the obtained evaluation results indicate that there are target items unrelated to learning or the placement of target items related to learning does not meet the standards indicated by the item placement rules, mark the item information set with a category label for indicating non-focused learning, and when the obtained evaluation results indicate that there are no target items unrelated to learning and the placement of target items related to learning meets the standards indicated by the item placement rules, mark the item information set with a category label for indicating focused learning; Obtaining a noise intensity index from the environmental detection result, calling the concentration evaluation model to evaluate whether the noise in the learning environment where the user is located exceeds the standard based on the noise detection rule in the multiple second evaluation rules, and marking a noise intensity category label for the noise intensity index according to the obtained evaluation result, wherein when the obtained evaluation result indicates that the noise in the learning environment where the user is located exceeds the standard, the noise intensity index is marked with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that the noise in the learning environment where the user is located does not exceed the standard, the noise intensity index is marked with a category label for indicating focused learning; Obtaining the noise type from the environment detection result, calling the concentration evaluation model to evaluate whether there is an interference source in the learning environment where the user is located based on the noise detection rule, and marking the noise type with an interference source category label according to the obtained evaluation result, wherein when the obtained evaluation result indicates that there is an interference source in the learning environment where the user is located, the noise type is marked with a category label for indicating non-focused learning, and when the obtained evaluation result indicates that there is no interference source in the learning environment where the user is located, the noise type is marked with a category label for indicating focused learning; The item information set, the noise intensity index and the noise type that have completed the category labeling are used as the second evaluation result.

8. A device for evaluating a user's learning concentration, characterized in that: include: An extraction module, used to obtain a video stream collected when the user is learning, and extract a plurality of image frames to be detected from the video stream; A detection module, configured to detect the eyes and behavioral actions of the user during learning based on the multiple image frames to be detected, and obtain an action detection result; The detection module is further used to detect objects placed in the learning environment where the user is learning based on the multiple image frames to be detected, and to perform environmental noise detection based on the video stream to obtain an environmental detection result; The evaluation module is used to evaluate the user's concentration during learning by using the behavior detection result and the learning environment detection result to obtain a concentration evaluation result.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images