Learning concentration evaluation method and related device

Through joint training prediction model, combining object detection and gaze area heat map regression branches, the interaction between learners and objects is evaluated, and the problem of insufficient efficiency and accuracy of learning focus assessment in the existing technology is solved, achieving more efficient and Accurate learning focus assessment.

CN118968558APending Publication Date: 2024-11-15IFLYTEK CO LTD

Patent Information

Application Number
CN202411120920.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing learning focus assessment techniques are insufficient in terms of efficiency and accuracy, and a method is needed to improve the efficiency of learning focus assessment and the accuracy of evaluation results.

Method used

Using a prediction model, the learner's concentration is evaluated by jointly training the target detection branch and the heat map regression prediction branch of the gaze area heat map regression. The prediction model includes head detection, gaze area heat map regression prediction branches, and calculates the duration of concentration to evaluate whether the learner is focused by analyzing the learner's interaction with the item in each frame of video.

Benefits of technology

The training process of the prediction model is simplified, the efficiency and accuracy of the interaction between learners and objects are improved, and the efficiency of learning concentration evaluation and the accuracy of results are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968558A_ABST
    Figure CN118968558A_ABST
Patent Text Reader

Abstract

The invention discloses a learning concentration degree evaluation method and a related device, and relates to the technical field of computer vision, and the method comprises the steps: respectively determining the interaction relation between a learner and an article in each frame of original image in a to-be-processed learning video of the learner through a prediction model, and obtaining the learning concentration degree of the learner according to the interaction relation between the learner and the article in each frame of original image; and determining a learning concentration degree evaluation result of the learner. The prediction model is obtained by integrating a target detection branch and a fixation area heat map regression prediction branch into a unified network framework and carrying out joint training on the target detection branch and the fixation area heat map regression prediction branch. The joint training not only simplifies the training process of the prediction model, but also significantly improves the reasoning efficiency and effect of the prediction model, and improves the determination efficiency and determination precision of the interaction relationship between the learner and the article, so that the learning concentration degree evaluation efficiency and the accuracy of the concentration degree evaluation result are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a learning concentration assessment method and related devices. Background Art

[0002] Learning concentration assessment technology refers to the use of a series of methods and tools to measure and analyze a person's concentration level during the learning process. This technology has significant application value in many scenarios, such as online classroom scenarios, monitored self-study scenarios (such as classrooms, study rooms or libraries, etc.). For example, in online classroom scenarios, assessing students' learning concentration can help teachers better understand and improve the online teaching environment, so that the teaching content and methods are more in line with students' needs. In monitored self-study scenarios, assessing students' learning concentration can not only monitor students' learning behavior in real time, but also provide timely reminders and suggestions, which is extremely important for promoting students to form good learning habits and improve learning efficiency.

[0003] The current learning concentration assessment technology is mainly based on computer vision. This type of method usually relies on cameras to capture students' learning videos in real time in online classroom scenes, monitored self-study scenes and other scenes, and then relies on deep learning algorithms to analyze and process students' learning videos to achieve the assessment of students' learning concentration. Among them, the method of relying on deep learning algorithms to analyze and process students' learning videos to achieve the assessment of students' learning concentration requires extracting behavioral features (such as facial expressions, eye contact, and head posture, etc.), and analyzing the behavioral features to obtain the learning concentration assessment results. However, the current learning concentration assessment technology still needs to be improved in terms of assessment efficiency and accuracy.

[0004] Therefore, how to provide a learning concentration assessment method to improve the efficiency of learning concentration assessment and the accuracy of concentration assessment results has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] In view of the above problems, the present application provides a learning concentration evaluation method and related devices to achieve the purpose of improving the efficiency of learning concentration evaluation and the accuracy of concentration evaluation results. The specific scheme is as follows:

[0006] The first aspect of the present application provides a method for evaluating learning concentration, comprising:

[0007] Obtaining a learning video to be processed of a learner, wherein the learning video to be processed includes a plurality of continuous frames of original images;

[0008] For each frame of the original image, determining the interaction relationship between the learner and the object in the original image using a prediction model, wherein the prediction model includes a target detection branch and a gaze area heat map regression prediction branch, and the prediction model is obtained by jointly training the target detection branch and the gaze area heat map regression prediction branch;

[0009] According to the interactive relationship between the learner and the object in each frame of the original image, the learning concentration evaluation result of the learner is determined. In a possible implementation,

[0010] In a possible implementation, the using a prediction model to determine the interaction relationship between the learner and the object in the original image includes:

[0011] Performing head detection on the original image to obtain a head detection result, wherein the head detection result is used to indicate whether a head is detected in the image;

[0012] If the head detection result indicates that a head is detected in the original image, a head image is determined, and based on the original image and the head image, the interactive relationship of the learner item is determined using the prediction model.

[0013] In a possible implementation, determining the interaction relationship between the learner and the object by using the prediction model based on the original image and the head image includes:

[0014] Based on the original image and the head image, using the prediction model to determine the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image;

[0015] Determining the learner's sight point based on the overlap between the learner's gaze area and the areas of the objects;

[0016] Based on the category of the object where the learner's sight line falls, the interactive relationship between the learner and the object is determined.

[0017] In a possible implementation, determining the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image by using the prediction model based on the original image and the head image includes:

[0018] Input the original image into the target detection branch of the prediction model to obtain the object detection result output by the target detection branch, and input the head image into the gaze area heat map regression prediction branch of the prediction model to obtain the gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0019] According to the object detection result and the gaze area heat map prediction result, the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image are determined.

[0020] In a possible implementation, determining the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image by using the prediction model based on the original image and the head image includes:

[0021] Based on the head image, determining an eye mask image;

[0022] Input the original image into the target detection branch of the prediction model to obtain the object detection result output by the target detection branch, and input the head image and the eye mask image into the gaze area heat map regression prediction branch of the prediction model to obtain the gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0023] According to the object detection result and the gaze area heat map prediction result, the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image are determined.

[0024] In a possible implementation, determining the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image according to the object detection result and the gaze area heat map prediction result includes:

[0025] Filtering the object detection result output by the target detection branch to obtain the region of each object in the original image and the category of each object in the original image;

[0026] The gaze area of ​​the learner is determined based on the gaze area heat map prediction result output by the gaze area heat map regression prediction branch.

[0027] In a possible implementation, determining the learning concentration evaluation result of the learner according to the interaction relationship between the learner and the object in each frame of the original image includes:

[0028] According to the interactive relationship between the learner and the object in each frame of the original image, the learner's concentration time is counted to obtain the learner's concentration time;

[0029] If the learner's concentration duration meets a preset condition, determining that the learner is focused on learning;

[0030] If the learner's concentration duration does not meet a preset condition, it is determined that the learner is not focused on learning.

[0031] A second aspect of the present application provides a prediction model training method, comprising:

[0032] Obtain training samples and sample labels;

[0033] Input the training sample into the target detection branch of the model to be trained and the gaze area heat map regression prediction branch of the model to be trained, and obtain the object detection box prediction result and the object category prediction result output by the target detection branch, and the gaze area existence prediction result and the gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0034] Determine the prediction loss of the model to be trained according to the object detection box prediction result, the object category prediction result, the gaze area existence prediction result and the gaze area heat map prediction result, and the sample label;

[0035] The model to be trained is trained according to the prediction loss of the model to be trained to obtain the prediction model.

[0036] In a possible implementation, the target detection branch includes a global image feature extraction module, a feature fusion module and a target detection module, and the gaze area heat map regression prediction branch includes a head image feature extraction module, an eye mask image feature extraction module, a feature aggregation module and a gaze area heat map regression prediction module.

[0037] In a possible implementation, determining the prediction loss of the to-be-trained model according to the object detection box prediction result, the object category prediction result, the gaze area existence prediction result, the gaze area heat map prediction result, and the sample label includes:

[0038] Determining an object detection frame prediction loss according to the object detection frame prediction result and the object detection frame label;

[0039] Determining an item category prediction loss according to the item category prediction result and the item category label;

[0040] Determining whether there is a prediction loss in the gaze area according to whether there is a prediction result in the gaze area and whether there is a label in the gaze area;

[0041] Determining a gaze area heat map prediction loss according to the gaze area heat map prediction result and the gaze area label;

[0042] The prediction loss of the model to be trained is determined according to the prediction loss of the object detection frame, the prediction loss of the object category, the prediction loss of whether the gaze area exists, and the prediction loss of the gaze area heat map.

[0043] A third aspect of the present application provides a learning concentration assessment device, comprising:

[0044] An acquisition unit, used for acquiring a learning video to be processed of a learner, wherein the learning video to be processed includes a plurality of continuous frames of original images;

[0045] A prediction unit, configured to determine, for each frame of the original image, an interaction relationship between the learner and the object in the original image using a prediction model, wherein the prediction model comprises a target detection branch and a gaze area heat map regression prediction branch, and the prediction model is obtained by jointly training the target detection branch and the gaze area heat map regression prediction branch;

[0046] An evaluation result determination unit is used to determine the evaluation result of the learner's learning concentration according to the interactive relationship between the learner and the object in each frame of the original image.

[0047] In a possible implementation, the prediction unit includes:

[0048] A head detection unit, used to perform head detection on the original image to obtain a head detection result, wherein the head detection result is used to indicate whether a head is detected in the image;

[0049] The interaction relationship prediction unit is used to determine the head image if the head detection result indicates that the head is detected in the original image, and determine the interaction relationship of the learner item using the prediction model based on the original image and the head image.

[0050] In a possible implementation, the interactive relationship prediction unit includes:

[0051] A prediction result determination unit, configured to determine, based on the original image and the head image, a gaze area of ​​the learner in the original image, an area of ​​each object in the original image, and a category of each object in the original image by using the prediction model;

[0052] A sight point determination unit, used for determining the sight point of the learner based on the degree of overlap between the learner's gaze area and the area of ​​each object;

[0053] The interactive relationship determining unit is used to determine the interactive relationship between the learner and the object based on the category of the object at the point where the learner's sight falls.

[0054] In a possible implementation, the prediction result determination unit includes:

[0055] A first model application unit is configured to input the original image into a target detection branch of a prediction model to obtain an object detection result output by the target detection branch, and input the head image into a gaze area heat map regression prediction branch of the prediction model to obtain a gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0056] The first determination unit determines the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image according to the object detection result and the gaze area heat map prediction result.

[0057] In a possible implementation, the prediction result determination unit includes:

[0058] an eye mask image determining unit, configured to determine an eye mask image based on the head image;

[0059] A second model application unit is used to input the original image into a target detection branch of a prediction model to obtain an object detection result output by the target detection branch, and input the head image and the eye mask image into a gaze area heat map regression prediction branch of the prediction model to obtain a gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0060] The second determination unit is used to determine the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image according to the object detection result and the gaze area heat map prediction result.

[0061] In a possible implementation, the first determining unit or the second determining unit is specifically configured to:

[0062] Filtering the object detection result output by the target detection branch to obtain the region of each object in the original image and the category of each object in the original image;

[0063] The gaze area of ​​the learner is determined based on the gaze area heat map prediction result output by the gaze area heat map regression prediction branch.

[0064] In a possible implementation, the evaluation result determination unit is specifically configured to:

[0065] According to the interactive relationship between the learner and the object in each frame of the original image, the learner's concentration time is counted to obtain the learner's concentration time;

[0066] If the learner's concentration duration meets a preset condition, determining that the learner is focused on learning;

[0067] If the learner's concentration duration does not meet a preset condition, it is determined that the learner is not focused on learning.

[0068] A fourth aspect of the present application provides a prediction model training device, comprising:

[0069] A sample acquisition unit, used to acquire training samples and sample labels;

[0070] A sample prediction result determination unit is used to input the training sample into the target detection branch of the model to be trained and the gaze area heat map regression prediction branch of the model to be trained, and obtain the object detection box prediction result and the object category prediction result output by the target detection branch, and the gaze area existence prediction result and the gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0071] A prediction loss determination unit, configured to determine the prediction loss of the model to be trained according to the prediction result of the object detection frame, the prediction result of the object category, the prediction result of whether the gaze area exists and the prediction result of the gaze area heat map, and the sample label;

[0072] A training unit is used to train the model to be trained according to the prediction loss of the model to be trained to obtain the prediction model.

[0073] In a possible implementation, the target detection branch includes a global image feature extraction module, a feature fusion module and a target detection module, and the gaze area heat map regression prediction branch includes a head image feature extraction module, an eye mask image feature extraction module, a feature aggregation module and a gaze area heat map regression prediction module.

[0074] In a possible implementation, the predicted loss determination unit is specifically configured to:

[0075] Determining an object detection frame prediction loss according to the object detection frame prediction result and the object detection frame label;

[0076] Determining an item category prediction loss according to the item category prediction result and the item category label;

[0077] Determining whether there is a prediction loss in the gaze area according to whether there is a prediction result in the gaze area and whether there is a label in the gaze area;

[0078] Determining a gaze area heat map prediction loss according to the gaze area heat map prediction result and the gaze area label;

[0079] According to the prediction loss of the object detection frame, the prediction loss of the object category, whether there is a prediction loss in the gaze area, and the prediction loss of the gaze area heat map, the prediction loss of the model to be trained is determined. A fifth aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the learning concentration evaluation method of the first aspect or any implementation of the first aspect, and / or the learning concentration evaluation method of the second aspect or any implementation of the second aspect.

[0080] A sixth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0081] The memory is used to store computer programs;

[0082] The processor is used to execute the computer program so that the electronic device can implement the learning concentration assessment method of the above-mentioned first aspect or any implementation of the first aspect, and / or the learning concentration assessment method of the above-mentioned second aspect or any implementation of the second aspect.

[0083] In a seventh aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can perform the learning concentration assessment method of the above-mentioned first aspect or any implementation of the first aspect, and / or the learning concentration assessment method of the above-mentioned second aspect or any implementation of the second aspect.

[0084] By means of the above technical scheme, the learning concentration evaluation method and related devices provided by the present application use the prediction model to respectively determine the interaction relationship between the learner and the object in each frame of the original image in the learner's learning video to be processed, and determine the learner's learning concentration evaluation result based on the interaction relationship between the learner and the object in each frame of the original image. The prediction model is obtained by integrating the target detection branch and the gaze area heat map regression prediction branch into a unified network framework, and by jointly training the target detection branch and the gaze area heat map regression prediction branch. Since this joint training not only simplifies the training process of the prediction model, but also significantly improves the reasoning efficiency and effect of the prediction model, and improves the efficiency and accuracy of determining the interaction relationship between the learner and the object, therefore, the efficiency of learning concentration evaluation and the accuracy of concentration evaluation results are guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0086] Figure 1 A flowchart of a method for evaluating learning concentration provided in an embodiment of the present application;

[0087] Figure 2 A schematic diagram of the structure of a model to be trained disclosed in an embodiment of the present application;

[0088] Figure 3 A schematic diagram of the structure and application of a prediction model disclosed in an embodiment of the present application;

[0089] Figure 4 A schematic diagram of the structure and application of another prediction model disclosed in an embodiment of the present application;

[0090] Figure 5 A schematic diagram of the structure of a learning concentration assessment device provided in an embodiment of the present application;

[0091] Figure 6 A schematic diagram of the structure of a prediction model training device provided in an embodiment of the present application;

[0092] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0094] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0095] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0096] Learning concentration assessment technology refers to the use of a series of methods and tools to measure and analyze a person's concentration level during the learning process. This technology has significant application value in many scenarios, such as online classroom scenarios, monitored self-study scenarios (such as classrooms, study rooms or libraries, etc.). For example, in online classroom scenarios, assessing students' learning concentration can help teachers better understand and improve the online teaching environment, so that the teaching content and methods are more in line with students' needs. In monitored self-study scenarios, assessing students' learning concentration can not only monitor students' learning behavior in real time, but also provide timely reminders and suggestions, which is extremely important for promoting students to form good learning habits and improve learning efficiency.

[0097] The current learning concentration assessment technology is mainly based on computer vision. This type of method usually relies on cameras to capture students' learning videos in real time in many scenarios such as online classroom scenarios and monitored self-study scenarios, and then relies on deep learning algorithms to analyze and process the learning videos to achieve an assessment of students' learning concentration.

[0098] Among them, the method of evaluating students' learning concentration by analyzing and processing students' learning videos with deep learning algorithms requires extracting behavioral features (such as facial expressions, eye contact, and head posture, etc.), analyzing the behavioral features, and quantifying the results of the behavioral feature analysis to obtain the learning concentration evaluation results. For example, the fatigue state of students can be analyzed through yawn detection and eye opening calculation, and the students' attention direction can be analyzed through head posture estimation and line of sight estimation. The fatigue state of students and the students' attention instructions are quantified to obtain the learning concentration evaluation results.

[0099] However, for each behavioral feature, a separate model needs to be configured to align for extraction and analysis, which makes the learning concentration evaluation inefficient in practical applications because multiple models need to be run.

[0100] In addition, each model needs to be trained, which not only increases the complexity of model training, but also cannot guarantee the training effect of each model at the same time, affecting the accuracy of the learning concentration evaluation results.

[0101] In order to solve the above problems, the embodiment of the present application provides a method for evaluating learning concentration. The following is a detailed introduction to the method for evaluating learning concentration in the embodiment of the present application in conjunction with the accompanying drawings.

[0102] Reference Figure 1 , Figure 1 A flow chart of a method for evaluating learning concentration provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, a learning concentration assessment method provided in an embodiment of the present application may include the following steps, which are described in detail below.

[0103] S101, obtaining a learning video to be processed of a learner, wherein the learning video to be processed includes a plurality of continuous frames of original images;

[0104] In this application, any acquisition device (such as a pan / tilt camera, a 4mm gun camera, a 6mm gun camera, a laptop computer, a learning machine, etc.) can be used to collect the learning process of any learner in any scenario (home scenario, study room scenario, etc.) to obtain the learner's original learning video. It can be collected in real time or collected and stored in advance, and this application does not impose any restrictions on this.

[0105] It should be noted that the duration of the learner's learning video to be processed can be a preset fixed duration. In this application, a video of the preset fixed duration can be intercepted from the learner's original learning video as the learning video to be processed. The specific value of the preset fixed duration can be set based on the needs of the scene, and this application does not impose any limitation.

[0106] As an embodiment, a sliding time window and a sliding step can be preset. The size of the sliding time window can be consistent with the duration of the learner's learning video to be processed, and the size of the sliding step needs to be smaller than the size of the sliding time window. Then, each time the sliding time window slides once from the collected original learning video of the learner based on the sliding step, a learning video to be processed can be determined, and each learning video to be processed is processed using the learning concentration evaluation method of the present application.

[0107] To facilitate understanding, assume that the learner's learning video to be processed is 10 seconds long, the size of the sliding time window is 10 seconds, and the size of the sliding step is 2 seconds. If the current learning video to be processed starts at 8:30:00 and ends at 8:30:10, then the next learning video to be processed starts at 8:30:02 and ends at 8:30:12. It should be understood that the above times are all video acquisition times of the acquisition device.

[0108] S102: for each frame of the original image, determine the interaction relationship between the learner and the object in the original image using a prediction model, wherein the prediction model includes a target detection branch and a gaze area heat map regression prediction branch, and the prediction model is obtained by jointly training the target detection branch and the gaze area heat map regression prediction branch;

[0109] It should be noted that, in the present application, the items include a first category of items and a second category of items, the first category of items are items not related to learning (such as mobile phones, game consoles, etc.), and the second category of items are items related to learning (such as books, stationery, learning machines, etc.); the interaction relationship between the learner and the item is used to characterize which category of items the learner has an interaction relationship with.

[0110] It should be further explained that the specific structure and training method of the prediction model will be described in detail through subsequent embodiments and will not be described in detail here.

[0111] S103: Determine the learning concentration evaluation result of the learner according to the interactive relationship between the learner and the object in each frame of the original image.

[0112] In the present application, the learner's concentration time can be counted according to the interaction between the learner and the object in each frame of the original image to obtain the learner's concentration time. If the learner's concentration time meets the preset conditions, it is determined that the learner is focused on learning. If the learner's concentration time does not meet the preset conditions, it is determined that the learner is not focused on learning. Further, if it is determined that the learner is not focused on learning, reminders and suggestions can be given to the learner.

[0113] The preset condition may be greater than the preset concentration time, or greater than or equal to the preset concentration time, and this application does not impose any limitation on this. The specific value of the preset concentration time may be based on the scene requirement device, and this application does not impose any limitation on this.

[0114] Among them, the method of counting the learner's concentration time according to the interaction relationship between the learner and the object in each frame of the original image can be that, for each frame of the original image, if the interaction relationship between the learner and the object in the original image represents that the learner has an interaction relationship with the first category of objects, it means that in the original image, the learner's behavior is a distracting behavior, and the distraction time is accumulated; if the interaction relationship between the learner and the object in the original image represents that the learner has an interaction relationship with the second category of objects, it means that in the original image, the learner's behavior is a learning behavior, and the concentration time is accumulated.

[0115] In the present application, by analyzing multiple consecutive frames of images, a more stable and accurate learning concentration evaluation result can be obtained.

[0116] In the present application, a prediction model is used to determine the interaction relationship between the learner and the object in each frame of the original image in the learner's learning video to be processed, and the learner's learning concentration evaluation result is determined based on the interaction relationship between the learner and the object in each frame of the original image. The prediction model is obtained by integrating the target detection branch and the gaze area heat map regression prediction branch into a unified network framework, and by jointly training the target detection branch and the gaze area heat map regression prediction branch. Since this joint training not only simplifies the training process of the prediction model, but also significantly improves the reasoning efficiency and effect of the prediction model, and improves the efficiency and accuracy of determining the interaction relationship between the learner and the object, therefore, the efficiency of learning concentration evaluation and the accuracy of concentration evaluation results are guaranteed.

[0117] In another embodiment of the present application, a specific implementation method of using a prediction model to determine the interaction relationship between the learner and the object in the original image in step S102 is described. The method may include the following steps:

[0118] S201: Performing head detection on the original image to obtain a head detection result, where the head detection result is used to indicate whether a head is detected in the image;

[0119] In the present application, the original image can be input into a head detection model to obtain a head detection result output by the head detection model. The head detection model can adopt an existing head detection model, which is not described in detail in the present application. If the head detection result indicates that a head is detected in the original image, the head detection result can also include a head detection frame, which can be displayed in the original image.

[0120] S202: If the head detection result indicates that a head is detected in the original image, a head image is determined, and based on the original image and the head image, the interactive relationship of the learner-item is determined using the prediction model.

[0121] In the present application, a head image can be extracted from an original image using a head detection frame. The head image can be an RGB image cropped from the original image. It should be noted that if the head detection result indicates that the head is not detected in the original image, the specific step of determining the interaction relationship between the learner and the object in the original image is skipped, and the interaction relationship between the learner and the object in the original image is directly determined to be empty, so as to avoid unnecessary computational consumption and improve the efficiency of learning concentration assessment.

[0122] In another embodiment of the present application, a specific implementation method of determining the interaction relationship between the learner and the object based on the original image and the head image by using the prediction model in step S202 is described. The method may include the following steps:

[0123] S301: Based on the original image and the head image, using the prediction model to determine the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image;

[0124] In the present application, the structure of the prediction model may be different. For prediction models with different structures, based on the original image and the head image, the specific implementation methods of using the prediction model to determine the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image are different. The details will be described in detail through the following embodiments.

[0125] S302: Determine the learner's sight point based on the degree of overlap between the learner's gaze area and the areas of each object.

[0126] In the present application, the intersection over union (IoU) of the learner's gaze area and the area of ​​each object can be calculated to represent the degree of overlap between the learner's gaze area and the area of ​​the object. Based on the degree of overlap between the learner's gaze area and the area of ​​each object, the learner's sight point is determined, and specifically, the area of ​​the object with the largest IoU is determined as the learner's sight point.

[0127] S303: Determine the interaction relationship between the learner and the object based on the category of the object where the learner's sight line falls.

[0128] If the category of the object where the learner's line of sight falls is the first category of objects, it is determined that the learner has an interactive relationship with the first category of objects; if the category of the object where the learner's time sequence falls is the second category of objects, it is determined that the learner has an interactive relationship with the second category of objects.

[0129] Before introducing S301 in detail, the training process of the prediction model of the present application is described first. The training process of the prediction model specifically includes the following steps:

[0130] S401: Obtain training samples and sample labels.

[0131] In the present application, the training samples include original image samples, head image samples and eye mask image samples, and the sample labels include object detection frame labels, object category labels, gaze area existence labels and gaze area labels;

[0132] In the present application, any acquisition device (such as a pan-tilt machine, a 4mm gun machine, a 6mm gun machine, a laptop computer, a learning machine, etc.) can be used to collect video of the learning process of any learner (such as people with ages ranging from 3 to 50 years old, covering multiple age groups) in any scene (family scene, study room scene, etc.) (such as the learning process between 9 am and 10 pm, covering multiple time periods) to obtain multiple groups of learning videos as learning video training samples. Each learning video training sample contains multiple continuous frames of original image samples. After obtaining the original image samples, head image samples and eye mask image samples can be extracted from them. Specifically, target detection can be performed on the original image samples to obtain a head detection frame and an eye detection frame. The head detection frame and the eye detection frame are used to extract the head image sample and the eye mask image sample from the original image sample. It should be noted that the head image sample is an RGB image cropped from the original image sample, and the eye mask image sample is a binary image.

[0133] In the present application, after determining the training samples, the training samples can be labeled to obtain sample labels. The sample labels include an object detection frame label, an object category label, a gaze area existence label, and a gaze area label. Among them, the object detection frame label is used to indicate the position of the object in the image, and the object category label is used to indicate the category of the object. The category of the object includes a first category of objects and a second category of objects. The first category of objects are objects that are not related to learning (such as mobile phones, game consoles, etc.), and the second category of objects are objects related to learning (such as books, stationery, learning machines, etc.). The gaze area existence label is used to indicate whether the gaze area is in the image, and a 0-1 label can be used, where 0 means not in the image and 1 means in the image. The gaze area label is used to indicate the position of the gaze area in the image.

[0134] In this application, the labeling method can be manual labeling or combined with manual labeling with the help of some tools. For example, in this application, Grounding DINO can be used to generate pseudo labels for targets such as heads, eyes, pens, books, mobile phones, and handheld game consoles in the image, and then the pseudo labels can be manually adjusted to obtain object detection frame labels and object category labels. Before labeling the gaze area label, you can first use the tool to label the gaze point. After the gaze point is labeled, the gaze area label is generated at the gaze point using Gaussian distribution.

[0135] S402: Input the training sample into the target detection branch of the model to be trained and the gaze area heat map regression prediction branch of the model to be trained, and obtain the object detection box prediction result and the object category prediction result output by the target detection branch, and the gaze area existence prediction result and the gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0136] Specifically, the original image samples can be input into the target detection branch of the model to be trained to obtain the object detection box prediction results and the object category prediction results output by the target detection branch, and the head image samples and the eye mask image samples can be input into the gaze area heat map regression prediction branch of the model to be trained to obtain the gaze area existence prediction results and the gaze area heat map prediction results output by the gaze area heat map regression prediction branch.

[0137] For ease of understanding, refer to Figure 2 , Figure 2 Schematic diagram of a model to be trained disclosed in the embodiment of the present application. Figure 2 As shown, the model to be trained contains two branches, one of which is the target detection branch, and the other is the gaze area heat map regression prediction branch.

[0138] In a possible implementation, the target detection branch includes a global image feature extraction module, a feature fusion module, and a target detection module, and the gaze area heat map regression prediction branch includes a head image feature extraction module, a feature aggregation module, and a gaze area heat map regression prediction module. Considering the importance of eye features to gaze area prediction, in order to strengthen the prediction model's attention to binocular areas, the gaze area heat map regression prediction branch also includes an eye mask image feature extraction module.

[0139] The input of the global image feature extraction module is the original image sample, the input of the head image feature extraction module is the head image sample, and the input of the eye mask image feature extraction module is the eye mask image sample. In the object detection branch, the global image feature extraction module, the feature fusion module, and the object detection module are connected in sequence, and the output of the previous module is used as the input of the next module. The final output of the object detection module is the object detection box prediction result and the object category prediction result of the training sample.

[0140] In the gaze area heat map regression prediction branch, the head image feature extraction module, feature aggregation module and gaze area heat map regression prediction module are also connected in sequence, and the output of the previous module is used as the input of the next module. In addition, the input of the feature aggregation module in the gaze area heat map regression prediction branch also includes the output of the global image feature extraction module in the target detection branch. The eye attention map extracted by the eye mask image feature extraction module is also used as the input of the gaze area heat map regression prediction module. The final output of the gaze area heat map regression prediction module is the prediction result of whether the gaze area of ​​the training sample exists and the gaze area heat map prediction result.

[0141] It should be noted that since the target detection branch and the gaze area heat map regression prediction branch perform different prediction tasks respectively, in order to ensure the prediction effect of each branch, in this application, the global image feature extraction module and the head image feature extraction module may not share parameters.

[0142] Taking into account that the global image feature extraction module and the head image feature extraction module do not share parameters, which will result in too many parameters in the prediction model, in order to ensure the prediction effect of each branch and reduce the number of parameters in the prediction model, in the present application, in a possible implementation, the global image feature extraction module may include a first global image feature extraction submodule and a second global image feature extraction submodule, and the head image feature extraction module may include a first head image feature extraction submodule and a second head image feature extraction submodule, wherein the first global image feature extraction submodule and the first head image feature extraction submodule do not share parameters, and the second global image feature extraction submodule and the second head image feature extraction submodule share parameters.

[0143] In this case, in the target detection branch, the first global image feature extraction submodule, the second global image feature extraction submodule, the feature fusion module and the target detection module are connected in sequence, and the output of the previous module is used as the input of the next module. In the gaze area heat map regression prediction branch, the first head image feature extraction submodule and the second head image feature extraction submodule, the feature aggregation module and the gaze area heat map regression prediction module are also connected in sequence, and the output of the previous module is used as the input of the next module. In addition, the input of the feature aggregation module in the gaze area heat map regression prediction branch also includes the output of the second global image feature extraction submodule in the target detection branch.

[0144] It should be noted that in this application, there is no restriction on the specific structure of each module. As an implementable method, the global image feature extraction module and the head image feature extraction module can both adopt the ResNet18 structure. The target detection module can adopt RTMDet. The feature aggregation module can adopt the SE (Squeeze-and-Excitation) attention mechanism structure. The gaze area heat map regression prediction module can adopt an encoder-decoder structure. The encoder can be composed of three convolutional layers, and the decoder can be composed of three deconvolutional layers.

[0145] If the global image feature extraction module and the head image feature extraction module both adopt the ResNet18 structure, then as a feasible implementation method, the first global image feature extraction submodule and the first head image feature extraction submodule can be the first 9 layers of ResNet18, and the second global image feature extraction submodule and the second head image feature extraction submodule can be the last 9 layers of ResNet18.

[0146] In the present application, whether a prediction result of whether a gaze area exists may be a predicted probability value of the gaze area being within the image, which value is between 0 and 1. The closer the value is to 1, the higher the probability that the gaze point is within the image.

[0147] S403: Determine the prediction loss of the model to be trained according to the prediction result of the object detection box, the prediction result of the object category, the prediction result of whether the fixation area exists and the prediction result of the fixation area heat map, and the sample label;

[0148] In the present application, for the target detection branch, its prediction loss includes the item detection box prediction loss and the item category prediction loss. For the gaze area heat map regression prediction branch, its prediction loss includes the gaze area existence prediction loss and the gaze area heat map prediction loss. The item detection box prediction loss is used to indicate the difference between the item detection box prediction result and the item detection box label, the item category prediction loss is used to indicate the difference between the item category prediction result and the item category label, the gaze area existence prediction loss is used to indicate the difference between the gaze area existence prediction result and the gaze area existence label, and the gaze area heat map prediction loss is used to indicate the difference between the gaze area heat map prediction result and the gaze area label. Each prediction loss can be characterized by a different loss function. As an implementable embodiment, the item detection box prediction loss can adopt GIoULoss, the item category prediction loss can adopt QualityFocalLoss, the gaze area existence prediction loss can adopt BCELoss, and the gaze area heat map prediction loss can adopt MSELoss.

[0149] Then in the present application, the prediction loss of the item detection frame can be determined according to the prediction result of the item detection frame and the label of the item detection frame; the prediction loss of the item category can be determined according to the prediction result of the item category and the label of the item category; the prediction loss of whether the gaze area exists can be determined according to the prediction result of whether the gaze area exists and the label of the gaze area; the prediction loss of the gaze area heat map can be determined according to the prediction result of the gaze area heat map and the label of the gaze area; the prediction loss of the model to be trained can be determined according to the prediction loss of the item detection frame, the prediction loss of the item category, the prediction loss of whether the gaze area exists and the prediction loss of the gaze area heat map. As an implementable method, the prediction loss of the model to be trained can be the weighted sum of the above-mentioned losses.

[0150] For ease of understanding, assume that for the target detection branch, its prediction loss is L det , L det =L cls +L bbox , the weight ratio is: L cls :L bbox =1:2. Among them, L cls is the item category prediction loss, L bbox is the prediction loss of the object detection box. For the gaze area heat map regression prediction branch, whether the gaze area exists is predicted by L p , the prediction loss of the gaze area heat map is L M , then the prediction loss L of the model to be trained total It can be as follows:

[0151] ;

[0152] Among them, the weighting coefficients λ1, λ2, and λ3 can all be set to 1.

[0153] S404: Training the model to be trained according to the prediction loss of the model to be trained to obtain the prediction model.

[0154] In the present application, after the training of the model to be trained is completed according to the prediction loss of the model to be trained, the prediction model can be obtained. Specifically, the training of the model to be trained is completed when a preset number of iterations is reached or the prediction loss converges.

[0155] It should be noted that after the training of the model to be trained is completed, the entire trained model can be used as the prediction model, or the eye mask image feature extraction module in the trained model can be discarded and the remaining part can be used as the prediction model. This application does not impose any restrictions on this.

[0156] As an implementable method, refer to Figure 3 , Figure 3 Schematic diagram of the structure and application of a prediction model disclosed in the embodiment of this application. Figure 3 As shown, the prediction model includes two branches, one of which is a target detection branch, and the other is a gaze area heat map regression prediction branch. The target detection branch includes a global image feature extraction module, a feature fusion module, and a target detection module, and the gaze area heat map regression prediction branch includes a head image feature extraction module, a feature aggregation module, and a gaze area heat map regression prediction module. The global image feature extraction module includes a first global image feature extraction submodule and a second global image feature extraction submodule, and the head image feature extraction module includes a first head image feature extraction submodule and a second head image feature extraction submodule.

[0157] Among them, the input of the global image feature extraction module is the original image, and the input of the head image feature extraction module is the head image. In the target detection branch, the global image feature extraction module, the feature fusion module and the target detection module are connected in sequence, and the output of the previous module is used as the input of the next module. The final output of the target detection module is the object detection result (i.e., the object detection box prediction result and the object category prediction result). In the gaze area heat map regression prediction branch, the head image feature extraction module, the feature aggregation module and the gaze area heat map regression prediction module are also connected in sequence, and the output of the previous module is used as the input of the next module. In addition, the input of the feature aggregation module in the gaze area heat map regression prediction branch also includes the output of the global image feature extraction module in the target detection branch. The final output of the gaze area heat map regression prediction module is the gaze area heat map prediction result.

[0158] Specifically, after the original image is input into the target detection branch of the prediction model, the global image feature extraction module performs feature extraction on the original image to obtain multi-scale global image features, and the multi-scale global image features are sent to the feature fusion module for feature fusion to obtain fused global image features, and the fused global image features are sent to the target detection module for prediction of the object detection frame and the object category to obtain the object detection result. After the head image is input into the gaze area heat map regression prediction branch of the prediction model, the head image feature extraction module performs feature extraction on the head image to obtain head image features, and the head image features and multi-scale global image features are sent to the feature aggregation module for feature aggregation to obtain aggregated image features, and the aggregated image features are sent to the gaze area heat map regression prediction module for gaze area heat map prediction to obtain the gaze area heat map prediction result.

[0159] based on Figure 3 The prediction model shown in FIG. 1 is used to determine the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image based on the original image and the head image, using the prediction model, including:

[0160] Input the original image into the target detection branch of the prediction model to obtain the object detection result output by the target detection branch, and input the head image into the gaze area heat map regression prediction branch of the prediction model to obtain the gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0161] According to the object detection result and the gaze area heat map prediction result, the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image are determined.

[0162] As another possible implementation method, refer to Figure 4 , Figure 4 FIG. 1 is a schematic diagram of the structure and application of another prediction model disclosed in the embodiment of the present application. Figure 4 As shown, the prediction model includes two branches, one of which is a target detection branch, and the other is a gaze area heat map regression prediction branch. The target detection branch includes a global image feature extraction module, a feature fusion module, and a target detection module, and the gaze area heat map regression prediction branch includes a head image feature extraction module, an eye mask image feature extraction module, a feature aggregation module, and a gaze area heat map regression prediction module. The global image feature extraction module includes a first global image feature extraction submodule and a second global image feature extraction submodule, and the head image feature extraction module includes a first head image feature extraction submodule and a second head image feature extraction submodule.

[0163] Among them, the input of the global image feature extraction module is the original image, the input of the head image feature extraction module is the head image, and the input of the eye mask image feature extraction module is the eye mask image. In the target detection branch, the global image feature extraction module, the feature fusion module and the target detection module are connected in sequence, and the output of the previous module is used as the input of the next module. The final output of the target detection module is the object detection result (i.e., the object detection box prediction result and the object category prediction result). In the gaze area heat map regression prediction branch, the head image feature extraction module, the feature aggregation module and the gaze area heat map regression prediction module are also connected in sequence, and the output of the previous module is used as the input of the next module. In addition, the input of the feature aggregation module in the gaze area heat map regression prediction branch also includes the output of the global image feature extraction module in the target detection branch. The eye attention map extracted by the eye mask image feature extraction module is also used as the input of the gaze area heat map regression prediction module. The final output of the gaze area heat map regression prediction module is the gaze area heat map prediction result.

[0164] Specifically, after the original image is input into the target detection branch of the prediction model, the global image feature extraction module performs feature extraction on the original image to obtain multi-scale global image features, and the multi-scale global image features are sent to the feature fusion module for feature fusion to obtain fused global image features, and the fused global image features are sent to the target detection module to predict the object detection frame and object category to obtain the object detection result.

[0165] After the head image and the eye mask image are input into the gaze area heat map regression prediction branch of the prediction model, the head image feature extraction module performs feature extraction on the head image to obtain the head image features, the eye mask image feature extraction module performs feature extraction on the eye mask image to obtain the eye attention map (AttentionMap), the head image features and the multi-scale global image features are sent to the feature aggregation module for feature aggregation to obtain the aggregated image features, the aggregated image features and the eye attention map are sent to the gaze area heat map regression prediction module for gaze area heat map prediction to obtain the gaze area heat map prediction result.

[0166] based on Figure 4 The prediction model shown in FIG. 1 is used to determine the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image based on the original image and the head image, using the prediction model, including:

[0167] Based on the head image, determining an eye mask image;

[0168] Input the original image into the target detection branch of the prediction model to obtain the object detection result output by the target detection branch, and input the head image and the eye mask image into the gaze area heat map regression prediction branch of the prediction model to obtain the gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0169] According to the object detection result and the gaze area heat map prediction result, the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image are determined.

[0170] It should be noted that the eye mask image is a binary image.

[0171] Specifically, determining the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image based on the object detection result and the gaze area heat map prediction result includes: filtering the object detection result output by the target detection branch to obtain the area of ​​each object in the original image and the category of each object in the original image; determining the learner's gaze area based on the gaze area heat map prediction result output by the gaze area heat map regression prediction branch.

[0172] In the present application, the item detection result includes an item detection frame and an item category corresponding to each item detection frame. Then, filtering the item detection result output by the target detection branch refers to filtering out repeated item detection frames in the item detection result output by the target detection branch and the item category corresponding to the repeated item detection frame. Specifically, the NMS (Non-maximum suppression) method can be used.

[0173] In the present application, the heat map area corresponding to a specific color in the gaze area heat map prediction result can be determined as the gaze area of ​​the learner. For example, in the gaze area heat map, red represents the area with the most concentrated gaze, and yellow and green represent the areas with less gaze. In the present application, the heat map area corresponding to red in the gaze area heat map prediction result can be determined as the gaze area of ​​the learner.

[0174] A method for evaluating learning concentration provided by an embodiment of the present application is introduced above, and a related device for executing the above-mentioned method for evaluating learning concentration will be introduced below.

[0175] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a learning concentration assessment device provided in an embodiment of the present application. Figure 5 As shown, the learning concentration assessment device comprises:

[0176] An acquisition unit 11 is used to acquire a learning video to be processed of a learner, wherein the learning video to be processed includes a plurality of continuous original image frames;

[0177] A prediction unit 12 is used to determine the interaction relationship between the learner and the object in each frame of the original image by using a prediction model, wherein the prediction model includes a target detection branch and a gaze area heat map regression prediction branch, and the prediction model is obtained by jointly training the target detection branch and the gaze area heat map regression prediction branch;

[0178] The evaluation result determining unit 13 is used to determine the evaluation result of the learner's learning concentration according to the interaction relationship between the learner and the object in each frame of the original image.

[0179] In a possible implementation, the prediction unit includes:

[0180] A head detection unit, used to perform head detection on the original image to obtain a head detection result, wherein the head detection result is used to indicate whether a head is detected in the image;

[0181] The interaction relationship prediction unit is used to determine the head image if the head detection result indicates that the head is detected in the original image, and determine the interaction relationship of the learner item using the prediction model based on the original image and the head image.

[0182] In a possible implementation, the interactive relationship prediction unit includes:

[0183] A prediction result determination unit, configured to determine, based on the original image and the head image, a gaze area of ​​the learner in the original image, an area of ​​each object in the original image, and a category of each object in the original image by using the prediction model;

[0184] A sight point determination unit, used for determining the sight point of the learner based on the degree of overlap between the learner's gaze area and the area of ​​each object;

[0185] The interactive relationship determining unit is used to determine the interactive relationship between the learner and the object based on the category of the object at the point where the learner's sight falls.

[0186] In a possible implementation, the prediction result determination unit includes:

[0187] A first model application unit is configured to input the original image into a target detection branch of a prediction model to obtain an object detection result output by the target detection branch, and input the head image into a gaze area heat map regression prediction branch of the prediction model to obtain a gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0188] The first determination unit determines the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image according to the object detection result and the gaze area heat map prediction result.

[0189] In a possible implementation, the prediction result determination unit includes:

[0190] an eye mask image determining unit, configured to determine an eye mask image based on the head image;

[0191] A second model application unit is used to input the original image into a target detection branch of a prediction model to obtain an object detection result output by the target detection branch, and input the head image and the eye mask image into a gaze area heat map regression prediction branch of the prediction model to obtain a gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0192] The second determination unit is used to determine the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image according to the object detection result and the gaze area heat map prediction result.

[0193] In a possible implementation, the first determining unit or the second determining unit is specifically configured to:

[0194] Filtering the object detection result output by the target detection branch to obtain the region of each object in the original image and the category of each object in the original image;

[0195] The gaze area of ​​the learner is determined based on the gaze area heat map prediction result output by the gaze area heat map regression prediction branch.

[0196] In a possible implementation, the evaluation result determination unit is specifically configured to:

[0197] According to the interactive relationship between the learner and the object in each frame of the original image, the learner's concentration time is counted to obtain the learner's concentration time;

[0198] If the learner's concentration duration meets a preset condition, determining that the learner is focused on learning;

[0199] If the learner's concentration duration does not meet a preset condition, it is determined that the learner is not focused on learning.

[0200] See also Figure 6 , Figure 6 A schematic diagram of the structure of a prediction model training device provided in an embodiment of the present application. Figure 6 As shown, the prediction model training device comprises:

[0201] A sample acquisition unit 21 is used to acquire training samples and sample labels;

[0202] The sample prediction result determination unit 22 is used to input the training sample into the target detection branch of the model to be trained and the gaze area heat map regression prediction branch of the model to be trained, and obtain the object detection box prediction result and the object category prediction result output by the target detection branch, and the gaze area existence prediction result and the gaze area heat map prediction result output by the gaze area heat map regression prediction branch;

[0203] A prediction loss determination unit 23, configured to determine the prediction loss of the model to be trained according to the prediction result of the object detection frame, the prediction result of the object category, the prediction result of whether the gaze area exists and the prediction result of the gaze area heat map, and the sample label;

[0204] The training unit 24 is used to train the model to be trained according to the prediction loss of the model to be trained to obtain the prediction model.

[0205] In a possible implementation, the target detection branch includes a global image feature extraction module, a feature fusion module and a target detection module, and the gaze area heat map regression prediction branch includes a head image feature extraction module, an eye mask image feature extraction module, a feature aggregation module and a gaze area heat map regression prediction module.

[0206] In a possible implementation, the predicted loss determination unit is specifically configured to:

[0207] Determining an object detection frame prediction loss according to the object detection frame prediction result and the object detection frame label;

[0208] Determining an item category prediction loss according to the item category prediction result and the item category label;

[0209] Determining whether there is a prediction loss in the gaze area according to whether there is a prediction result in the gaze area and whether there is a label in the gaze area;

[0210] Determining a gaze area heat map prediction loss according to the gaze area heat map prediction result and the gaze area label;

[0211] The prediction loss of the model to be trained is determined according to the prediction loss of the object detection frame, the prediction loss of the object category, whether there is a prediction loss in the gaze area, and the prediction loss of the gaze area heat map. An electronic device is also provided in an embodiment of the present application. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0212] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0213] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0214] Also provided in an embodiment of the present application is a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the learning concentration assessment methods and / or the prediction model training method provided in the embodiments of the present application.

[0215] A computer-readable storage medium is also provided in an embodiment of the present application. The computer storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any one of the learning concentration assessment methods and / or prediction model training methods provided in the embodiments of the present application.

[0216] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0217] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0218] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0219] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for evaluating learning concentration, characterized in that: include: Obtaining a learning video to be processed of a learner, wherein the learning video to be processed includes a plurality of continuous frames of original images; For each frame of the original image, determining the interaction relationship between the learner and the object in the original image using a prediction model, wherein the prediction model includes a target detection branch and a gaze area heat map regression prediction branch, and the prediction model is obtained by jointly training the target detection branch and the gaze area heat map regression prediction branch; An evaluation result of the learner's learning concentration is determined according to the interactive relationship between the learner and the object in each frame of the original image.

2. The method according to claim 1, characterized in that The using a prediction model to determine the interactive relationship between the learner and the object in the original image includes: Performing head detection on the original image to obtain a head detection result, wherein the head detection result is used to indicate whether a head is detected in the image; If the head detection result indicates that a head is detected in the original image, a head image is determined, and based on the original image and the head image, the interactive relationship between the learner and the object is determined using the prediction model.

3. The method according to claim 2, characterized in that The determining the interactive relationship between the learner and the object by using the prediction model based on the original image and the head image includes: Based on the original image and the head image, using the prediction model to determine the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image; Determining the learner's sight point based on the overlap between the learner's gaze area and the areas of the objects; Based on the category of the object where the learner's sight line falls, the interactive relationship between the learner and the object is determined.

4. The method according to claim 3, characterized in that The method of determining the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image by using the prediction model based on the original image and the head image includes: Input the original image into the target detection branch of the prediction model to obtain the object detection result output by the target detection branch, and input the head image into the gaze area heat map regression prediction branch of the prediction model to obtain the gaze area heat map prediction result output by the gaze area heat map regression prediction branch; According to the object detection result and the gaze area heat map prediction result, the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image are determined.

5. The method according to claim 4, characterized in that The step of inputting the head image into the gaze area heat map regression prediction branch of the prediction model to obtain a gaze area heat map prediction result output by the gaze area heat map regression prediction branch includes: Based on the head image, determining an eye mask image; The head image and the eye mask image are input into the gaze area heat map regression prediction branch of the prediction model to obtain a gaze area heat map prediction result output by the gaze area heat map regression prediction branch.

6. The method according to claim 4 or 5, characterized in that: Determining the learner's gaze area in the original image, the area of ​​each object in the original image, and the category of each object in the original image according to the object detection result and the gaze area heat map prediction result includes: Filtering the object detection result output by the target detection branch to obtain the region of each object in the original image and the category of each object in the original image; The gaze area of ​​the learner is determined based on the gaze area heat map prediction result output by the gaze area heat map regression prediction branch.

7. The method according to claim 1, characterized in that Determining the learning concentration evaluation result of the learner according to the interactive relationship between the learner and the object in each frame of the original image includes: According to the interactive relationship between the learner and the object in each frame of the original image, the learner's concentration time is counted to obtain the learner's concentration time; If the learner's concentration duration meets a preset condition, determining that the learner is focused on learning; If the learner's concentration duration does not meet a preset condition, it is determined that the learner is not focused on learning.

8. A prediction model training method, characterized in that: include: Obtain training samples and sample labels; Input the training sample into the target detection branch of the model to be trained and the gaze area heat map regression prediction branch of the model to be trained, and obtain the object detection box prediction result and the object category prediction result output by the target detection branch, and the gaze area existence prediction result and the gaze area heat map prediction result output by the gaze area heat map regression prediction branch; Determine the prediction loss of the model to be trained according to the object detection box prediction result, the object category prediction result, the gaze area existence prediction result and the gaze area heat map prediction result, and the sample label; The model to be trained is trained according to the prediction loss of the model to be trained to obtain the prediction model.

9. The method according to claim 8, characterized in that The target detection branch includes a global image feature extraction module, a feature fusion module and a target detection module, and the gaze area heat map regression prediction branch includes a head image feature extraction module, an eye mask image feature extraction module, a feature aggregation module and a gaze area heat map regression prediction module.

10. The method according to claim 8, characterized in that The step of determining the prediction loss of the model to be trained according to the prediction result of the object detection frame, the prediction result of the object category, the prediction result of whether the gaze area exists and the prediction result of the gaze area heat map, and the sample label includes: Determining an object detection frame prediction loss according to the object detection frame prediction result and the object detection frame label; Determining an item category prediction loss according to the item category prediction result and the item category label; Determining whether there is a prediction loss in the gaze area according to whether there is a prediction result in the gaze area and whether there is a label in the gaze area; Determining a gaze area heat map prediction loss according to the gaze area heat map prediction result and the gaze area label; The prediction loss of the model to be trained is determined according to the prediction loss of the object detection frame, the prediction loss of the object category, the prediction loss of whether the gaze area exists, and the prediction loss of the gaze area heat map.

11. A learning concentration assessment device, characterized in that: include: An acquisition unit, used for acquiring a learning video to be processed of a learner, wherein the learning video to be processed includes a plurality of continuous frames of original images; A prediction unit, configured to determine, for each frame of the original image, an interaction relationship between the learner and the object in the original image using a prediction model, wherein the prediction model comprises a target detection branch and a gaze area heat map regression prediction branch, and the prediction model is obtained by jointly training the target detection branch and the gaze area heat map regression prediction branch; An evaluation result determination unit is used to determine the evaluation result of the learner's learning concentration according to the interactive relationship between the learner and the object in each frame of the original image.

12. A prediction model training device, characterized in that: include: A sample acquisition unit, used to acquire training samples and sample labels; A sample prediction result determination unit is used to input the training sample into the target detection branch of the model to be trained and the gaze area heat map regression prediction branch of the model to be trained, and obtain the object detection box prediction result and the object category prediction result output by the target detection branch, and the gaze area existence prediction result and the gaze area heat map prediction result output by the gaze area heat map regression prediction branch; A prediction loss determination unit, configured to determine the prediction loss of the model to be trained according to the prediction result of the object detection frame, the prediction result of the object category, the prediction result of whether the gaze area exists and the prediction result of the gaze area heat map, and the sample label; A training unit is used to train the model to be trained according to the prediction loss of the model to be trained to obtain the prediction model.

13. A computer program product, characterized in that It includes computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the learning concentration assessment method as described in any one of claims 1 to 7, and / or the prediction model training method as described in any one of claims 8 to 10.

14. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the learning concentration assessment method as described in any one of claims 1 to 7, and / or the prediction model training method as described in any one of claims 8 to 10.

15. A computer storage medium, characterized in that The computer storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the learning concentration assessment method as described in any one of claims 1 to 7, and / or the prediction model training method as described in any one of claims 8 to 10.

Citation Information

Patent Citations

  • Concentration degree evaluation method and device, computer equipment and storage medium

    CN113762062A

  • Online learning concentration evaluation method based on multimode feature fusion

    CN117746096A

  • Artificial intelligence-based question generation learning skill habit training system, method, and computer program

    WO2023172056A1

Cited By

  • Multi-mode joint attention automatic detection method, device and terminal

    CN120783390A