Teaching input degree evaluation method based on implicit parameter modeling and application

Abstract

The invention discloses a teaching input degree evaluation method based on implicit parameter modeling, and relates to the technical field of informatization education, and the method comprises the steps: obtaining teaching behavior resources collected in a smart classroom environment, carrying out the preprocessing of the teaching behavior resources, and converting the teaching behavior resources into a face image and an eye image of a teacher; inputting the preprocessed face image and eye image into a pre-trained network model to obtain the facial expression and sight direction of the teacher in the teaching scene; according to the facial expressions and the sight directions of the teacher in the teaching process at different moments, the emotion features and the attention features of the teacher are obtained respectively, and the emotion features and the attention features are adopted as implicit state indexes to obtain a teaching input degree evaluation result. The teaching behavior of the teacher can be monitored and evaluated in a multi-dimensional manner, so that the accuracy and the real-time performance of evaluation of the teaching input degree of the teacher are improved, personalized feedback and improvement suggestions are provided, and the teacher is helped to improve professional accomplishment and teaching ability.

Description

Technical Field

[0001] The present application relates to the field of information-based education technology, and more specifically, to a teaching engagement evaluation method, device, electronic device and computer storage medium based on implicit parameter modeling. Background Art

[0002] With the development of educational technology, teachers' teaching methods and teaching scenarios are also constantly evolving. Driven by the intelligent teaching environment, the evaluation of teachers' teaching engagement has become an important topic in educational research. Specifically, the evaluation of teachers' teaching engagement refers to the evaluation of teachers' input in the teaching process by analyzing the energy, time and emotional investment shown by teachers in the teaching process, in order to improve the overall teaching effect and students' learning experience.

[0003] At present, the evaluation of teacher teaching engagement faces many challenges. Traditional methods of evaluating engagement often rely on the subjective judgment of teachers and school administrators, and there are problems such as inconsistent evaluation standards and lack of objectivity in evaluation results. In addition, teacher teaching engagement is not just a simple statistics of class time, but also involves implicit states such as emotional changes and attention levels of teachers during the teaching process. How to comprehensively evaluate these factors in real time and accurately has brought huge challenges to the comprehensive perception of teacher teaching engagement.

[0004] With the deep integration of information technology and education and teaching, the technology-supported smart classroom provides a new means for collecting and analyzing teachers' teaching behaviors. For example, through the real-time classroom data collection and analysis system, teachers' teaching activities, classroom attention, and emotional engagement can be recorded more accurately. However, in practical applications, smart classrooms still face problems such as incomplete data collection and untimely analysis. Therefore, how to systematically study the corresponding evaluation and feedback mechanism, conduct multi-dimensional monitoring and evaluation of teachers' teaching behaviors, improve the accuracy and real-time evaluation of teachers' teaching engagement, and provide personalized feedback and improvement suggestions is an urgent problem to be solved. Summary of the invention

[0005] In response to at least one defect or improvement need in the prior art, the present invention provides a teaching engagement assessment method and application based on implicit parameter modeling, which aims to monitor and evaluate teachers' teaching behaviors in multiple dimensions, so as to improve the accuracy and real-time performance of the evaluation of teachers' teaching engagement, thereby providing personalized feedback and improvement suggestions, and helping teachers improve their professional quality and teaching ability.

[0006] To achieve the above-mentioned purpose, according to the first aspect of the present invention, a teaching engagement evaluation method based on implicit parameter modeling is provided, comprising: obtaining teaching behavior resources collected in a smart classroom environment, preprocessing the teaching behavior resources, and converting them into facial images and eye images of the teacher; inputting the preprocessed facial images and eye images into a pre-trained network model to obtain the teacher's facial expressions and sight directions in the teaching scene; obtaining the teacher's emotional characteristics and attention characteristics according to the teacher's facial expressions and sight directions at different times in the teaching process, and using the emotional characteristics and the attention characteristics as implicit state indicators to obtain the teaching engagement evaluation results.

[0007] In one embodiment of the present invention, the acquisition of teaching behavior resources collected in the smart classroom environment includes: collecting the behavior information V of the teacher in the teaching process; fe ; The collected video data V fe Decompose into several sub-video sets Set of preset segment lengths in chronological order fe = {V fe1 ,V fe2 ,…,V fe5}, and add time-location-teacher number notes to each sub-video; sub-video set Set fe All sub-videos in the video are processed by frame division, and a frame set converted from each sub-video is obtained by randomly extracting frames, and the frame set from the same sub-video is stored in a folder corresponding to the remark information.

[0008] In one embodiment of the present invention, the teaching behavior resources are preprocessed and converted into facial images and eye images, including: performing face recognition on a set of frames in the same folder to retain the facial image area and adjust it to a target pixel size, classifying the facial images according to six basic expressions of happiness, surprise, neutrality, basic sadness, anger and fear, and marking them with corresponding labels; performing eye area detection on a set of frames in the same folder to retain the eye image area and adjust it to a target pixel size, classifying the visual directions shown by the eye images according to looking at students, looking at the blackboard, looking at textbooks and abnormal directions, and marking them with corresponding labels; performing data enhancement on the preprocessed facial images and eye images, including center cropping the images to the target size, and then performing random flipping and strong exposure processing with a preset probability, and finally dividing the facial images and eye images into two equal parts according to a two-fold cross-validation method, one for model training and the other for model testing.

[0009] In one embodiment of the present invention, the network model includes a first sub-network model SwinD2C and a second sub-network model TokenGE; the first sub-network model SwinD2C is used to obtain the teacher's facial expression recognition result Exp in the smart classroom environment. Prediction The second sub-network model TokenGE is used to obtain the teacher's gaze direction recognition result Gaze in the teaching scenario prediction .

[0010] In one embodiment of the present invention, the method of obtaining the facial expression recognition result of the teacher in the smart classroom environment includes: sending the input facial image to the SwinTransformer backbone network for feature extraction, obtaining the facial visual feature map extracted at each stage to form a facial visual feature set [visual] = {F1, F2, ... F n}, where n is the number of stages of SwinTransformer, F i is the facial visual feature map extracted in the i-th stage, i∈{1,2,…,n}; the feature map in the facial visual feature set [visual] is converted into a tensor of the same dimension through convolution processing, and then F i The dimensions are R C×H×W , where C is the number of channels, H and W are the height and width of the feature map respectively, and then the facial visual feature set [visual] is sent to the attention network composed of multiple self-attention mechanisms and spatial self-attention mechanisms; after being processed by the attention network, the n facial visual features are spliced ​​in the channel dimension, and then sent to the graph neural network for further processing, so as to obtain a facial visual feature map that integrates multi-level semantic information; the facial global visual feature map is input into the convolution list list composed of multiple convolution kernels of different scales CNN Processing in the middle to obtain a collection of local visual feature maps of different scales Where m is the convolution list list CNN The number of convolution kernels is then weighted to fuse the local visual feature maps of different scales, and the fused results are pooled to obtain the final facial discriminative feature F exp ; The final facial discriminative feature F exp Send it to the fully connected layer to obtain the final facial expression recognition result Exp prediction .

[0011] In one embodiment of the present invention, the step of obtaining the teacher's sight direction recognition result in the teaching scene includes: sending the input eye image to the IR50 network to extract basic image feature information F∈R C×H×W, where C is the number of channels, H and W are the height and width of the feature map respectively; the extracted feature map is spatially offset in different diagonal directions and concatenated with the input feature map, and then the concatenated feature map is divided into multiple regions, and the regions at corresponding positions are combined, and a visual Token sequence is formed after linear mapping and adding position encoding; on the basis of the visual Token sequence, a learnable coordinate Token sequence [coordinate] is added to form the Transformer input T = {[visual], [coordinate]}, and after processing by N Transformer blocks, the coordinate Token sequence [coordinate] is output; the coordinate Token sequence [coordinate] is input into the final gaze discrimination network to obtain the accurate gaze direction Gaze prediction .

[0012] In one embodiment of the present invention, the method of using emotion characteristics and attention characteristics as implicit state indicators to evaluate teaching engagement and providing corresponding intervention measures based on the evaluation results includes: establishing a teaching engagement evaluation index system corresponding to the implicit state performance of teaching engagement, and formulating corresponding intervention measures according to the evaluated engagement level; obtaining the emotion characteristics and attention characteristics of the teacher in the teaching process based on the facial expression recognition results and line of sight direction analysis results of the teacher in the teaching scenario, and evaluating the teaching engagement of the teacher according to the teaching engagement evaluation system; and adopting corresponding intervention measures according to the evaluation results of the teacher's teaching engagement.

[0013] According to the second aspect of the present invention, there is also provided a teaching engagement evaluation device based on implicit parameter modeling, which includes: a resource conversion module, used to obtain teaching behavior resources collected in a smart classroom environment, pre-process the teaching behavior resources, and convert them into facial images and eye images of the teacher; a network model processing module, used to input the pre-processed facial image and the eye image into a pre-trained network model to obtain the teacher's facial expression and line of sight direction in the teaching scene; an engagement evaluation module, used to obtain the teacher's emotional characteristics and attention characteristics according to the teacher's facial expression and line of sight direction at different times in the teaching process, and use the emotional characteristics and the attention characteristics as implicit state indicators to obtain the teaching engagement evaluation results.

[0014] According to the third aspect of the present invention, there is also provided an electronic device, which comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method described in any one of the above embodiments.

[0015] According to a fourth aspect of the present invention, there is also provided a computer-readable storage medium storing a computer program executable by an access authentication device, wherein when the computer program runs on the access authentication device, the access authentication device executes the steps of the method described in any one of the above embodiments.

[0016] In general, the above technical solutions conceived by the present invention can achieve at least the following beneficial effects compared with the prior art:

[0017] By collecting teaching behavior resources and pre-processing them into teacher's facial images and eye images, and inputting them into pre-trained specific facial expression recognition model and gaze direction analysis model to obtain the corresponding teacher's facial expression and gaze direction respectively, the teacher's emotional characteristics and attention characteristics are obtained as implicit state indicators to accurately evaluate the teaching engagement, so that the teacher's teaching behavior can be monitored and evaluated in multiple dimensions, and the accuracy and real-time performance of the evaluation of teacher's teaching engagement can be improved, thereby providing personalized feedback and improvement suggestions to help teachers improve their professional quality and teaching ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a teaching engagement evaluation method based on implicit parameter modeling provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of specific execution steps of a teaching engagement evaluation method based on implicit parameter modeling provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of collecting teaching behavior resources in a smart classroom environment provided in an embodiment of the present application;

[0022] Figure 4 A network diagram of a facial expression recognition model provided in an embodiment of the present application;

[0023] Figure 5 A network diagram of a line of sight direction analysis model provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of a teaching engagement evaluation device based on implicit parameter modeling provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0029] like Figure 1 As shown, the first embodiment of the present invention proposes a teaching engagement evaluation method based on implicit parameter modeling, for example, including: step S1, obtaining teaching behavior resources collected in a smart classroom environment, preprocessing the teaching behavior resources, and converting them into facial images and eye images of the teacher; step S2, inputting the preprocessed facial images and eye images into a pre-trained network model to obtain the facial expressions and sight directions of the teacher in the teaching scene; step S3, obtaining the teacher's emotional characteristics and attention characteristics according to the facial expressions and sight directions of the teacher at different times in the teaching process, and using the emotional characteristics and the attention characteristics as implicit state indicators to obtain the teaching engagement evaluation results.

[0030] Combination Figure 2 As shown, in step S1, for example, a video resource of a teacher's teaching behavior captured by a fisheye camera in a smart classroom environment is obtained, and information such as time and place are added to the captured resource.

[0031] In one embodiment, step S1 includes the following steps:

[0032] Step 1.1: Start the fisheye camera device and start collecting the teacher's behavior information during the teaching process Vfe For example, the video length is 10 minutes and the video frame rate is 30fps.

[0033] Step 1.2: Collect the video data V fe For example, the sub-video set Set is decomposed into each segment with a length of 2 minutes in chronological order. fe = {V fe1 ,V fe2 ,…,V fe5}, and add time-place-teacher number notes to each sub-video.

[0034] Step 1.3: Set the sub-video set fe All sub-videos are divided into frames and randomly extracted, for example, 2 frames are extracted every three seconds. At this time, each sub-video is converted into a frame set, each frame set contains 80 frames, and the frame sets from the same sub-video are stored in a folder named "time-place-teacher number".

[0035] like Figure 3 As shown in the figure, the teacher is conducting teaching activities in a smart classroom environment. In this scenario, a fisheye camera is used to collect videos of the teacher's teaching behaviors, and then the behavior video resources are preprocessed to obtain eye images for gaze direction analysis and facial images for facial expression recognition, thereby providing an important data source for the fine facial expression recognition network model SwinD2C and the gaze direction analysis network model TokenGE.

[0036] In one embodiment, preprocessing the video resource of the teacher's teaching behavior in the teaching environment and converting it into facial images and eye images includes the following steps:

[0037] Step 2.1: For the frame collection in the same folder, use the InsightFace toolbox to perform face recognition, detection and alignment, ensuring that only the facial image area is retained in the original image, such as adjusting the image to 600×600 pixels. Finally, the obtained facial image is annotated and the expressions shown in the facial image are divided into six basic expressions, among which happy and surprised expressions correspond to positive emotions, neutral expressions contain neutral emotions, and sad, angry and afraid expressions express negative emotions.

[0038] Step 2.2: For the frame set in the same folder, use the MTCNN network to detect and crop the eye area to ensure that the original image only retains the eye area. For example, adjust the image to 600×600 pixels. Finally, annotate the obtained eye image and divide the visual direction shown in the eye image into four categories: looking at students, looking at the blackboard, looking at textbooks, and abnormal directions.

[0039] Step 2.3: Perform data enhancement on the preprocessed facial and eye images, specifically including center cropping the images to a size of 224×224, and then randomly flipping them with a probability of 30%, and performing strong exposure processing with a probability of 50%. Finally, the facial and eye images are divided according to the two-fold cross-validation method, and the data is divided into two equal parts, one for model training and the other for model testing, that is, the training set and the test set account for 50% each.

[0040] In step S2, for example, the preprocessed facial image X and eye image V are input into the ISMAE network model, the ISMAE network model includes a first sub-network model SwinD2C and a second sub-network model TokenGE, the first sub-network model SwinD2C is used to obtain the facial expression recognition result Exp of the teacher in the smart teacher environment Prediction The second sub-network model TokenGE is used to obtain the teacher's gaze direction recognition result Gaze in the teaching scenario. prediction .

[0041] Combination Figure 4 As shown in the figure, the specific training process of the fine expression recognition model SwinD2C is as follows:

[0042] Step 3.1: Send the input RGB facial image X to the backbone network SwinTransformer for feature extraction, obtain the facial visual feature map at each stage of the SwinTransformer feature extraction process, and form a facial visual feature set [visual] = {F1, F2, ... F n}, where n is the number of stages of SwinTransformer, F i is the facial visual feature map proposed in the i-th stage, i∈{1,2,…,n}, where the execution process of the SwinTransformer block in each stage is described as follows:

[0043]

[0044] in, and c l They represent the output patch vectors of the (S)W-MSA module and the MLP module in the i-th Transformer block, respectively. LN(·) refers to layer normalization, MLP(·) represents multiple fully connected layer operations, W-MSA(·) represents the window-based multi-head self-attention mechanism, and SW-MSA(·) is the shifted window partitioning configuration adopted in the Swin Transformer. These Transformer blocks are divided into n stages through the Patch merging layer.

[0045] Step 3.2: The feature map in the facial visual feature set [visual] is convolved and converted into a tensor of the same dimension. At this time, F i The dimensions are R C×H×W , where the value of C is 1024, and the values ​​of H and W are both 14. Then the facial visual feature set [visual] is sent to the attention network composed of multiple self-attention mechanisms and spatial self-attention mechanisms. The processing process of each attention network is as follows:

[0046]

[0047] Among them, selfAttention(·) and seAttention(·) represent the self-attention mechanism operation and spatial attention operation respectively. It represents the intermediate result obtained after the i-th facial spatial feature map passes through the self-attention mechanism. It is the final facial visual feature map obtained after the i-th facial spatial feature map is processed by the attention network.

[0048] Step 3.3: After being processed by the attention network, the n facial visual features are concatenated in the channel dimension and then sent to the graph neural network for further processing to obtain the facial visual feature map F that integrates multi-level semantic information. This process can be expressed as:

[0049]

[0050] Among them, Concat(·) and GCN(·) represent the concatenation operation and graph convolution operation respectively, and F SSA Represents the intermediate result obtained by concatenating n facial visual features in the channel dimension.

[0051] Step 3.4: Input the facial visual features F that integrate multi-level semantic information into the convolution list list composed of multiple convolution kernels of different scales CNN ={k1,k2,…,k m} to obtain a set of local visual feature maps of different scales Where m is the convolution list list CNN The number of convolution kernels in the process is mathematically described as:

[0052]

[0053] Among them, Conv i (·) represents the convolution operation corresponding to the i-th convolution kernel, It is the i-th local visual feature map.

[0054] Then the local visual feature maps of different scales are weighted summed and then globally averaged. The process is described as follows:

[0055]

[0056] in, is the feature map obtained by summing up m local visual feature maps, For The features obtained by global average pooling, H and W are the height and width corresponding to the local visual feature map respectively.

[0057] The obtained global average pooling local features It is sent to two fully connected layers for dimensionality increase and dimensionality reduction operations, and then the weight z corresponding to each local visual feature map is obtained through the Softmax function, which is specifically described as:

[0058]

[0059] Among them, FC2 is the fully connected layer for dimensionality increase operation, and FC1 is the fully connected layer for dimensionality reduction operation. At this time, the dimension of z is m, which is consistent with the number of local feature maps.

[0060] Then, the local visual feature maps of different scales are weighted fused, and the fused results are pooled to obtain the final facial discriminative feature F exp , this process is mathematically described as follows:

[0061]

[0062] Among them, F sum is the result of weighted fusion of local visual feature maps, and Avgpool(·) represents the average pooling operation.

[0063] Step 3.5: The final facial discriminative feature F exp Send it to the fully connected layer to obtain the final refined facial expression recognition result Exp prediction , as follows:

[0064] Exp prediction =FC(F exp )

[0065] Among them, FC(·) represents the fully connected layer network, and the expression prediction result tensor Exp prediction ∈R J , J represents the category of facial expression recognition. The categories of facial expression recognition shown are divided into six categories: happy, surprised, neutral, afraid, sad and angry.

[0066] Combination Figure 5As shown, the preprocessed eye region image V is input into the trained gaze direction analysis model to obtain the teacher's gaze direction analysis result. The specific training process is as follows:

[0067] Step 4.1: Feed the input facial RGB image X into the IR50 network to extract the basic feature information F∈R C ×H×W , where C is the number of channels, H and W are the height and width of the feature map respectively. This process can be expressed as:

[0068] F = Conv(X)

[0069] Wherein, Conv(·) represents the operation corresponding to IR50, specifically, C is 512, and H and W are both 7.

[0070] Step 4.2: Spatially shift the extracted feature map F in different diagonal directions and concatenate it with the input feature map. Then divide the concatenated feature map into multiple regions and combine the regions at corresponding positions. After linear mapping and adding position encoding, a visual token sequence is formed, which is denoted as [visual] here. The specific formula is as follows:

[0071] [visual]={v1+p1,v2+p2,…,v m +p m}

[0072] Among them, v i represents the i-th visual feature patch, p i represents the position embedding corresponding to the i-th visual feature patch, where i ranges from {1,2,…,m} and m represents the total number of visual tokens, which is 196.

[0073] Step 4.3: Add the learnable coordinate Token sequence [coordinate] based on [visual] to form the Transformer input T = {[visual], [coordinate]}, and then after being processed by the Transformer composed of N Transformer blocks, the coordinate Token sequence [coordinate] is output for subsequent processing. The processing process of each Transformer encoding block is similar, specifically including the multi-head self-attention mechanism and the feedforward network. First, the multi-head self-attention mechanism needs to calculate the query, key, and value. The calculation process is as follows:

[0074]

[0075] Among them, l represents the number of layers of the current Transformer, h represents the number of heads of the multi-head self-attention mechanism, LN represents the normalization operation, and W Q , W V and W K They correspond to the weight matrices when query, value and key are generated respectively.

[0076] After calculating the query, value, and key vectors, the single-head attention is calculated as follows:

[0077]

[0078] Among them, d m Represents the latent space feature length of each attention head.

[0079] The multi-head self-attention of layer l can be expressed as follows:

[0080] MHSA(T)=concat(A1,A2,…,A h )

[0081] In general, the calculation process of each Transformer encoding block is as follows:

[0082]

[0083] Among them, T l ′ and T l They represent the outputs of the multi-head self-attention and multi-layer perceptron of the lth layer respectively. The MLP includes the Hardswish activation function and the fully connected layer.

[0084] Step 4.4: Input the coordinate Token sequence [coordinate] processed by the Transformer into the final line of sight discrimination network to obtain the accurate line of sight direction. This process can be described as:

[0085] Gaze prediction =MLP([coordinate])

[0086] Among them, Gaze prediction represents the predicted gaze direction, MLP(·) represents the multi-layer perceptron operation, Gaze prediction The dimension is 4, representing 4 different gaze directions.

[0087] In step S3, according to the teacher's emotion and line of sight perception results at different moments in the teaching process, the teacher's teaching engagement evaluation index is used to judge the teaching engagement, and corresponding intervention measures are provided based on the judgment results. The teacher's teaching engagement evaluation index system is as follows:

[0088]

[0089] To summarize, the first embodiment of the present invention proposes a teaching engagement evaluation method based on implicit parameter modeling, which collects teaching behavior resources, pre-processes them into teacher's facial images and eye images, and inputs them into a pre-trained specific facial expression recognition model and gaze direction analysis model to obtain the corresponding teacher's facial expression and gaze direction, respectively. In this way, the teacher's emotional characteristics and attention characteristics are obtained as implicit state indicators to accurately evaluate the teaching engagement, thereby enabling multi-dimensional monitoring and evaluation of teachers' teaching behaviors, improving the accuracy and real-time performance of teacher teaching engagement evaluation, and providing personalized feedback and improvement suggestions to help teachers improve their professional quality and teaching ability.

[0090] In addition, if Figure 6 As shown, the second embodiment of the present invention proposes a teaching engagement evaluation device 20 based on implicit parameter modeling, for example, including: a resource conversion module 201, a network model processing module 202 and an engagement evaluation module 203.

[0091] Among them, the resource conversion module 201 is used to obtain the teaching behavior resources collected in the smart classroom environment, pre-process the teaching behavior resources, and convert them into the teacher's facial image and eye image. The network model processing module 202 is used to input the pre-processed facial image and eye image into the pre-trained network model to obtain the teacher's facial expression and line of sight direction in the teaching scene. The engagement evaluation module 203 is used to obtain the teacher's emotional characteristics and attention characteristics according to the teacher's facial expression and line of sight direction during the teaching process at different times, and use the emotional characteristics and attention characteristics as implicit state indicators to obtain the teaching engagement evaluation results.

[0092] It is worth mentioning that the method implemented by the teaching engagement evaluation device 20 based on implicit parameter modeling disclosed in the second embodiment of the present invention is as described in the first embodiment, so it will not be described in detail here. Optionally, the various modules and the above-mentioned other operations or functions in the second embodiment are respectively for implementing the teaching engagement evaluation method based on implicit parameter modeling described in the first embodiment, and the beneficial effects of this embodiment are the same as those of the first embodiment, and for the sake of brevity, they are not repeated here.

[0093] The third embodiment of the present invention further proposes an electronic device 30, for example comprising: at least one processing unit 31, and at least one storage unit 32, wherein the storage unit 32 stores a computer program, and when the computer program is executed by the processing unit 31, the processing unit 31 executes the method described in the first embodiment, and the beneficial effects of the electronic device 30 provided by this embodiment are the same as the beneficial effects of the teaching engagement assessment method based on implicit parameter modeling provided by the first embodiment.

[0094] The fourth embodiment of the present invention also provides a computer-readable storage medium 40 on which a computer program is stored. When the program is executed by a processor, the steps of the above method are implemented. The beneficial effects of the computer-readable storage medium 40 provided by this embodiment are the same as the beneficial effects of the teaching engagement evaluation method based on implicit parameter modeling provided by the first embodiment.

[0095] In addition, it can be understood that the aforementioned embodiments are merely illustrative descriptions of the present application. Under the premise that the technical features do not conflict, the structures do not contradict, and the purpose of the invention of the present application is not violated, the technical solutions of the various embodiments can be arbitrarily combined and used in combination.

[0096] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and / or methods can be implemented in other ways. For example, the device embodiments described above are merely schematic, and the division of the units / modules is merely a logical function division. There may be other division methods in actual implementation, such as multiple units or modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0097] The units / modules described as separate components may or may not be physically separated, and the components shown as units / modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units / modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, each functional unit / module in each embodiment of the present application may be integrated into one processing unit / module, or each unit / module may exist physically separately, or two or more units / modules may be integrated into one unit / module. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of hardware plus software functional units / modules.

[0099] The above-mentioned integrated unit / module implemented in the form of a software functional unit / module can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for enabling one or more processors of a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A teaching engagement evaluation method based on implicit parameter modeling, characterized in that: include: Acquire the teaching behavior resources collected in the smart classroom environment, pre-process the teaching behavior resources, and convert them into facial images and eye images of the teacher; Inputting the preprocessed facial image and eye image into a pre-trained network model to obtain the teacher's facial expression and sight direction in the teaching scenario; The emotional characteristics and attention characteristics of the teacher are obtained according to the facial expressions and the sight lines of the teacher at different moments in the teaching process, and the teaching engagement evaluation result is obtained by using the emotional characteristics and the attention characteristics as implicit state indicators.

2. The teaching engagement evaluation method based on implicit parameter modeling according to claim 1 is characterized in that: The obtaining of teaching behavior resources collected in the smart classroom environment includes: Collecting the teacher's behavior information during the teaching process fe ; The collected video data V fe Decompose into several sub-video sets Set of preset segment lengths in chronological order fe = {V fe1 ,V fe2 ,…,V fe5 }, and add time-location-teacher number notes to each sub-video; Set the sub-video set fe All sub-videos in the video are processed by frame division, and a frame set converted from each sub-video is obtained by randomly extracting frames, and the frame set from the same sub-video is stored in a folder corresponding to the remark information.

3. The teaching engagement evaluation method based on implicit parameter modeling according to claim 2 is characterized in that: The preprocessing of the teaching behavior resources to convert them into the teacher's facial image and eye image includes: Performing face recognition on the frame set in the same folder to retain the facial image area and adjust to the target pixel size, classifying the facial image according to six basic expressions of happiness, surprise, neutral, basic sadness, anger and fear and annotating them with corresponding labels; Performing eye region detection on the frame set in the same folder to retain the eye image area and adjust it to the target pixel size, classifying the visual direction shown in the eye image into gaze at students, gaze at the blackboard, gaze at textbooks and abnormal directions and marking them with corresponding labels; The preprocessed facial image and the eye image are subjected to data enhancement, including center cropping the image to a target size, and then performing random flipping and strong exposure processing with a preset probability. Finally, the facial image and the eye image are divided into two equal parts according to a two-fold cross-validation method, one of which is used for model training and the other is used for model testing.

4. The teaching engagement evaluation method based on implicit parameter modeling according to claim 1 is characterized in that: The network model includes a first sub-network model SwinD2C and a second sub-network model TokenGE; The first sub-network model SwinD2C is used to obtain the teacher's facial expression recognition results Exp in the smart classroom environment. Prediction ; The second sub-network model TokenGE is used to obtain the teacher's gaze direction recognition result Gaze in the teaching scene. prediction .

5. The teaching engagement evaluation method based on implicit parameter modeling according to claim 4 is characterized in that: The step of obtaining the facial expression recognition result of the teacher in the smart classroom environment includes: The input facial image is sent to the SwinTransformer backbone network for feature extraction, and the facial visual feature map extracted at each stage is obtained to form a facial visual feature set [visual] = {F1, F2, ... F n }, where n is the number of stages of SwinTransformer, F i is the facial visual feature map extracted in the i-th stage, i∈{1,2,…,n}; The feature map in the facial visual feature set [visual] is converted into a tensor of the same dimension through convolution. i The dimensions are R C×H×W , where C is the number of channels, H and W are the height and width of the feature map respectively, and then the facial visual feature set [visual] is sent to the attention network composed of multiple self-attention mechanisms and spatial self-attention mechanisms; After being processed by the attention network, the n facial visual features are concatenated in the channel dimension and then sent to the graph neural network for further processing to obtain a facial visual feature map that integrates multi-level semantic information; Input the global visual feature map of the face into a convolution list consisting of multiple convolution kernels of different scales. CNN Processing in the middle to obtain a collection of local visual feature maps of different scales Where m is the convolution list list CNN The number of convolution kernels is then weighted to fuse the local visual feature maps of different scales, and the fused results are pooled to obtain the final facial discriminative feature F. exp ; The final facial discriminative feature F exp Send it to the fully connected layer to obtain the final facial expression recognition result Exp prediction .

6. The teaching engagement evaluation method based on implicit parameter modeling according to claim 4 is characterized in that: The step of obtaining the teacher's sight direction recognition result in the teaching scene includes: The input eye image is sent to the IR50 network to extract the basic feature information of the image F∈R C×H×W , where C is the number of channels, H and W are the height and width of the feature map respectively; The extracted feature map is spatially offset in different diagonal directions and concatenated with the input feature map. The concatenated feature map is then divided into multiple regions, and the regions at corresponding positions are combined to form a visual token sequence through linear mapping and adding position encoding. Add a learnable coordinate Token sequence [coordinate] to the visual Token sequence to form the Transformer input T = {[visual], [coordinate]}. After being processed by N Transformer blocks, the coordinate Token sequence [coordinate] is output; Input the coordinate Token sequence [coordinate] into the final gaze discrimination network to obtain the accurate gaze direction Gaze prediction .

7. The teaching engagement evaluation method based on implicit parameter modeling according to claim 1 is characterized in that: The method of using the emotion feature and the attention feature as implicit state indicators to obtain the teaching engagement evaluation result includes: Establish a teaching engagement evaluation index system corresponding to the implicit state of teaching engagement, and formulate corresponding intervention measures according to the assessed engagement level; Based on the teacher's facial expression recognition results and gaze direction analysis results in the teaching scene, the teacher's emotional characteristics and attention characteristics in the teaching process are obtained, and the teacher's teaching engagement is evaluated according to the teaching engagement evaluation system; Adopt corresponding intervention measures according to the evaluation results of teachers' teaching engagement.

8. A teaching engagement evaluation device based on implicit parameter modeling, characterized in that: include: A resource conversion module is used to obtain the teaching behavior resources collected in the smart classroom environment, pre-process the teaching behavior resources, and convert them into the teacher's facial image and eye image; A network model processing module, used for inputting the pre-processed facial image and the eye image into a pre-trained network model to obtain the teacher's facial expression and sight direction in the teaching scene; The engagement evaluation module is used to obtain the teacher's emotional characteristics and attention characteristics according to the teacher's facial expressions and the direction of sight at different times during the teaching process, and use the emotional characteristics and the attention characteristics as implicit state indicators to obtain the teaching engagement evaluation result.

9. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by the access authentication device. When the computer program runs on the access authentication device, the access authentication device executes the steps of the method according to any one of claims 1 to 7.

Images