Teaching quality evaluation method and system

Abstract

The invention relates to a teaching quality evaluation method and system. The method comprises the following steps: simultaneously acquiring video clips and corresponding audio clips in different classroom areas during the same teaching activity; for each classroom area, inputting the corresponding video clip into a face recognition network to obtain a face area in the video clip; inputting the face region into an expression recognition network to obtain the expression of the face; inputting the corresponding video clip into a target detection network, and detecting the behavior of the person; inputting the corresponding audio clip into a voice emotion recognition network to obtain the emotion of the audio clip; generating a comprehensive expression, a comprehensive behavior and a comprehensive emotion based on the expression, the behavior and the emotion corresponding to each classroom area; and based on the comprehensive expression, the comprehensive behavior and the comprehensive emotion, evaluating the teaching quality. According to the invention, the teaching quality evaluation accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and recognition, and in particular to a teaching quality evaluation method and system. Background Art

[0002] As the AI ​​craze sweeps the world, using AI to solve problems in education has become a method and approach that people are gradually using. Cameras in traditional classrooms are usually installed on the ceiling, which can only capture the overall view of students' classroom situation, and cannot fully and accurately evaluate students' specific behaviors in the classroom and the teaching quality of teachers. Therefore, it is necessary to provide a teaching quality evaluation method and system. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a teaching quality evaluation method and system, which improves the problem of low accuracy of teaching quality evaluation in the prior art due to only being able to capture local information of students.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a method for evaluating teaching quality, comprising: simultaneously acquiring video clips and corresponding audio clips of different classroom areas during the same teaching activity; for each classroom area: inputting the corresponding video clip into a face recognition network to obtain the face area in the video clip; inputting the face area into an expression recognition network to obtain the expression of the face; inputting the corresponding video clip into a target detection network to detect the behavior of the character; inputting the corresponding audio clip into a speech emotion recognition network to obtain the emotion of the audio clip; based on the expressions, behaviors and emotions corresponding to each classroom area, generating comprehensive expressions, comprehensive behaviors and comprehensive emotions; and evaluating the teaching quality based on the comprehensive expressions, the comprehensive behaviors and the comprehensive emotions.

[0005] In one embodiment of the present invention, the facial region is input into an expression recognition network to obtain facial expressions, including: performing initial feature extraction on the facial region to obtain initial features of the facial region; performing deep feature extraction on the initial features of the facial region based on multi-level residual connections to obtain deep features; and performing feature mapping on the deep features to obtain facial expressions.

[0006] In one embodiment of the present invention, the initial feature extraction of the face region to obtain the initial features of the face region includes: normalizing the size of the face region; and extracting initial features of the normalized face region to obtain the initial features of the face region.

[0007] In one embodiment of the present invention, the corresponding video clip is input into the target detection network to detect the behavior of the person, including: decomposing the video clip into multiple video frames; for each video frame: inputting the video frame into the backbone network of the target detection network to obtain feature maps of different scales; inputting the feature maps of different scales into the neck network of the target detection network, and based on the feature pyramid structure and the path aggregation structure, bidirectionally fusing the feature maps of each scale to obtain multiple fused feature maps; inputting each fused feature map into the prediction network of the target detection network to obtain the behavior of the person in the video frame; and combining the behavior of the person in each video frame to obtain the behavior of the person in the video clip.

[0008] In one embodiment of the present invention, the speech emotion recognition network is an LSTM, and the corresponding audio segment is input into the speech emotion recognition network to obtain the emotion of the audio segment, including: extracting multiple feature vectors from the audio segment according to the time sequence; arranging the extracted feature vectors in sequence to form a feature vector sequence; capturing the time dependency of each feature vector in the feature vector sequence to generate an emotion embedding vector;

[0009] The emotion embedding vector is feature mapped to obtain the emotion corresponding to the audio clip.

[0010] In one embodiment of the present invention, the generation of comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area includes: summarizing the expressions corresponding to each classroom area, and determining the comprehensive expressions based on the majority voting method; summarizing the behaviors corresponding to each classroom area, and determining the comprehensive behaviors based on the majority voting method; summarizing the emotions corresponding to each classroom area, and determining the comprehensive emotions based on the majority voting method.

[0011] In one embodiment of the present invention, the expressions corresponding to each classroom area are summarized, and a comprehensive expression is determined based on a majority voting method, including: summarizing the expressions corresponding to each classroom area; counting the number of occurrences of each category of expressions in all summarized expressions, and determining the expression with the largest number of occurrences, and using it as a candidate expression; analyzing the number of categories of the candidate expressions: if there is only one category, the candidate expression is used as a comprehensive expression; if there are multiple categories, the comprehensive expression is determined based on the confidence of each candidate expression.

[0012] In one embodiment of the present invention, the comprehensive expression is determined based on the confidence of each candidate expression, including: for all candidate expressions of the same category: counting the confidence of each candidate expression to generate a comprehensive confidence of the category; from each comprehensive confidence, selecting the candidate expression with the highest comprehensive confidence as the comprehensive expression.

[0013] In one embodiment of the present invention, the teaching quality is evaluated based on the comprehensive expression, the comprehensive behavior and the comprehensive emotion, including: assigning preset scores to the comprehensive expression, the comprehensive behavior and the comprehensive emotion, weighting each score according to a preset weight to obtain a comprehensive score; and evaluating the teaching quality according to the comprehensive score.

[0014] In one embodiment of the present invention, a teaching quality evaluation system is also provided, and the system includes: a data acquisition module, which is used to simultaneously acquire video clips and corresponding audio clips of different classroom areas during the same teaching activity; for each classroom area: a face detection module, which is used to input the corresponding video clip into a face recognition network to obtain the face area in the video clip; an expression recognition module, which is used to input the face area into the expression recognition network to obtain the expression of the face; a behavior detection module, which is used to input the corresponding video clip into a target detection network to detect the behavior of the character; an emotion recognition module, which is used to input the corresponding audio clip into a speech emotion recognition network to obtain the emotion of the audio clip; a comprehensive analysis module, which generates comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area; and an evaluation module, which is used to evaluate the teaching quality based on the comprehensive expressions, the comprehensive behaviors and the comprehensive emotions.

[0015] As described above, a teaching quality evaluation method and system of the present invention has the following beneficial effects: video clips and corresponding audio clips of different classroom areas during the same teaching activity are simultaneously obtained to achieve comprehensive and multi-angle information collection, and obtain the class status information of students in different classroom areas. Through face recognition, expression recognition, target detection and voice emotion recognition network, the acquired video clips and audio clips are processed and analyzed to generate the corresponding expressions, behaviors and emotions of each classroom area. By summarizing the expressions, behaviors and emotions of all classroom areas, the comprehensive expressions, behaviors and emotions corresponding to the teaching activity can be obtained. By analyzing the comprehensive expressions, behaviors and emotions, an objective evaluation of the teaching quality can be achieved. By acquiring student information from different areas, the present invention can analyze students' classroom performance more accurately and comprehensively, and realize multi-angle monitoring of teaching activities. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of an environment for a teaching quality evaluation method provided by an embodiment of the present invention;

[0017] Figure 2 An overall schematic diagram of a collection and processing device provided by an embodiment of the present invention;

[0018] Figure 3 A flow chart of a method for evaluating teaching quality provided by an embodiment of the present invention;

[0019] Figure 4 Shown is a structural block diagram of a teaching quality evaluation system provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0022] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0023] The present invention provides a method for evaluating teaching quality, and simultaneously obtains video clips and corresponding audio clips of different classroom areas during the same teaching activity, so as to realize comprehensive and multi-angle information collection and obtain the class status information of students in different classroom areas. Through face recognition, expression recognition, target detection and voice emotion recognition network, the obtained video clips and audio clips are processed and analyzed, so as to generate the expression, behavior and emotion corresponding to each classroom area. The expression, behavior and emotion of all classroom areas are summarized to obtain the comprehensive expression, behavior and emotion corresponding to the teaching activity. By analyzing the comprehensive expression, behavior and emotion, an objective evaluation of teaching quality can be achieved. By combining audio and video multimodal data, the expression, behavior and emotion information of students in different classroom areas can be obtained at the same time, and the learning status of students can be comprehensively and comprehensively analyzed from multiple dimensions to ensure the accuracy of the evaluation results. In addition, through this fusion of multiple data sources, even if the data quality of a certain mode is poor, a robust judgment can still be made through the data of other modes, avoiding the situation that the final analysis result is seriously affected when the data is abnormal due to excessive reliance on a single data. By obtaining student information from different areas, the present invention can analyze students' classroom performance more accurately and comprehensively, and realize multi-angle monitoring of teaching activities. On the one hand, the recognized student's expression, behavior and emotion can be used to specifically record each student's class situation, making it easier for teachers to provide one-on-one customized teaching. On the other hand, the present invention can capture each student's dynamic information more comprehensively through multiple (such as one per person) acquisition and processing devices, improving the problem that ordinary cameras are difficult to capture the situation of all classmates and have limitations such as occlusion and blind spots. It helps teachers to more efficiently grasp the learning progress of each student, understand the shortcomings of their own teaching links, and is more conducive to regulating students' classroom order.

[0024] See also Figure 1The present invention provides a teaching quality evaluation system, the system includes a collection and processing device 2 arranged in different classroom areas in a teaching scene and a server 1 arranged in the cloud, the server 1 includes a face detection module, an expression recognition module, a behavior detection module, an emotion recognition module, and a comprehensive analysis module, wherein: the collection and processing device 2 is used to collect video clips and corresponding audio clips of the listeners in the corresponding classroom area; for each classroom area: the face detection module of the server 1 is used to input the corresponding video clip into a face recognition network to obtain the face area in the video clip; the expression recognition module of the server 1 is used to input the face area into the expression recognition network to obtain the expression of the face; the behavior detection module of the server 1 is used to input the corresponding video clip into a target detection network to detect the behavior of the character; the emotion recognition module of the server 1 is used to input the corresponding audio clip into a voice emotion recognition network to obtain the emotion of the audio clip; the comprehensive analysis module of the server 1 is used to generate comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area, and evaluate the teaching quality based on the comprehensive expressions, the comprehensive behaviors and the comprehensive emotions.

[0025] See also Figure 1 and Figure 2 Specifically, several acquisition and processing devices 2 can be set in the teaching scene. Each acquisition and processing device 2 can be a camera and a corresponding sound receiving device (such as a microphone, etc.) set on the desk. Each camera and the corresponding sound receiving device cover a certain classroom area. Furthermore, the camera and the sound receiving device can be integrated into a whole to form a portable video auxiliary system, such as Figure 2 As shown. It collects video clips of students in the classroom area through a camera, and collects audio clips of students in the corresponding classroom area through a microphone on it. Each acquisition and processing device 2 uploads the collected video clips and corresponding audio clips to the server 1 in the cloud for processing, and evaluates the teaching quality of the teacher according to the processing results. Among them, the server 1 in the cloud inputs each video clip into the face recognition network, and obtains the face area in each video clip, and inputs each face area into the expression recognition network to obtain the expression of each face, and detects the behavior of the characters in each video clip and the emotion of the corresponding audio clip through the target detection network and the speech emotion recognition network. By analyzing the expression, behavior and emotion corresponding to each classroom area, the comprehensive expression, comprehensive behavior and comprehensive emotion of the entire teaching activity in the current period are determined based on the majority voting method, and the teaching quality of the current period is evaluated based on the comprehensive expression, comprehensive behavior and comprehensive emotion of the period. Further, in order to improve the processing rate, the behavior and emotion of the characters can also be detected in the acquisition and processing device 2, and the face area and expression can be detected in the server 1 in the cloud.

[0026] See also Figure 3 , the evaluation method of teaching quality includes the following steps:

[0027] S1. Simultaneously obtain video clips and corresponding audio clips of different classroom areas during the same teaching activity.

[0028] See also Figure 1 and Figure 3 During the same teaching activity, use the acquisition and processing devices 2 installed at different locations in the teaching scene to collect video clips and synchronized audio clips of each classroom area in real time. Among them, teaching activities refer to teaching and interactive activities conducted by teachers and students in a specific space, including but not limited to lectures, experiments, multimedia presentations, etc., and classroom areas refer to specific areas monitored by different acquisition and processing devices 2 during teaching activities. Each acquisition and processing device 2 uploads the collected video and audio clips to the server 1 in the cloud so that the server 1 can perform subsequent processing. Exemplarily, at least two laboratory tables are set in the teaching scene, and a acquisition and processing device 2 is installed on each laboratory table, which is used to use the laboratory table as a classroom area to collect video clips and corresponding audio clips of students in the classroom area. At the same preset time interval, each acquisition and processing device 2 uploads the acquired video clips and corresponding audio clips to the server 1 in the cloud.

[0029] S2. Input the corresponding video clip into the face recognition network to obtain the face area in the video clip.

[0030] See also Figure 1 , for each video clip: the server 1 in the cloud decomposes the video clip into multiple video frames, inputs each video frame into the face recognition network, obtains the face area and face identification in the video frame, wherein the face identification is used to represent the identity information of the person corresponding to the face. Based on the face area corresponding to each video frame, the detection and tracking of the face area in the video clip can be achieved. Specifically, the face recognition network in the present invention is MobileFaceNet. For each video frame in the video clip, detecting the face area and face identification from the video frame includes: inputting the video frame into the face detector to detect the face area; inputting the face area into the face recognition network, extracting the facial features of the face area according to the depth separable convolution and residual connection, and obtaining the facial feature vector; determining the corresponding face identification based on the facial feature vector. Through the above process, the face area and the corresponding face identification of each video frame in the video clip can be obtained, and the face area in the video clip can be obtained by time-sequential tracking of these face areas. The detection frames of the face regions of the same face identifier may be connected on the time axis to obtain the time sequence trajectory of each independent face in the video clip, so as to determine the face region in the video clip.

[0031] S3: inputting the face region into an expression recognition network to obtain the facial expression.

[0032] In one embodiment of the present invention, the step of inputting the face region into an expression recognition network to obtain the facial expression includes:

[0033] Performing initial feature extraction on the face region to obtain initial features of the face region;

[0034] Based on multi-level residual connections, deep features are extracted from the initial features of the face area to obtain deep features;

[0035] The deep features are feature mapped to obtain facial expressions.

[0036] The expression recognition network in the present invention is a residual network, which includes but is not limited to Resnet18, Resnet34, Resnet50, etc., which are not limited here. In order to balance the computational complexity and recognition accuracy, Resnet18 is used as the expression recognition network in the present invention. Since Resnet18 has a shallower network depth, it can more effectively extract the expression features of the face under less computing power to achieve the real-time and accuracy requirements of expression recognition. Specifically, the face area in the video clip is input into the expression recognition network, and the initial features such as the edge and texture of the face area are extracted through the first convolutional layer in the expression recognition network. Then, through multiple residual modules, the input information of each residual module is added to the output information of the residual module by means of jump connections, so that the input information is combined with the output information as the input of the next residual module, thereby forming a residual mapping, which can effectively alleviate the problem of gradient disappearance in the model training stage. Through multiple residual modules, the initial features of the face area are extracted in depth to obtain deep features. Furthermore, the deep features are feature mapped to obtain the facial expression, including: performing global average pooling on the deep features, taking the average of the features of each channel to obtain a feature vector of fixed length. The feature vector is input into a fully connected layer, mapped to the expression category space, and expressions of preset categories are obtained. Softmax processing is performed on the expressions of preset categories to obtain the confidence of each category of expression, and the expression with the highest confidence is selected as the facial expression.

[0037] In one embodiment of the present invention, the initial feature extraction of the face region to obtain the initial features of the face region includes:

[0038] Normalize the size of the face area;

[0039] Initial feature extraction is performed on the normalized face region to obtain initial features of the face region.

[0040] Normalize the size of the face region to ensure that the images input to the expression recognition network have the same size. Input the normalized face region image to the first convolution layer of the expression recognition network, and perform sliding convolution operations on the image through each convolution kernel to extract the initial features of the face region. Obtain the initial feature map of the face region.

[0041] S4: Input the corresponding video clip into the target detection network to detect the behavior of the person.

[0042] In the present invention, the target detection network is a YOLO series network, and those skilled in the art can adaptively select it based on actual needs. However, in order to achieve a balance between detection accuracy and computational efficiency, YOLOv5 is used as the target detection network in the present invention.

[0043] Specifically, in one embodiment of the present invention, the corresponding video clip is input into the target detection network to detect the behavior of the person, including:

[0044] Decomposing a video clip into multiple video frames;

[0045] For each video frame:

[0046] The video frames are input into the backbone network of the target detection network to obtain feature maps of different scales; the feature maps of different scales are input into the neck network of the target detection network, and based on the feature pyramid structure and path aggregation structure, the feature maps of each scale are bidirectionally fused to obtain multiple fused feature maps;

[0047] The fused features are input into the prediction network of the target detection network to obtain the behavior of the person in the video frame;

[0048] The behaviors of the characters in each video frame are combined to obtain the behaviors of the characters in the video clip.

[0049] The video clip is decomposed into multiple video frames arranged in sequence frame by frame or by calling key frames (such as extracting one frame every two frames, etc.). For each video frame: it is input into the backbone network of the target detection network, and the feature information of the image is extracted through multiple convolutional layers, and multiple feature maps of different scales are obtained accordingly. The feature maps of each scale are respectively input into the neck network of the target detection network. Through the feature pyramid structure, the deep high-level semantic feature map is upsampled step by step and fused with the shallow low-level semantic feature map in a bottom-up manner, so as to improve the detection ability of small targets, and multiple preliminary fused feature maps are obtained, wherein each preliminary fused feature map has a different scale. Then, through the path aggregation structure, the preliminary fused feature maps are further fused in a top-down manner to obtain multiple fused feature maps, wherein each fused feature map has a different scale. These fused feature maps of different scales are input into the prediction network, and the behavior of the person in the corresponding video frame is obtained through the non-maximum suppression method, wherein the behavior of the person includes the coordinates of the person's bounding box and the confidence of the corresponding category. Based on the majority voting method, the behavior category that appears most frequently in the video clip is selected as the behavior of the person in the video clip.

[0050] S5. Input the corresponding audio clip into the speech emotion recognition network to obtain the emotion of the audio clip.

[0051] After preprocessing the audio clips such as denoising and normalization, the preprocessed audio clips are input into the speech emotion recognition network to obtain the corresponding emotions. The speech emotion recognition network includes but is not limited to LSTM, BERT, etc., and those skilled in the art can adaptively select the corresponding network based on actual needs, which is not limited here.

[0052] In one embodiment of the present invention, the speech emotion recognition network is an LSTM, and the step of inputting the corresponding audio segment into the speech emotion recognition network to obtain the emotion of the audio segment includes:

[0053] Extract multiple feature vectors from the audio clips based on the time sequence;

[0054] Arrange the extracted feature vectors in order to form a feature vector sequence;

[0055] Capturing the time dependency of each feature vector in the feature vector sequence to generate a sentiment embedding vector;

[0056] The emotion embedding vector is feature mapped to obtain the emotion corresponding to the audio clip.

[0057] According to the preset sampling duration, the audio clip is divided into multiple audio frames, wherein each audio frame contains the audio information within the sampling duration. Exemplarily, the sampling duration is 10 milliseconds, and each audio frame includes 10 milliseconds of audio information. And based on the audio feature extraction method, a feature vector is extracted from each audio frame, wherein the audio feature extraction method includes but is not limited to Mel frequency cepstral coefficients, pitch-based feature extraction method, spectrum centroid method, etc. All feature vectors in the audio clip are arranged according to the extracted time sequence to form a feature vector sequence. The feature vector sequence is input into the LSTM network, and each feature vector in the input feature vector sequence is processed according to the corresponding time step through the gating mechanism to obtain the emotion embedding vector. The emotion embedding vector is input into the fully connected layer of the LSTM network, mapped to the emotion category space, and the confidence of each emotion category is obtained, and the emotion category with the highest confidence is selected as the emotion corresponding to the audio clip.

[0058] S6. Generate comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area.

[0059] Specifically, in one embodiment of the present invention, generating comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area includes:

[0060] Summarize the expressions corresponding to each classroom area and determine the comprehensive expression based on majority voting:

[0061] Aggregate the behaviors corresponding to each classroom area and determine the comprehensive behavior based on majority voting;

[0062] The sentiments corresponding to each classroom area were summarized and the comprehensive sentiment was determined based on the majority voting method.

[0063] The expressions corresponding to each classroom area are summarized, and based on the majority voting method, the expression category with the most occurrences is selected as the comprehensive expression during the current video clip acquisition. Similarly, the comprehensive behavior and comprehensive emotion during the current video clip acquisition can be obtained.

[0064] In one embodiment of the present invention, the steps of aggregating the expressions corresponding to the various classroom areas and determining the comprehensive expression based on a majority voting method include:

[0065] Summarize the expressions corresponding to each classroom area;

[0066] Count the number of times each category of expressions appears in all the summarized expressions, and determine the expression with the most occurrences as a candidate expression:

[0067] Analyze the number of categories of candidate expressions:

[0068] If the category is one, the candidate expression is taken as a comprehensive expression;

[0069] If there are multiple categories, a comprehensive expression is determined based on the confidence of each candidate expression.

[0070] For the current teaching activity (such as a class), the expressions detected in all classroom areas of the teaching activity within the same time period are summarized. And the number of times each expression category appears in the summarized results is counted, and the expression category with the most appearances is used as a candidate expression. It can be understood that if the number of appearances of multiple expression categories is the same and the largest number, these categories will all be used as candidate expressions. Determine the number of categories of candidate expressions: If there is only one category of candidate expressions, it means that only one expression appears the most times, and the candidate expression category can be directly used as the comprehensive expression of the current video clip. Anyway, if the number of candidate expression categories is greater than one, the confidence of each candidate expression category is compared, and the category with the highest confidence is selected as the comprehensive expression of the video clip.

[0071] Specifically, in one embodiment of the present invention, determining the comprehensive expression based on the confidence of each candidate expression includes:

[0072] For all candidate expressions of the same category: calculate the confidence of each candidate expression and generate the comprehensive confidence of the category;

[0073] From each comprehensive confidence, select the candidate expression with the highest comprehensive confidence as the comprehensive expression.

[0074] For each candidate expression category, the confidence of the category in all classroom areas is summarized, and the comprehensive confidence of the category can be determined by summing or averaging. Then the comprehensive confidence of all candidate expression categories is compared, and the category with the highest confidence is selected as the comprehensive expression.

[0075] S7. Evaluate the teaching quality based on the comprehensive expression, the comprehensive behavior and the comprehensive emotion.

[0076] In one embodiment of the present invention, the evaluating of teaching quality based on the comprehensive expression, the comprehensive behavior and the comprehensive emotion includes:

[0077] Assigning preset scores to the comprehensive expressions, the comprehensive behaviors, and the comprehensive emotions, and weighting the scores according to preset weights to obtain a comprehensive score;

[0078] The teaching quality is evaluated based on the comprehensive score.

[0079] Based on the importance of each expression, behavior or emotion in classroom teaching, a preset score is assigned to each comprehensive expression, comprehensive behavior and comprehensive emotion. For example, the score corresponding to the comprehensive expression of happiness is 70, and the score corresponding to the comprehensive expression of confusion is 10. For each category, a corresponding weight is also set, for example, the weight of the comprehensive expression is 0.3, and the weight of the comprehensive behavior is 0.5. The weight and the corresponding score are weighted to obtain a comprehensive score. The comprehensive score is compared with the preset scoring standard to obtain the teaching quality corresponding to the current moment of obtaining the video clip. For example, if the comprehensive score is 0-50, the teaching quality is poor, the comprehensive score is 51-70, the teaching quality is qualified, the comprehensive score is 71-90, the teaching quality is good, and the comprehensive score is greater than 90 points, then the teaching quality is excellent. Further, all the comprehensive scores in the teaching activity can be summarized to obtain the comprehensive score corresponding to the entire teaching activity, and the comprehensive score is compared with the preset scoring standard to evaluate the teaching quality.

[0080] It should be noted that the above processing process can be processed entirely by the server 1 in the cloud, or it can be based on a combination of edge computing and cloud computing. Considering that face recognition and speech emotion recognition are processed in the cloud due to their large data volume or high computational complexity, the ResNet classification network and the YOLO target detection network can be deployed on the local device to achieve real-time recognition. Specifically, the acquisition and processing device 2 obtains the video clips and the corresponding audio clips of the corresponding classroom area, and uploads them to the server 1. The server 1 inputs the video clips into the face recognition network to obtain the face area, and inputs the audio clips into the speech emotion recognition network to obtain the emotion of the audio clip; the server 1 sends the face area to the corresponding acquisition and processing device 2, and the acquisition and processing device 2 inputs the face area into the expression recognition network to obtain the expression of the face, and at the same time inputs the video clips into the target detection network to detect the behavior of the character, and uploads the expression of the face and the behavior of the character to the server 1. The server 1 generates comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area, and evaluates the teaching quality based on the comprehensive expressions, the comprehensive behaviors and the comprehensive emotions. This combination of cloud and local ends enables efficient reasoning and calculation of the model.

[0081] See also Figure 4, the teaching quality evaluation system 100 includes: a data acquisition module 110, a face detection module 120, an expression recognition module 130, a behavior detection module 140, an emotion recognition module 150, a comprehensive analysis module 160 and an evaluation module 170. The above-mentioned data acquisition module 110 is used to simultaneously obtain video clips and corresponding audio clips of different classroom areas during the same teaching activity. For each classroom area: the face detection module 120 is used to input the corresponding video clip into the face recognition network to obtain the face area in the video clip. The expression recognition module 130 is used to input the face area into the expression recognition network to obtain the expression of the face. The behavior detection module 140 is used to input the corresponding video clip into the target detection network to detect the behavior of the character. The emotion recognition module 150 is used to input the corresponding audio clip into the voice emotion recognition network to obtain the emotion of the audio clip. The comprehensive analysis module 160 generates comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area. The evaluation module 170 is used to evaluate the teaching quality based on the comprehensive expression, the comprehensive behavior and the comprehensive emotion.

[0082] The specific definition of the teaching quality evaluation system can be found in the definition of the teaching quality evaluation method above, which will not be repeated here. Each module in the above teaching quality evaluation system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory in the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0083] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0084] In summary, the present invention discloses a method and system for evaluating teaching quality, which simultaneously obtains video clips and corresponding audio clips of different classroom areas during the same teaching activity, so as to realize comprehensive and multi-angle information collection and obtain the class status information of students in different classroom areas. Through face recognition, expression recognition, target detection and voice emotion recognition network, the acquired video clips and audio clips are processed and analyzed to generate the expression, behavior and emotion corresponding to each classroom area. By summarizing the expressions, behaviors and emotions of all classroom areas, the comprehensive expressions, behaviors and emotions corresponding to the teaching activity can be obtained. By analyzing the comprehensive expressions, behaviors and emotions, an objective evaluation of teaching quality can be achieved. By obtaining student information from different areas, the present invention can analyze students' classroom performance more accurately and comprehensively, and realize multi-angle monitoring of teaching activities. On the one hand, the class situation of each student can be specifically recorded through the expressions, behaviors and emotions of the identified students, which is convenient for teachers to do one-to-one customized teaching. On the other hand, the present invention can capture the dynamic information of each student more comprehensively through multiple (such as one per person) acquisition and processing devices, improving the problem that ordinary cameras are difficult to capture the situation of all classmates and have the limitations of occlusion and blind spots. It helps teachers to more efficiently grasp the learning progress of each student, understand the deficiencies of their own teaching links, and is more conducive to regulating the classroom order of students. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0085] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for evaluating teaching quality, characterized in that: The method comprises: Simultaneously obtain video clips and corresponding audio clips of different classroom areas during the same teaching activity; For each classroom area: Input the corresponding video clip into the face recognition network to obtain the face area in the video clip; Inputting the face region into an expression recognition network to obtain the facial expression; Input the corresponding video clips into the object detection network to detect the behavior of the person; Input the corresponding audio clip into the speech emotion recognition network to obtain the emotion of the audio clip; Generate comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area; The teaching quality is evaluated based on the comprehensive expression, the comprehensive behavior and the comprehensive emotion.

2. The teaching quality evaluation method according to claim 1, characterized in that: The step of inputting the face region into an expression recognition network to obtain the expression of the face includes: Extracting initial features of the face region to obtain initial features of the face region; Based on multi-level residual connections, deep features are extracted from the initial features of the face area to obtain deep features; The deep features are feature mapped to obtain facial expressions.

3. The teaching quality evaluation method according to claim 2, characterized in that: The extracting initial features of the face region to obtain initial features of the face region includes: Normalize the size of the face area; Initial feature extraction is performed on the normalized face region to obtain initial features of the face region.

4. The teaching quality evaluation method according to claim 1, characterized in that: The corresponding video clip is input into the target detection network to detect the behavior of the person, including: Decomposing a video clip into multiple video frames; For each video frame: Input the video frame into the backbone network of the target detection network to obtain feature maps of different scales; The feature maps of different scales are input into the neck network of the target detection network. Based on the feature pyramid structure and path aggregation structure, the feature maps of each scale are bidirectionally fused to obtain multiple fused feature maps. The fused features are input into the prediction network of the target detection network to obtain the behavior of the person in the video frame; The behaviors of the characters in each video frame are combined to obtain the behaviors of the characters in the video clip.

5. The teaching quality evaluation method according to claim 1, characterized in that: The speech emotion recognition network is an LSTM, and the corresponding audio segment is input into the speech emotion recognition network to obtain the emotion of the audio segment, including: Extract multiple feature vectors from the audio clips based on the time sequence; Arrange the extracted feature vectors in order to form a feature vector sequence; Capturing the time dependency of each feature vector in the feature vector sequence to generate a sentiment embedding vector; The emotion embedding vector is feature mapped to obtain the emotion corresponding to the audio clip.

6. The teaching quality evaluation method according to claim 1, characterized in that: The generation of comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area includes: Summarize the expressions corresponding to each classroom area and determine the comprehensive expression based on majority voting: Aggregate the behaviors corresponding to each classroom area and determine the comprehensive behavior based on majority voting; The sentiments corresponding to each classroom area were summarized and the comprehensive sentiment was determined based on the majority voting method.

7. The teaching quality evaluation method according to claim 6, characterized in that: The expression corresponding to each classroom area is summarized, and a comprehensive expression is determined based on a majority voting method, including: Summarize the expressions corresponding to each classroom area; Count the number of times each category of expressions appears in all the summarized expressions, and determine the expression with the most occurrences as a candidate expression: Analyze the number of categories of candidate expressions: If the category is one, the candidate expression is taken as a comprehensive expression; If there are multiple categories, a comprehensive expression is determined based on the confidence of each candidate expression.

8. The teaching quality evaluation method according to claim 7, characterized in that: The step of determining a comprehensive expression based on the confidence level of each candidate expression comprises: For all candidate expressions of the same category: calculate the confidence of each candidate expression and generate the comprehensive confidence of the category; From each comprehensive confidence, select the candidate expression with the highest comprehensive confidence as the comprehensive expression.

9. The teaching quality evaluation method according to claim 1, characterized in that: The evaluating of teaching quality based on the comprehensive expression, the comprehensive behavior and the comprehensive emotion includes: Assigning preset scores to the comprehensive expressions, the comprehensive behaviors, and the comprehensive emotions, and weighting the scores according to preset weights to obtain a comprehensive score; The teaching quality is evaluated based on the comprehensive score.

10. A teaching quality evaluation system, characterized in that: The system comprises: A data acquisition module, used to simultaneously acquire video clips and corresponding audio clips of different classroom areas during the same teaching activity; For each classroom area: A face detection module is used to input the corresponding video clip into the face recognition network to obtain the face area in the video clip; An expression recognition module, used for inputting the face area into an expression recognition network to obtain the facial expression; The behavior detection module is used to input the corresponding video clips into the object detection network to detect the behavior of the person; The emotion recognition module is used to input the corresponding audio clip into the speech emotion recognition network to obtain the emotion of the audio clip; The comprehensive analysis module generates comprehensive expressions, comprehensive behaviors and comprehensive emotions based on the expressions, behaviors and emotions corresponding to each classroom area; An evaluation module is used to evaluate the teaching quality based on the comprehensive expression, the comprehensive behavior and the comprehensive emotion.

Images