A method and system for evaluating student classroom performance based on multimodal audio and video.

Abstract

This invention discloses a method and system for evaluating student classroom performance based on multimodal audio and video, belonging to the field of classroom education technology. The method includes acquiring audio and video streams from the classroom; separating the independent audio stream for each student and converting it into first text information; acquiring the position coordinates of the blackboard writing implements and the second text information of the blackboard content for each frame of the video stream; simultaneously estimating the position coordinates of the pupil center point and corneal reflection point based on the gradient map of the student's eye region; estimating the student's visual axis direction based on the pupil center point and corneal reflection point position coordinates, and connecting the pupil center point and the position coordinates of the blackboard writing implements to predict the student's visual direction; using the overlap between the predicted and estimated visual axis directions, as well as the correlation between the two text information, as scoring indicators to evaluate the student's classroom performance. This invention can improve the accuracy of evaluating student classroom performance.

Description

Technical Field

[0001] This invention belongs to the field of classroom education technology, specifically relating to a method and system for evaluating student classroom performance based on multimodal audio and video. Background Technology

[0002] In the field of education, student classroom performance is one of the important indicators for measuring teaching quality and learning outcomes. It not only reflects the degree to which students absorb and understand knowledge, but also reflects whether the teacher's teaching methods and strategies are appropriate.

[0003] Patent application number 202210971306.6 discloses a method for evaluating student classroom performance based on multimodal audio and video technology. It uses voiceprint recognition technology to statistically analyze student speaking frequency and evaluate student activity levels. Keyframe extraction technology is used to extract keyframes from the video, selecting student facial images that reflect focus as positive or negative samples. A machine learning model is used to train the positive and negative training samples to obtain a focus detection model. The detection model is then used to detect keyframe images to obtain the frequency of students' small movements in class. Based on the records in the classroom evaluation data table, student performance is categorized.

[0004] However, the existing technology has the following problems: The existing technology assesses students’ classroom performance based on their classroom speaking activity, classroom learning focus, and frequency of classroom fidgeting. However, some students may contain content unrelated to the classroom teaching during their speeches. At the same time, students’ classroom learning focus is based on the detection of students’ facial focus. Some students may be facing the direction of the lecture, but their eyes are not. Also, some students may have classroom fidgeting, but they are still listening attentively. In view of this, the above methods are difficult to accurately assess students’ classroom performance. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a method and system for evaluating student classroom performance based on multimodal audio and video, which can further improve the accuracy of student classroom performance evaluation.

[0006] This invention provides the following technical solution:

[0007] Firstly, a method for evaluating student classroom performance based on multimodal audio and video is provided, including:

[0008] Obtain the audio and video streams from the classroom;

[0009] The audio stream is preprocessed to separate the independent audio stream for each student, and each independent audio stream is converted into the corresponding first text information;

[0010] For each frame of the series of image frames corresponding to the video stream, the blackboard content and blackboard tools are identified, and the position coordinates of the blackboard tools and the second text information of the blackboard content are obtained. At the same time, for each frame of the image, the student's eye area is detected, and the position coordinates of the pupil center point and corneal reflection point are estimated based on the gradient map of the student's eye area.

[0011] The direction of the student's line of sight is estimated based on the position coordinates of the pupil center point and the corneal reflection point, and the position coordinates of the pupil center point and the blackboard writing tool are connected to predict the direction of the student's line of sight.

[0012] The degree of overlap between the predicted student gaze direction and the estimated student gaze optical axis direction, as well as the correlation between the two textual pieces of information, are used as scoring indicators to evaluate students' classroom performance.

[0013] Optionally, the audio and video streams of the classroom can be acquired synchronously and separately, or the classroom audio and video captured by the camera can be acquired separately.

[0014] Optionally, separating the independent audio stream for each student specifically includes:

[0015] The preprocessed audio stream is decomposed into multiple short frames and then windowed.

[0016] Perform a Fast Fourier Transform on each windowed frame to convert it from the time domain to the frequency domain, and solve for the modulus square to obtain the power spectrum of the speech signal.

[0017] The power spectrum is passed through a set of Mel-scale filters to obtain the Mel spectrum, and then the logarithm is taken to obtain the logarithmic Mel spectrum.

[0018] The MFCC eigenvectors are obtained by performing a discrete cosine transform on the logarithmic Mel spectrum.

[0019] The FastICA algorithm is used to process the MFCC feature vectors and separate the independent audio streams of each student.

[0020] Optionally, the process of recognizing the blackboard content and the blackboard writing tools, and obtaining the position coordinates of the blackboard writing tools and the second text information of the blackboard content, specifically involves:

[0021] For each frame in the series of image frames corresponding to the video stream, a pre-trained optical character recognition model is used to recognize the whiteboard content and convert it into second text information;

[0022] For each frame in the series of image frames corresponding to the video stream, the Gaussian mixture background difference method is used to obtain the image of the whiteboard tool, and the whiteboard user image is binarized. All pixels are traversed to obtain the position coordinates of the whiteboard tool.

[0023] Optionally, the step of using the Gaussian mixture background difference method to obtain the image of the blackboard writing tool specifically involves:

[0024] Several Gaussian distributions are selected to describe the gray value changes of each pixel. In the initial image frame without any writing tools, the initial mean, variance and weight of each Gaussian distribution are set to construct and initialize the background model.

[0025] Read a series of image frames sequentially, iterate through each pixel of each image frame, and match it with the Gaussian distribution of the background model to obtain the difference between each pixel and the corresponding Gaussian distribution. Pixels with a difference value exceeding the preset variance are marked as blackboard tool points, and pixels with a difference value not exceeding the preset variance are marked as blackboard background points, thus obtaining the blackboard tool image of the series of image frames.

[0026] Optionally, the step of detecting the student's eye region and estimating the coordinates of the pupil center point and corneal reflection point based on the gradient map of the student's eye region specifically involves:

[0027] The student's eye region was detected in each frame of a series of image frames using a pre-trained eye detection model.

[0028] The gradient map of the student's eye region corresponding to each frame of the image was obtained using the median difference method.

[0029] The coordinates of the center point of the student's pupil and the corneal reflection point are estimated based on the gradient map, specifically as follows:

[0030] Iterate through each pixel in the gradient map of the student's eye region and calculate the position vector of the current pixel relative to all remaining pixels.

[0031]

[0032] in, Let x be the position vector of the current pixel and pixel j. i ,y i (x) represents the coordinates of the current pixel. j ,y j () represents the coordinates of another pixel.

[0033] Perform a dot product between the gradient value and the position vector of the current pixel, and accumulate all the dot product operations of the current pixel to obtain the sum of the dot products of the current pixel;

[0034]

[0035] Among them, Y i This is the sum of the dot products of the current pixel i; Let i be the gradient value of the current pixel. Let i be the position vector between the current pixel i and the nth pixel.

[0036] Compare the sum of the dot products of all pixels. The pixel with the largest sum of dot products is estimated to be the center point of the student's pupil. Obtain the coordinates of the center point of the student's pupil.

[0037] A preset segmentation threshold is used to segment corneal reflective points and background. Pixels exceeding the segmentation threshold are considered corneal reflective points, while pixels below the threshold are considered background. The position coordinates of the corneal reflective points are then obtained.

[0038] Optionally, estimating the optical axis direction of the student's gaze based on the coordinates of the pupil center point and the corneal reflection point specifically involves:

[0039] Assuming the cornea is spherical, the incident angle θ1 and refraction angle θ2 of the light on the cornea are calculated based on the coordinates of the center of the pupil and the corneal reflection point. The direction of the optical axis of the student's line of sight is estimated by the intersection of the pupil center and the refraction angle.

[0040] θ1 = arccos(cos(θ1));

[0041]

[0042] in, Let P be the vector from the center of the student's pupil to the corneal reflective point, where P and R represent the positions of the student's pupil center and corneal reflective point, respectively. The normal to the cornea;

[0043] θ2 = arcsin(K·sin(θ1));

[0044]

[0045] Here, K is a parameter used to simplify calculations and has no practical meaning.

[0046] Optionally, the correlation between the two textual pieces of information can be determined as follows:

[0047] The two text messages are segmented into words, irrelevant words are removed, and a pre-trained bag-of-words model is used to convert the segmented text messages into numerical feature vectors.

[0048] Calculate the cosine of the angle between two numerical feature vectors, and obtain the correlation between the two text information based on the cosine of the angle.

[0049]

[0050] Where θ is the angle between the two numerical eigenvectors. and These are the magnitudes of the two numerical eigenvectors. Cosθ is the dot product of two numerical feature vectors, where cosθ ∈ [0, 1], and the closer it is to 1, the higher the correlation.

[0051] Optionally, the method of using the degree of overlap between the predicted student's gaze direction and the estimated student's gaze optical axis direction, as well as the correlation between the two textual pieces of information, as scoring indicators to evaluate the student's classroom performance is as follows:

[0052] An initial score for classroom performance is preset for each student. Thresholds are set for overlap and relevance. If the overlap exceeds the threshold, it is used as the first bonus item, and a specified score is added. Otherwise, it is used as the first deduction item, and a specified score is subtracted. If the relevance exceeds the threshold, it is used as the second bonus item, and a specified score is added. Otherwise, it is used as the second deduction item, and a specified score is subtracted. This yields the final classroom performance score for each student.

[0053] Secondly, a student classroom performance evaluation system based on multimodal audio and video is provided, including:

[0054] The data acquisition module is used to acquire the audio and video streams from the classroom.

[0055] The audio stream processing module is used to preprocess the audio stream, separate the independent audio stream of each student, and convert each independent audio stream into the corresponding first text information;

[0056] The video stream processing module is used to identify the blackboard content and blackboard tools for each frame in the series of image frames corresponding to the video stream, and to obtain the position coordinates of the blackboard tools and the second text information of the blackboard content. At the same time, it also detects the student's eye area for each frame, and estimates the position coordinates of the pupil center point and corneal reflection point based on the gradient map of the student's eye area. Based on the position coordinates of the pupil center point and corneal reflection point, it estimates the optical axis direction of the student's gaze, and connects the position coordinates of the pupil center point and the blackboard tools to predict the direction of the student's gaze.

[0057] The scoring module is used to score students' classroom performance by using the degree of overlap between the predicted student's gaze direction and the estimated student's gaze axis direction, as well as the correlation between the two text information.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] This invention extracts the independent audio stream of each student from the audio stream and performs text recognition to output the first text information. It also extracts the second text information of the whiteboard content from each video frame from the video stream. The correlation between the two text information is compared based on timestamps to determine the student's performance in classroom language. Simultaneously, the image of the whiteboard tool used is extracted based on background subtraction from the video stream to determine its coordinates. The student's eye region is detected based on the video stream, and the coordinates of the pupil center and corneal reflection point are determined based on the student's eye region. The student's gaze direction is predicted based on the timestamp, the coordinates of the whiteboard tool, and the pupil center coordinates. The optical axis direction of the student's gaze is estimated based on the pupil center coordinates and the corneal reflection point. The overlap between the predicted and estimated gaze directions is compared based on the timestamps to determine the student's performance in classroom gaze. An initial value for classroom performance is added to or subtracted from both values ​​to evaluate the student's classroom performance. This application can avoid evaluation errors and improve the accuracy of classroom performance evaluation. Attached Figure Description

[0060] Figure 1 This is a flowchart of the student classroom performance evaluation method based on multimodal audio and video of the present invention;

[0061] Figure 2 This is a structural block diagram of a student classroom performance evaluation system based on multimodal audio and video according to the present invention;

[0062] Figure 3 This is another structural block diagram of the student classroom performance evaluation system based on multimodal audio and video of the present invention. Detailed Implementation

[0063] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be used to limit the scope of protection of the present invention. It should be noted that the term "comprising" and any variations thereof in the specification, claims and the above-mentioned drawings of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or devices.

[0064] Example 1

[0065] like Figure 1 As shown, a method for evaluating student classroom performance based on multimodal audio and video is provided, including the following steps:

[0066] S1: Obtain the audio and video streams from the classroom.

[0067] The audio and video streams of the classroom are captured synchronously and separately, or the classroom audio and video captured by the camera are obtained separately.

[0068] When separately capturing audio and video streams in the classroom, the timestamps of the capture need to be aligned. During the monitoring process, the whiteboard and all students can be captured directly.

[0069] Based on the classroom audio and video captured by the surveillance camera, the file format of the acquired classroom audio and video is first identified, and the corresponding decoder is selected to read the data in the classroom audio and video file. During the reading process, the audio stream and video stream are separated and decoded; that is, the video stream is decoded and recorded as a series of image frames corresponding to the video stream.

[0070] S2: Preprocess the audio stream, separate the independent audio stream for each student, and convert each independent audio stream into the corresponding first text information.

[0071] In this embodiment, the specific steps include: S21: preprocessing the audio stream, including conventional preprocessing methods such as noise reduction and volume adjustment.

[0072] S22: Separate the independent audio stream for each student, specifically including:

[0073] S221: Decompose the preprocessed audio stream into multiple short frames and perform windowing processing;

[0074] S222: Perform a fast Fourier transform on each windowed frame to convert it from the time domain to the frequency domain, and solve for the modulus square to obtain the power spectrum of the speech signal;

[0075] S223: Pass the power spectrum through a set of Mel-scale filters to obtain the Mel spectrum, then take the logarithm to obtain the logarithmic Mel spectrum;

[0076] S224: Perform a discrete cosine transform on the logarithmic Mel spectrum to obtain the MFCC eigenvectors;

[0077] S225: The FastICA algorithm is used to process the MFCC feature vectors and separate the independent audio streams of each student.

[0078] S23: Convert each independent audio stream into the corresponding first text information.

[0079] Specifically, a language model can be pre-trained to convert each independent audio stream into corresponding first text information. The pre-trained language model can refer to existing technologies.

[0080] S3: For each frame of the series of image frames corresponding to the video stream, identify the content written on the blackboard and the blackboard writing tools, and obtain the position coordinates of the blackboard writing tools and the second text information of the content written on the blackboard; at the same time, for each frame of the image, detect the student's eye area, and estimate the position coordinates of the center point of the pupil and the corneal reflection point based on the gradient map of the student's eye area.

[0081] S31: For each frame in the series of image frames corresponding to the video stream, use a pre-trained optical character recognition model to recognize the whiteboard content and convert it into second text information.

[0082] Optical character recognition models can refer to existing technologies.

[0083] S32: For each frame in the series of image frames corresponding to the video stream, the Gaussian mixture background difference method is used to obtain the image of the whiteboard tool, and the whiteboard user image is binarized. All pixels are traversed to obtain the position coordinates of the whiteboard tool.

[0084] S321: The image of the blackboard writing tools is obtained using the Gaussian mixture background difference method, specifically:

[0085] S321-1: Select several Gaussian distributions to describe the gray value changes of each pixel. In the initial image frame without any whiteboard tools, set the initial mean, variance and weight of each Gaussian distribution to build and initialize the background model.

[0086] The specific expression is:

[0087]

[0088] Where, χ bg·t Let α be the background value of the image frame at time t. i,t Let β be the weight of the i-th pixel at time t. i,t Let be the mean of the i-th pixel at time t, m represent the background of the blackboard tool image generated by the Gaussian components of m pixels, b represent the bias, and g represent the Gaussian distribution function.

[0089] S321-2: Read the series of image frames sequentially, traverse each pixel of each image frame, and match it with the Gaussian distribution of the background model to obtain the difference between each pixel and the corresponding Gaussian distribution. Mark the pixels with a difference value exceeding the preset variance as blackboard tool points, and mark the pixels with a difference value not exceeding the preset variance as blackboard background points, and obtain the blackboard tool images of the series of image frames.

[0090] S322: Binarize the image of the user writing on the whiteboard, traverse all pixels, and obtain the position coordinates of the whiteboard writing tool.

[0091] Binarization methods can refer to existing technologies.

[0092] S33: For each frame in the series of image frames corresponding to the video stream, detect the student's eye region, and estimate the coordinates of the center point of the pupil and the corneal reflection point based on the gradient map of the student's eye region.

[0093] S331: Detect the student's eye region in each frame of a series of image frames using a pre-trained eye detection model.

[0094] S332: Use the median difference method to obtain the gradient map of the student's eye region for each frame of the image.

[0095] Specifically, a blank image of the same size as the student's eye region is created as a gradient map to store the calculated gradient values. Each pixel in the image of the student's eye region is iterated through, and the gradient values ​​in the horizontal and vertical directions are calculated for each pixel, expressed as:

[0096]

[0097] Where Gray is the gray value of a pixel in the image, and (x,y) are the pixel coordinates of the image.

[0098] The calculated gradient values ​​in the horizontal and vertical directions are stored at the corresponding pixel positions in the gradient map. The expression for the gradient map is: G(x,y)=dx(x,y)+dy(x,y).

[0099] S333: Estimating the coordinates of the center point of the pupil and the corneal reflection point of the student's eye based on gradient graphs.

[0100] Step S333 specifically includes the following steps:

[0101] S333-1: Traverse each pixel of the gradient map of the student's eye region and calculate the position vector of the current pixel relative to all remaining pixels;

[0102]

[0103] in, Let x be the position vector of the current pixel and pixel j. i ,y i (x) represents the coordinates of the current pixel. j ,y j () represents the coordinates of another pixel.

[0104] S333-2: Perform a dot product between the gradient value and the position vector of the current pixel, and accumulate all the dot product operations of the current pixel to obtain the sum of the dot products of the current pixel;

[0105]

[0106] Among them, Y i This is the sum of the dot products of the current pixel i; Let i be the gradient value of the current pixel. Let i be the position vector between the current pixel i and the nth pixel.

[0107] S333-3: Compare the sum of the dot products of all pixels. The pixel with the largest sum of dot products is estimated to be the center point of the student's pupil. Obtain the coordinates of the center point of the student's pupil.

[0108] S333-4: Preset segmentation threshold, segment corneal reflective points and background based on the segmentation threshold, pixels exceeding the segmentation threshold are corneal reflective points, and pixels not exceeding the segmentation threshold are background, and obtain the position coordinates of corneal reflective points.

[0109] It is worth noting that the position coordinates of the corneal reflection point, the position coordinates of the center point of the student's pupil, and the position coordinates of the writing implement in this application are all pixel coordinates. In some other embodiments, multiple cameras can be used to capture videos of the writing implement and the student's eyes. In this case, it is necessary to perform extrinsic parameter calibration on multiple cameras to transform the position coordinates of the corneal reflection point, the position coordinates of the center point of the student's pupil, and the position coordinates of the writing implement to the same coordinate system.

[0110] S4: Estimate the direction of the student's visual axis based on the coordinates of the pupil center point and the corneal reflection point, and connect the coordinates of the pupil center point and the blackboard writing tool to predict the direction of the student's visual direction.

[0111] The method of predicting the direction of a student's gaze by connecting the center point of the pupil with the coordinates of the blackboard writing tool can refer to existing technologies.

[0112] The direction of the student's visual axis is estimated based on the coordinates of the pupil center point and the corneal reflection point.

[0113] Assuming the cornea is spherical, the incident angle θ1 and refraction angle θ2 of the light on the cornea are calculated based on the coordinates of the center of the pupil and the reflective point of the cornea. The direction of the optical axis of the student's line of sight is estimated by the intersection of the pupil center and the refraction angle.

[0114] θ1 = arccos(cos(θ1));

[0115]

[0116] in, Let P be the vector from the center of the student's pupil to the corneal reflective point, where P and R represent the positions of the student's pupil center and corneal reflective point, respectively. The normal to the cornea;

[0117] θ2 = arcsin(K·sin(θ1));

[0118]

[0119] Here, K is a parameter used to simplify calculations and has no practical meaning.

[0120] S5: The degree of overlap between the predicted student gaze direction and the estimated student gaze optical axis direction, as well as the correlation between the two text information, are used as scoring indicators to score the student's classroom performance.

[0121] In this embodiment, the method for determining the relevance between the first text information and the second text information for each student is as follows:

[0122] The two text messages are segmented into words, irrelevant words are removed, and a pre-trained bag-of-words model is used to convert the segmented text messages into numerical feature vectors. The cosine of the angle between the two numerical feature vectors is calculated, and the correlation between the two text messages is obtained based on the cosine of the angle.

[0123]

[0124] Where θ is the angle between the two numerical eigenvectors. and These are the magnitudes of the two numerical eigenvectors. Cosθ is the dot product of two numerical feature vectors, where cosθ ∈ [0, 1], and the closer it is to 1, the higher the correlation.

[0125] More specifically, an initial score for classroom performance is preset for each student, and thresholds are set for overlap and relevance. If the overlap exceeds the threshold, it is used as the first bonus item, and a specified score is added; otherwise, it is used as the first deduction item, and a specified score is subtracted. If the relevance exceeds the threshold, it is used as the second bonus item, and a specified score is added; otherwise, it is used as the second deduction item, and a specified score is subtracted. This yields the final classroom performance score for each student.

[0126] Optionally, an initial value for each student's classroom performance is preset. The correlation between two text messages at the same time is compared based on timestamps. The student's performance in classroom language is judged based on the correlation data between the two text messages. If the correlation is higher than a preset threshold, the classroom performance is considered good, and the initial value of classroom performance is incremented by one. If the correlation is lower than the preset threshold, the classroom performance is considered poor, and the initial value of classroom performance is decremented by one. The predicted student's gaze direction and the estimated gaze axis direction are compared based on timestamps to judge the student's performance in classroom gaze. If the predicted student's gaze direction and the estimated gaze axis direction coincide, the classroom performance is considered good, and the initial value of classroom performance is incremented by one. If the predicted student's gaze direction and the estimated gaze axis direction do not coincide, the classroom performance is considered poor, and the initial value of classroom performance is decremented by one.

[0127] In some other embodiments, the classroom performance is divided into three segments based on the difference between the maximum and minimum values ​​of the final classroom performance: the first segment represents good classroom performance, the middle segment represents good classroom performance, and the last segment represents poor classroom performance.

[0128] In some other embodiments, the initial score for classroom performance can be the classroom performance value of the previous assessment; when the positive difference between the classroom performance value of the same student and the previous classroom performance value exceeds a preset threshold, the grade advances by one level.

[0129] Example 2

[0130] like Figure 2 As shown, a student classroom performance evaluation system based on multimodal audio and video is provided, including:

[0131] The data acquisition module is used to acquire the audio and video streams from the classroom.

[0132] The audio stream processing module is used to preprocess the audio stream, separate the independent audio stream of each student, and convert each independent audio stream into the corresponding first text information;

[0133] The video stream processing module is used to identify the blackboard content and blackboard tools for each frame in the series of image frames corresponding to the video stream, and to obtain the position coordinates of the blackboard tools and the second text information of the blackboard content. At the same time, it also detects the student's eye area for each frame, and estimates the position coordinates of the pupil center point and corneal reflection point based on the gradient map of the student's eye area. Based on the position coordinates of the pupil center point and corneal reflection point, it estimates the optical axis direction of the student's gaze, and connects the position coordinates of the pupil center point and the blackboard tools to predict the direction of the student's gaze.

[0134] The scoring module is used to score students' classroom performance by using the degree of overlap between the predicted student's gaze direction and the estimated student's gaze axis direction, as well as the correlation between the two text information.

[0135] In some other embodiments, such as Figure 3 As shown, the data acquisition module includes an audio and video acquisition submodule and an audio and video separation module. For specific acquisition and separation methods, please refer to Example 1.

[0136] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section. Those skilled in the art will clearly understand that the technologies in the embodiments of this invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of this invention, in essence or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or certain parts of the embodiments of this invention.

[0138] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for evaluating student classroom performance based on multimodal audio and video, characterized in that, include: Obtain the audio and video streams from the classroom; The audio stream is preprocessed to separate the independent audio stream for each student, and each independent audio stream is converted into the corresponding first text information; For each frame of the series of image frames corresponding to the video stream, the blackboard content and blackboard tools are identified, and the position coordinates of the blackboard tools and the second text information of the blackboard content are obtained. At the same time, for each frame of the image, the student's eye area is detected, and the position coordinates of the pupil center point and corneal reflection point are estimated based on the gradient map of the student's eye area. The direction of the student's line of sight is estimated based on the position coordinates of the pupil center point and the corneal reflection point, and the position coordinates of the pupil center point and the blackboard writing tool are connected to predict the direction of the student's line of sight. The degree of overlap between the predicted student gaze direction and the estimated student gaze optical axis direction, as well as the correlation between the two textual information, are used as scoring indicators to score students' classroom performance. The process involves detecting the student's eye region and estimating the coordinates of the pupil center and corneal reflection point based on the gradient map of the student's eye region. The student's eye region was detected in each frame of a series of image frames using a pre-trained eye detection model. The gradient map of the student's eye region corresponding to each frame of the image was obtained using the median difference method. The coordinates of the center point of the student's pupil and the corneal reflection point are estimated based on the gradient map, specifically as follows: Iterate through each pixel in the gradient map of the student's eye region and calculate the position vector of the current pixel relative to all remaining pixels. ； in, For the current pixel and the pixel j The position vector, The coordinates of the current pixel. The coordinates of another pixel; Perform a dot product between the gradient value and the position vector of the current pixel, and accumulate all the dot product operations of the current pixel to obtain the sum of the dot products of the current pixel; ； in, For the current pixel The sum of the dot products; For the current pixel gradient value, For the current pixel With the n The position vector of each pixel; Compare the sum of the dot products of all pixels. The pixel with the largest sum of dot products is estimated to be the center point of the student's pupil. Obtain the coordinates of the center point of the student's pupil. A preset segmentation threshold is used to segment corneal reflective points and background. Pixels exceeding the segmentation threshold are considered corneal reflective points, while pixels below the threshold are considered background. The position coordinates of the corneal reflective points are then obtained.

2. The student classroom performance evaluation method based on multimodal audio and video according to claim 1, characterized in that, The audio and video streams of the classroom are collected synchronously and separately, or obtained by separating the classroom audio and video captured by the camera.

3. The student classroom performance evaluation method based on multimodal audio and video according to claim 1, characterized in that, The process of separating each student's independent audio stream specifically includes: The preprocessed audio stream is decomposed into multiple short frames and then windowed. Perform a Fast Fourier Transform on each windowed frame to convert it from the time domain to the frequency domain, and solve for the modulus square to obtain the power spectrum of the speech signal. The power spectrum is passed through a set of Mel-scale filters to obtain the Mel spectrum, and then the logarithm is taken to obtain the logarithmic Mel spectrum. The MFCC eigenvectors are obtained by performing a discrete cosine transform on the logarithmic Mel spectrum. The FastICA algorithm is used to process the MFCC feature vectors and separate the independent audio streams of each student.

4. The student classroom performance evaluation method based on multimodal audio and video according to claim 1, characterized in that, The process of recognizing the blackboard content and the blackboard tools, and obtaining the position coordinates of the blackboard tools and the second text information of the blackboard content, specifically involves: For each frame in the series of image frames corresponding to the video stream, a pre-trained optical character recognition model is used to recognize the whiteboard content and convert it into second text information; For each frame in the series of image frames corresponding to the video stream, the Gaussian mixture background difference method is used to obtain the image of the whiteboard tool, and the whiteboard user image is binarized. All pixels are traversed to obtain the position coordinates of the whiteboard tool.

5. The student classroom performance evaluation method based on multimodal audio and video according to claim 4, characterized in that, The method of obtaining images of blackboard writing tools using the Gaussian mixture background difference method is as follows: Several Gaussian distributions are selected to describe the gray value changes of each pixel. In the initial image frame without any writing tools, the initial mean, variance and weight of each Gaussian distribution are set to construct and initialize the background model. Read a series of image frames sequentially, iterate through each pixel of each image frame, and match it with the Gaussian distribution of the background model to obtain the difference between each pixel and the corresponding Gaussian distribution. Pixels with a difference value exceeding the preset variance are marked as blackboard tool points, and pixels with a difference value not exceeding the preset variance are marked as blackboard background points, thus obtaining the blackboard tool image of the series of image frames.

6. The student classroom performance evaluation method based on multimodal audio and video according to claim 1, characterized in that, The estimation of the student's visual axis direction based on the coordinates of the pupil center point and the corneal reflection point is as follows: Assuming the cornea is spherical, calculate the angle of incidence of light on the cornea based on the coordinates of the center of the student's pupil and the coordinates of the corneal reflective point. and angle of refraction And the direction of the optical axis of the student's line of sight is estimated by the intersection of the pupil center and the angle of refraction; ； ； in, Let be the vector from the center of the student's pupil to the corneal reflective point. and These represent the positions of the center point of the student's pupil and the corneal reflective point, respectively. The normal to the cornea; ； ； in, These are parameters used to simplify calculations and have no practical significance.

7. The student classroom performance evaluation method based on multimodal audio and video according to claim 1, characterized in that, The method for determining the correlation between the two text messages is as follows: The two text messages are segmented into words, irrelevant words are removed, and a pre-trained bag-of-words model is used to convert the segmented text messages into numerical feature vectors. Calculate the cosine of the angle between two numerical feature vectors, and obtain the correlation between the two text information based on the cosine of the angle. ； in, The angle between two numerical eigenvectors. and These are the magnitudes of the two numerical eigenvectors. The dot product of two numerical eigenvectors. Furthermore, the closer the value is to 1, the higher the correlation.

8. The student classroom performance evaluation method based on multimodal audio and video according to claim 1, characterized in that, The method of using the degree of overlap between the predicted student gaze direction and the estimated student gaze optical axis direction, as well as the correlation between the two textual pieces of information, as scoring indicators to evaluate students' classroom performance is as follows: An initial score for classroom performance is preset for each student. Thresholds are set for overlap and relevance. If the overlap exceeds the threshold, it is used as the first bonus item, and a specified score is added. Otherwise, it is used as the first deduction item, and a specified score is subtracted. If the relevance exceeds the threshold, it is used as the second bonus item, and a specified score is added. Otherwise, it is used as the second deduction item, and a specified score is subtracted. This yields the final classroom performance score for each student.

9. A student classroom performance evaluation system based on multimodal audio and video, comprising the steps of the student classroom performance evaluation method based on multimodal audio and video as described in any one of claims 1-8, characterized in that, include: The data acquisition module is used to acquire the audio and video streams from the classroom. The audio stream processing module is used to preprocess the audio stream, separate the independent audio stream of each student, and convert each independent audio stream into the corresponding first text information; The video stream processing module is used to identify the blackboard content and blackboard tools for each frame in the series of image frames corresponding to the video stream, and to obtain the position coordinates of the blackboard tools and the second text information of the blackboard content. At the same time, it also detects the student's eye area for each frame, and estimates the position coordinates of the pupil center point and corneal reflection point based on the gradient map of the student's eye area. Based on the position coordinates of the pupil center point and corneal reflection point, it estimates the optical axis direction of the student's gaze, and connects the position coordinates of the pupil center point and the blackboard tools to predict the direction of the student's gaze. The scoring module is used to score students' classroom performance by using the degree of overlap between the predicted student's gaze direction and the estimated student's gaze axis direction, as well as the correlation between the two text information.

Images