Teacher courseware dependency evaluation method based on audio and video analysis

Through audio and video analysis technology, combined with DETR, Canny edge detection and optical character recognition, the teacher's courseware dependence is automatically detected, which solves the problem of inefficiency in the existing technology, and realizes accurate quantitative analysis and feedback on the teacher's courseware dependence, improving the Teaching quality.

CN119048960APending Publication Date: 2024-11-29BEIJING NORMAL UNIVERSITY
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411166279.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing technology is inefficient in teacher courseware dependency analysis, cannot be quantified, and it is difficult to accurately capture the degree of teachers' dependence on courseware, and lacks automated detection and feedback.

Method used

Using an audio and video analysis method, the projection screen is positioned through DETR object detection and Canny edge detection algorithm to extract courseware videos; combined with optical character recognition and speech recognition technology, the courseware text and teaching text are extracted, and the courseware text is compared, and the teacher calculates the courseware degree of dependence.

Benefits of technology

The analysis speed is improved, quantitative analysis of courseware dependence is realized, and quantitative percentage data is provided to help teachers improve teaching strategies and improve teaching quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048960A_ABST
    Figure CN119048960A_ABST
Patent Text Reader

Abstract

The invention discloses a teacher courseware dependency evaluation method based on audio and video analysis, and the method comprises the steps: recording a teaching audio and video in a teaching process of a teacher, and obtaining the teaching content and courseware content of the teacher; positioning a projection screen position in a classroom through DETR target detection and a Canny edge detection algorithm, extracting a projection screen picture, and obtaining a courseware video only containing a PPT projection screen; performing text recognition on the PPT key frame of the courseware video through an optical character recognition algorithm, and extracting a courseware text; identifying the teaching content of the teacher by using a voice recognition technology to obtain a teaching text; and comparing and matching the courseware text with the teaching text to obtain a result of the degree of dependence of the teacher on the courseware in the teaching process. According to the method, the analysis speed is increased, and quantitative analysis is performed on the courseware dependency in a time-sharing and segmented manner, so that the degree of dependency of teachers on the courseware is effectively evaluated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of teaching evaluation systems, and in particular relates to a teacher courseware dependency evaluation method based on audio and video analysis. Background Art

[0002] The role of teachers is crucial. They are not only the implementers of education, but also the key factors of education quality and effect. The emotional state of teachers, the seriousness of lesson preparation, and whether they read PPT directly in class will affect the teaching effect. The use of PPT is crucial in the teaching process. The reasonable use of PPT can enrich the teaching content and make the courses taught by teachers clear, vivid and easy to understand. However, over-reliance on PPT will make the courses rigid and fail to mobilize students' learning enthusiasm. In particular, if teachers rely too much on and read PPT, and fail to adjust the specific content of the explanation according to the actual teaching situation, not only will the basic teaching skills not be cultivated and improved, but also the students' learning effect will be reduced, which will eventually affect the quality of classroom teaching. In the education process, teaching evaluation, especially the evaluation of teachers' dependence on courseware, is of great significance to improving classroom quality. However, the current analysis of teachers' dependence on courseware still relies on traditional manual work, which not only consumes a lot of time and cost, but also cannot be quantified and it is difficult to accurately capture the degree of teachers' dependence on courseware, and it is inefficient and cannot provide accurate and quantitative feedback to teachers. Therefore, it is particularly necessary to fully apply artificial intelligence algorithms to the evaluation of courseware dependence. Summary of the invention

[0003] The present invention aims to solve the limitations of current classroom quality assessment in teacher courseware dependency analysis, especially the deficiencies in efficiency and quantification.

[0004] To achieve the above object, the present invention provides a teacher courseware dependency evaluation method based on audio and video analysis, comprising:

[0005] Record the teacher's audio and video during the lecture to obtain the teacher's teaching content and courseware content;

[0006] The projection screen in the classroom is located through DETR target detection and Canny edge detection algorithms, and the projection screen image is extracted to obtain a courseware video containing only the PPT projection screen;

[0007] Perform text recognition on the PPT key frames of the courseware video using an optical character recognition algorithm to extract the courseware text; use speech recognition technology to recognize the teacher's teaching content to obtain the teaching text;

[0008] The courseware text is compared and matched with the teaching text to obtain the result of the degree of teacher's reliance on the courseware during the teaching process.

[0009] Preferably, the process of locating the projection screen position in the classroom by using DETR target detection and Canny edge detection algorithm and extracting the projection screen image includes:

[0010] The Canny edge detection algorithm is used to detect the edges of the recorded video images, and the projection screen position in the video image is located by taking advantage of the fact that the edges form a closed area with a large area and high brightness;

[0011] Input the recorded video image into the DETR model to analyze the video image and output the rectangular bounding box corresponding to the projection screen;

[0012] The video image is cropped according to the position information of the projection screen to obtain a courseware video containing only the PPT projection screen.

[0013] Preferably, the process of performing edge detection on the recorded video image using the Canny edge detection algorithm and locating the position of the projection screen in the video image includes:

[0014] The video frames are sampled at 5-second intervals. For each frame, the Canny edge detection algorithm is used to detect the image edge and obtain the potential contour.

[0015] The Sobel operator is used to calculate the gradient magnitude and direction of the image, and non-maximum suppression is performed to retain the points with the local maximum gradient magnitude, suppress other non-edge points, and refine the edges; at the same time, double threshold processing is used to detect and connect the edges to obtain all the edges in the video frame;

[0016] Delete the edges that do not form a closed area, and then use the characteristics of the closed area being large and bright to filter the projection screen area, and use the closed area with the largest value as the projection screen area of ​​the video frame;

[0017] The final projection screen contour is obtained by calculating the intersection-and-union ratio of the bounding boxes in all detected contours, and the original video image is cropped according to the final projection screen contour to obtain a courseware video containing only the PPT projection screen.

[0018] Preferably, the Sobel operator is used to calculate the gradient magnitude and direction of the image, rather than maximum suppression, to retain the point with the local maximum gradient magnitude and suppress other non-edge points. The process of edge thinning includes:

[0019] Combine the approximate horizontal and vertical gradients of each pixel in the image to calculate the gradient magnitude; check the pixels in the neighborhood along the gradient direction, and set the gradient magnitude of the current pixel to 0 if it is not the largest;

[0020] The formula expression of gradient amplitude is:

[0021]

[0022] The formula expression for the gradient direction is:

[0023]

[0024] in:

[0025]

[0026]

[0027] In the formula, A represents the original image, G x and G y Represent the images after horizontal and vertical edge detection respectively.

[0028] Preferably, the process of detecting and connecting edges using dual threshold processing to obtain all edges in the video frame includes:

[0029] Double threshold detection is performed on each pixel value. By setting a high threshold and a low threshold, pixels with gradient amplitude greater than the high threshold are considered strong edges, pixels between the high threshold and the low threshold are considered weak edges, and pixels below the low threshold are suppressed.

[0030] The weak edge is connected to the strong edge through edge connection to form a continuous edge. If the weak edge pixel is connected to the strong edge pixel, the weak edge pixel is considered to be an edge, otherwise it is suppressed.

[0031] Preferably, the process of inputting the recorded video image into the DETR model to analyze the video image and outputting a rectangular bounding box corresponding to the projection screen includes:

[0032] Randomly sample several frames from the input video image, use the DETR model to perform target detection on each sampled video frame, and obtain the corresponding bounding box;

[0033] Compare the bounding boxes detected in different sampled video frames in pairs, calculate the intersection-over-union ratio of different bounding boxes, classify the bounding boxes with an overlap greater than a preset threshold into one category, and obtain a classification set;

[0034] Select the set with the most bounding boxes and the number of bounding boxes greater than n, return the average value of the bounding boxes as the final detection result, crop the original video image according to the bounding boxes, and obtain the courseware video containing only the PPT projection screen;

[0035] If the number of bounding boxes is less than n, the entire process is retried until the preset maximum number of retries is reached, and the rectangular bounding box corresponding to the projection screen is output; wherein n is a positive integer customized according to requirements, and the output format is the coordinates of the four vertices of the rectangular bounding box.

[0036] Preferably, the process of performing text recognition on the PPT key frames of the courseware video by an optical character recognition algorithm and extracting the courseware text includes:

[0037] Use OCR technology to identify the text content on each page of the PPT key frame and extract the courseware text;

[0038] Using OCR technology to identify the text content on each page of the PPT key frame, the process of extracting the courseware text includes:

[0039] The key frame image H of PPT is H={h0,h1,…,h i}The input is put into the Paddle OCR model, first through the text detection module, after scaling and normalization, the convolutional neural network ResNet is used to extract the features of the input information, the convolutional layer is used to predict the text area, and the post-processing non-maximum suppression is performed to locate the text box; then it passes through the text recognition module, uses the CRNN architecture, and combines CTC for decoding to obtain the final character sequence; finally, the recognition result passes through the post-processing module, the recognition result is further optimized and formatted, and the recognition result is corrected in combination with the language model, and finally the recognized PPT text content is recorded as in, Represents the text content on the PPT key frame image on page i;

[0040] The process of using speech recognition technology to identify the teacher's teaching content and obtain the teaching text includes:

[0041] The audio signal of the teaching audio is converted into WAV format and normalized, and then the audio signal is Fourier transformed to obtain a spectrum diagram; the noise component in the spectrum diagram is identified, and the noise frequency band is removed by a high-frequency filter;

[0042] Use Whisper to perform speech recognition on the preprocessed audio data, convert the input audio data into the corresponding text sequence, and obtain the classroom teaching text sequence At the same time, the timestamp of each sentence is recorded; among them, the teaching text sentence represents the text content corresponding to the jth sentence, and the start and end times are and

[0043] Preferably, the process of comparing and matching the courseware text with the teaching text to obtain the result of the teacher's reliance on PPT during the teaching process includes:

[0044] Perform PPT page turning detection on the courseware video, cut the teaching text according to the PPT page turning time, and obtain the corresponding teaching text content of each PPT page;

[0045] Compare the PPT courseware text and the corresponding teaching text to obtain the number of repeated characters. If the repeated characters are greater than 75% of the total sentence length, it is considered that the sentence is reading the PPT;

[0046] The percentage of verbatim content and total content in each page of PPT was counted to obtain the degree of teacher's reliance on the courseware during the teaching process.

[0047] Preferably, the process of performing PPT page turning detection on the courseware video includes:

[0048] Based on the video screen of the courseware, the page turning of the PPT is recognized, the key frame of each page of the PPT is obtained, and the key frame image of each page of the PPT is saved;

[0049] Calculate the absolute difference of the pixels in the two key frame images, and compare the absolute difference with the dynamic similarity threshold to determine whether the key frame images are the same, identify the page turning of the PPT, and record the page turning time point, and save it in the local file together with the PPT key frame image.

[0050] Preferably, the process of calculating the absolute difference of pixels in two key frame images, and comparing the absolute difference with a dynamic similarity threshold, comprises:

[0051] Use the OpenCV library to read the video file, obtain the frame rate of the courseware video, and sample the video frames at 3-second intervals;

[0052] For the current frame f obtained by sampling n , if the previous frame f n-1 Does not exist, that is, the current frame is the first frame f n =f0, and sample the next image; if there is a previous frame image f n-1 , then the current frame f n and the previous frame f n-1 Convert to grayscale image for comparison, the comparison formula is:

[0053]

[0054] Among them, A ij f n The gray value of the i-th row and j-th column, B ij f n-1 Gray value of the i-th row and j-th column;

[0055] Then, the relationship between S and the dynamic similarity threshold T is calculated to determine whether the page is turned; if S>T, it is considered that the PPT has turned the page, and the current frame is saved and recorded as h i , and the timestamp t at this time n; If S<T, it is considered that the PPT does not turn the page, and finally the key frame set H={h0,h1,...,h i}, where h i Represents the key frame image of the PPT page i, and records the time when the PPT page is turned And the start and end time and duration of each page of PPT;

[0056] The formula expression of the dynamic similarity threshold T is:

[0057] T=W*H*C

[0058] Among them, W is the width of the image, H is the height, and C is the number of channels.

[0059] Compared with the prior art, the present invention has the following advantages and technical effects:

[0060] The present invention not only improves the analysis speed, but also performs quantitative analysis on the courseware dependency in different time periods and segments, thereby effectively evaluating the degree of teacher dependency on the courseware. By reasonably analyzing the teacher's dependency on PPT during teaching and providing feedback, it is helpful to improve the teacher's professional quality.

[0061] The present invention can automatically detect the content of the courseware, identify the page turning time of the PPT, and accurately transcribe the teacher's teaching voice into text, and then analyze the degree of overlap between the teacher's teaching content and the courseware content. Compared with the manual method, this method not only reduces the workload of manpower, but also can provide quantitative percentage data to show the degree of dependence, thereby providing teachers with practical feedback and improvement suggestions to optimize teaching strategies and improve teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0063] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0064] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0065] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0066] like Figure 1 As shown, this embodiment provides a teacher courseware dependency evaluation method based on audio and video analysis, including:

[0067] Record the teacher's audio and video during the lecture to obtain the teacher's teaching content and courseware content;

[0068] The projection screen in the classroom is located through DETR target detection and Canny edge detection algorithms, and the projection screen image is extracted to obtain a courseware video containing only the PPT projection screen;

[0069] Use optical character recognition algorithm to perform text recognition on the PPT key frames of the courseware video and extract the courseware text; use speech recognition technology to recognize the teacher's teaching content and obtain the teaching text;

[0070] Compare and match the courseware text with the lecture text to obtain the results of the teacher's reliance on the courseware during the lecture.

[0071] Furthermore, the specific steps are as follows:

[0072] Step 1: Data collection, recording the lecture audio and video to obtain the teacher’s lecture content and courseware content.

[0073] Through audio and video processing of teachers' classroom content, in-depth analysis of teachers' dependence on courseware can be achieved.

[0074] The purpose of video recording is to obtain courseware information. There are two main solutions: 1. Considering that classrooms are generally equipped with cameras facing the podium, video images including the projection screen can be obtained. You can give priority to using courseware videos obtained by existing equipment to reduce duplication of investment; 2. If the classroom does not have this condition, you can record the courseware video yourself, or directly upload the courseware.

[0075] Specifically, the classroom video recording of this embodiment uses Hikvision's camera to connect to the computer to record the video screen containing PPT in real time. The video resolution should be high enough to ensure that the text and images of the PPT projection content are clear and discernible. The frame width of this implementation case is 1920, the frame height is 1080, and the frame rate is 30 frames per second. The camera should have automatic focus and exposure adjustment functions to adapt to changes in light in the classroom and ensure data quality. The video data is transmitted to the connected computer in real time and stored on the local server. The video data of each class is marked according to the date and time, corresponding to the teacher's teaching audio, for subsequent classification and retrieval.

[0076] It should be noted that if the existing cameras in the classroom meet the above requirements to obtain images containing courseware and have sound pickup function, there is no need to add additional equipment. If the existing equipment does not meet the requirements, it is necessary to install sound pickup and cameras according to the above requirements, or the teacher can record the courseware video by himself, or directly upload the courseware.

[0077] There are also two options for recording the audio part: if the original camera in the classroom can achieve good sound reception and can fully record the audio during the classroom teaching process, the original equipment can be used for recording; if the existing equipment cannot record the audio clearly, a microphone must be deployed near the podium to record the audio information of the teacher's lecture.

[0078] Specifically, the audio recording of the lectures in this embodiment uses Hikvision's microphone to connect to a computer to collect audio data of the teacher's lectures in the classroom. The collected data covers the actual lecture audio of all classes in the classroom. The teacher turns on the microphone every time he teaches and turns off the microphone after class. Therefore, the collected audio data only contains classroom teaching content, avoiding the interference of noise between classes.

[0079] The microphone is installed in the classroom where the teacher's voice can be clearly captured, that is, above the center of the podium, to ensure high-quality audio information collection. The collected audio data is transmitted to the connected computer in real time and stored on the local server to ensure data security and accessibility. The audio files are saved in MP3 format and tagged according to the date and time of each class to facilitate subsequent classification and retrieval.

[0080] Step 2: Projection screen positioning and courseware video capture.

[0081] The video images recorded by existing classroom cameras contain a lot of irrelevant information. In order to better identify the content of the courseware, it is necessary to further use target detection technology and edge detection technology to extract the projection screen image, that is, the courseware video image, to filter out these irrelevant information. In this step, the DETR model and Canny edge detection algorithm are used in this embodiment. First, the recorded video image is input into the DETR model, and the rectangular bounding box corresponding to the projection screen is output by analyzing the video image. The DETR algorithm has good effect and high accuracy, but the projection screen may not be recognized. When this happens, the Canny algorithm is used for edge detection, and the projection screen position is located by using the characteristics of the edge forming a closed area and the large area and high brightness of the closed area. Finally, the video is cropped according to the obtained screen position information to obtain a courseware video containing only the PPT projection screen.

[0082] Considering the differences in classroom structures and camera placement, the recorded video may contain other information besides the projection screen, such as the teacher's teaching activities, desks, and blackboards. In order to better extract the PPT content and remove irrelevant information interference, this embodiment provides a solution for locating the projection screen from the video. In step 1, if the video data is a courseware video recorded by the teacher, that is, the video screen only contains the PPT, this step can be skipped.

[0083] Algorithm 1: DETR-based target detection algorithm

[0084] The DETR model is a target detection model based on the Transformer architecture that can identify and locate multiple different categories of target objects in an image. Unlike traditional target detection methods, such as region-based methods and the RCNN series, DETR does not require the use of manually designed candidate regions (Region Proposals) or anchor boxes (Anchor Boxes), but directly converts the target detection task into an end-to-end Transformer network.

[0085] The image is input into the DETR model, and the image is analyzed to output the bounding box corresponding to the projection screen, where the output format is the coordinates of the four vertices of the rectangular bounding box. However, due to the influence of the algorithm accuracy, the detection results of different video frames are different, and false detection or errors may occur. In order to achieve accurate positioning of the projection screen, this embodiment obtains the position of the projection screen by fusing the detection results of multiple frames. Specifically, multiple frames, such as 100 frames, are randomly sampled from the input video to ensure that various changes in the video can be covered. The DETR model is used for target detection on each sampled frame to obtain the corresponding bounding box. The bounding boxes detected by different video frames are compared pairwise, and the intersection over union (IoU) of different bounding boxes is calculated. The bounding boxes with high overlap, such as greater than 90%, are classified into one category to form several sets, each of which contains bounding boxes with high repetition. Finally, the set containing the most bounding boxes and the number of bounding boxes greater than n (for example, 2) is selected, and the average value of the bounding boxes is returned as the final detection result, and the original video image is cropped according to the bounding box to obtain a video containing only the projection screen. If the number of bounding boxes is less than n, the entire process is retried until the maximum number of retries is reached, such as 10 times.

[0086] Algorithm 2: Edge detection algorithm based on edge canny

[0087] Considering that although DERT detection has a high recognition accuracy, it may fail to extract the projection screen. Therefore, in this case, the present embodiment also combines the Canny algorithm to locate the projection screen. Specifically, the video frames are sampled at 5-second intervals. For each frame of the image, the Canny edge detection algorithm is used to detect the image edge in order to find potential contours. The Canny algorithm uses the Sobel operator to calculate the gradient amplitude and direction of the image, and performs non-maximum suppression to refine the edges. At the same time, double threshold processing is used to detect and connect edges. The algorithm for calculating the image amplitude and direction by the Sobel operator is as follows: If A represents the original image, G x and G y Respectively represent the images after horizontal and vertical edge detection, and the formulas are as follows:

[0088]

[0089]

[0090] The approximate horizontal and vertical gradients of each pixel of the image can be combined using the following formula to calculate the magnitude of the gradient.

[0091]

[0092] The gradient direction can then be calculated using the following formula:

[0093]

[0094] After calculating the gradient value and gradient direction, non-maximum suppression is used to retain the points with the local maximum gradient amplitude and suppress other non-edge points. The specific method is to check the pixels in the neighborhood along the gradient direction. If the gradient amplitude of the current pixel is not the maximum, it is set to 0. After the above non-maximum suppression operation, double threshold detection is performed on each pixel value, a high threshold and a low threshold are set, and the following operations are performed: pixels with gradient amplitudes greater than the high threshold are considered strong edges, pixels between the high threshold and the low threshold are considered weak edges, and pixels below the low threshold are suppressed. Finally, the weak edge is connected to the strong edge through edge connection to form a continuous edge. If a weak edge pixel is connected to a strong edge pixel, it is considered to be an edge, otherwise it is suppressed.

[0095] All edges in the video frame can be obtained through Canny edge detection. First, delete the edges that do not form a closed area. Then, use the characteristics of the closed area being large and bright to filter the projection screen area. The specific screening formula is: perimeter*0.4+area*0.4+brightness*0.2, and the closed area with the largest value is used as the projection screen area of ​​the video frame. In order to ensure the correctness of the algorithm, it is also necessary to fuse the detection results of multiple frames. Similar to the previous step, the final projection screen contour is filtered out by calculating the intersection over union (IoU) of the bounding box among all detected contours. Finally, the original video image is cropped according to the contour to obtain a courseware video containing only the projection screen.

[0096] This embodiment proposes a projection screen detection algorithm that combines the Transformer-based DETR model with the Canny edge detection model to locate the projection screen position in real time and obtain the courseware video screen without complex preprocessing or manual setting of anchor points.

[0097] Step 3: PPT page turning detection.

[0098] Based on the courseware video screen obtained in the previous step, the page turning of the PPT is identified, and the key frame of each page of the PPT is obtained, that is, one frame of image is saved for each page of the PPT. Dividing the PPT by page to obtain key frames can delete a large number of repeated redundant video frames and simplify subsequent processing. In addition, it is also convenient to count the teacher's dependence on each page of the PPT to show the dynamic situation of the classroom. Specifically, if the pixel difference between the two frames before and after the courseware video is large enough, it is considered that a page turn has occurred. Based on the above idea, by calculating the absolute difference between the pixels of the two images and comparing it with the dynamic similarity threshold T, the page turning of the PPT is identified, and the time point is recorded and saved in a local file together with the PPT key frame.

[0099] Specifically, this step calculates the pixel difference between the previous and next two frames of images and compares it with a threshold to determine whether the previous and next frames are the same, thereby obtaining the page turning status of the PPT.

[0100] First, use the OpenCV library to read the video file, obtain the frame rate of the courseware video, and sample the video frames at 3-second intervals. n , if the previous frame f n-1 Does not exist, that is, the current frame is the first frame f n =f0, and sample the next image; if there is a previous frame image f n-1 , then the current frame f n and the previous frame f n-1 Convert to grayscale image for comparison, the formula is as follows:

[0101]

[0102] Among them, A ij f n The gray value of the i-th row and j-th column, B ij f n-1 Then, by calculating the relationship between S and the dynamic similarity threshold T, it is determined whether to turn the page. Specifically, the dynamic similarity threshold T is:

[0103] T=W*H*C

[0104] Where W is the width of the image, H is the height, and C is the number of channels. If S>T, it is considered that the PPT page has been turned, and the current frame is saved and recorded as h i , and the timestamp t at this time n ; If S < T, it is considered that the PPT does not turn the page. Finally, the key frame set H = {h0, h1, ..., h i}, where h i Represents the key frame image of the PPT page i. It also records the time when the PPT page is turned. And the start and end time and duration of each page of PPT.

[0105] This embodiment proposes to use the difference between the previous and next video frames and the dynamic similarity threshold to detect the PPT page turning situation. This technology dynamically adjusts the similarity threshold based on the size and number of channels of the video frame, and can adapt to video data of different resolutions and qualities.

[0106] Step 4: Extract text from PPT courseware.

[0107] Furthermore, the OCR technology is used to identify the text content on each page of the PPT key frame and extract the courseware text. Specifically, the key frame image H of the PPT is converted into {h0, h1, ..., h i}The input is put into the Paddle OCR model, first through the text detection module, after scaling and normalization, the convolutional neural network ResNet extracts the input information, predicts the text area through the convolution layer, and performs post-processing non-maximum suppression to accurately locate the text box; then it passes through the text recognition module, uses the CRNN architecture, and combines CTC for decoding to obtain the final character sequence; finally, the recognition result passes through the post-processing module, the recognition result is further optimized and formatted, and the recognition result is corrected in combination with the language model (such as dictionary, n-gram model) to improve the accuracy and readability of the final output. Finally, the recognized PPT text content is recorded as in, Represents the text content on the PPT key frame image on page i.

[0108] Step 5: Convert lecture audio into lecture text.

[0109] The captured lecture audio is converted into lecture text using Whisper automatic speech recognition technology, while recording the timestamp of each sentence. Whisper in this embodiment is a Transformer-based deep learning model for automatic speech recognition. By training Whisper with a large amount of multilingual and multi-task supervised data, the model can understand and transcribe speech in multiple languages, and has strong adaptability to different accents and environmental noise.

[0110] Furthermore, the audio information of the lecture obtained in step 1 needs to be pre-processed by normalization and denoising. First, the audio signal is converted into WAV format and normalized so that the amplitude of the audio signal is within a fixed range. Then, the audio signal is Fourier transformed to obtain a spectrum diagram, identify the noise components in the spectrum, and remove the noise frequency band through a high-frequency filter.

[0111] Use Whisper to perform speech recognition on the preprocessed audio data. Whisper uses the encoder-decoder Transformer model and uses a large amount of multi-language and multi-task supervision data for training to convert the input audio into the corresponding text sequence. It has strong robustness and accuracy in Chinese speech recognition. Finally, the classroom teaching text sequence is obtained. Among them, the teaching text sentence represents the text content corresponding to the jth sentence, and the start and end times are and

[0112] Step 6: Compare the teaching text with the PPT text.

[0113] According to the PPT page turning time, the teaching text is cut to obtain the corresponding teaching text content of each PPT page. Then, the PPT text and the corresponding teaching text are compared to obtain the number of repeated characters. If the repeated characters are greater than 75% of the total sentence length, it is considered that the sentence is reading the PPT. Finally, the percentage of the reading content and the total content in each PPT page is counted to obtain the degree of teacher dependence on the courseware.

[0114] Specifically, the teaching text content is automatically matched and compared with the PPT text content to obtain the overlap between the two, and the results are quantified to evaluate the teacher's reliance on each page of the courseware. First, the teaching text is segmented using the start and end time of each PPT page to obtain the teaching content corresponding to each PPT page. Specifically, the start and end time of the nth PPT is recorded as y Traverse the teaching text and find all The teaching text sentence Thus, the teaching content corresponding to the nth PPT is formed. Considering the situation that the teacher switches PPT while speaking, that is, for In the special case of , the jth sentence is added to the teaching text corresponding to the n-1th PPT and the nth PPT at the same time. Then, the text matching algorithm is used to calculate the overlap between the PPT text content and the corresponding teaching text content. Specifically, for each teaching text sentence Calculate the number of overlapping words with the corresponding PPT text content. If the number of repeated words is greater than If the number of words in a sentence is 75% of the total number of words, it is considered that the sentence is reading the PPT verbatim. Finally, the ratio of the length of repeated text to the total length of the teaching text is calculated to obtain the teacher's reliance on the nth PPT. For visual display, the reliance on each page of PPT in the whole class is given in the form of a line graph with time as the horizontal axis.

[0115] This embodiment combines audio and video processing technology with text analysis methods to analyze in real time the teacher's reliance on courseware during teaching, provide feedback to the teacher, and assist the teacher in improving the quality of teaching.

[0116] This embodiment proposes a teacher courseware dependency assessment method based on audio and video analysis, which quantifies the teacher's reliance on PPT by obtaining audio and video information during the teacher's teaching process. The method specifically locates the projection screen position in the classroom through DETR target detection and Canny edge detection algorithms, and uses optical character recognition algorithms to identify courseware content. At the same time, it uses speech recognition technology to obtain the teacher's teaching content. By comparing and matching the courseware content and the teaching content, the teacher's reliance on PPT during the teaching process is obtained, so as to help teachers improve their professional quality and teaching level, promote teachers' continuous improvement, and ultimately improve the quality of classroom teaching.

[0117] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A teacher courseware dependency evaluation method based on audio and video analysis, characterized in that: include: Record the teacher's audio and video during the lecture to obtain the teacher's teaching content and courseware content; The projection screen in the classroom is located through DETR target detection and Canny edge detection algorithms, and the projection screen image is extracted to obtain a courseware video containing only the PPT projection screen; Perform text recognition on the PPT key frames of the courseware video using an optical character recognition algorithm to extract the courseware text; use speech recognition technology to recognize the teacher's teaching content to obtain the teaching text; The courseware text is compared and matched with the teaching text to obtain the result of the degree of teacher's reliance on the courseware during the teaching process.

2. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 1 is characterized in that: The process of locating the projection screen in the classroom through DETR target detection and Canny edge detection algorithm and extracting the projection screen image includes: The Canny edge detection algorithm is used to detect the edges of the recorded video images, and the projection screen position in the video image is located by taking advantage of the fact that the edges form a closed area with a large area and high brightness; Input the recorded video image into the DETR model to analyze the video image and output the rectangular bounding box corresponding to the projection screen; The video image is cropped according to the position information of the projection screen to obtain a courseware video containing only the PPT projection screen.

3. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 2 is characterized in that: The process of using the Canny edge detection algorithm to detect the edge of the recorded video image and locate the projection screen position in the video image includes: The video frames are sampled at 5-second intervals. For each frame, the Canny edge detection algorithm is used to detect the image edge and obtain the potential contour. The Sobel operator is used to calculate the gradient magnitude and direction of the image, and non-maximum suppression is performed to retain the points with the local maximum gradient magnitude, suppress other non-edge points, and refine the edges; at the same time, double threshold processing is used to detect and connect the edges to obtain all the edges in the video frame; Delete the edges that do not form a closed area, and then use the characteristics of the closed area being large and bright to filter the projection screen area, and use the closed area with the largest value as the projection screen area of ​​the video frame; The final projection screen contour is obtained by calculating the intersection-and-union ratio of the bounding boxes in all detected contours, and the original video image is cropped according to the final projection screen contour to obtain a courseware video containing only the PPT projection screen.

4. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 3 is characterized in that: The Sobel operator is used to calculate the gradient magnitude and direction of the image. It is not a maximum value suppression. It retains the point with the local maximum gradient magnitude and suppresses other non-edge points. The process of edge refinement includes: Combine the approximate horizontal and vertical gradients of each pixel in the image to calculate the gradient magnitude; check the pixels in the neighborhood along the gradient direction, and set the gradient magnitude of the current pixel to 0 if it is not the largest; The formula expression of gradient amplitude is: The formula expression for the gradient direction is: in: In the formula, A represents the original image, G x and G y Represent the images after horizontal and vertical edge detection respectively.

5. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 3 is characterized in that: The process of detecting and connecting edges using dual threshold processing to obtain all edges in the video frame includes: Double threshold detection is performed on each pixel value. By setting a high threshold and a low threshold, pixels with gradient amplitude greater than the high threshold are considered strong edges, pixels between the high threshold and the low threshold are considered weak edges, and pixels below the low threshold are suppressed. The weak edge is connected to the strong edge through edge connection to form a continuous edge. If the weak edge pixel is connected to the strong edge pixel, the weak edge pixel is considered to be an edge, otherwise it is suppressed.

6. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 2 is characterized in that: The process of inputting the recorded video image into the DETR model to analyze the video image and outputting the rectangular bounding box corresponding to the projection screen includes: Randomly sample several frames from the input video image, use the DETR model to perform target detection on each sampled video frame, and obtain the corresponding bounding box; Compare the bounding boxes detected in different sampled video frames in pairs, calculate the intersection-over-union ratio of different bounding boxes, classify the bounding boxes with an overlap greater than a preset threshold into one category, and obtain a classification set; Select the set with the most bounding boxes and the number of bounding boxes greater than n, return the average value of the bounding boxes as the final detection result, crop the original video image according to the bounding boxes, and obtain the courseware video containing only the PPT projection screen; If the number of bounding boxes is less than n, the entire process is retried until the preset maximum number of retries is reached, and the rectangular bounding box corresponding to the projection screen is output; wherein n is a positive integer customized according to requirements, and the output format is the coordinates of the four vertices of the rectangular bounding box.

7. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 1 is characterized in that: The process of performing text recognition on the PPT key frames of the courseware video by using an optical character recognition algorithm and extracting the courseware text includes: Use OCR technology to identify the text content on each page of the PPT key frame and extract the courseware text; Using OCR technology to identify the text content on each page of the PPT key frame, the process of extracting the courseware text includes: The key frame image H of PPT is H={h0,h1,…,h i }The input is put into the Paddle OCR model, first through the text detection module, after scaling and normalization, the convolutional neural network ResNet is used to extract the features of the input information, the convolutional layer is used to predict the text area, and the post-processing non-maximum suppression is performed to locate the text box; then it passes through the text recognition module, uses the CRNN architecture, and combines CTC for decoding to obtain the final character sequence; finally, the recognition result passes through the post-processing module, the recognition result is further optimized and formatted, and the recognition result is corrected in combination with the language model, and finally the recognized PPT text content is recorded as in, Represents the text content on the PPT key frame image on page i; The process of using speech recognition technology to identify the teacher's teaching content and obtain the teaching text includes: The audio signal of the teaching audio is converted into WAV format and normalized, and then the audio signal is Fourier transformed to obtain a spectrum diagram; the noise component in the spectrum diagram is identified, and the noise frequency band is removed by a high-frequency filter; Use Whisper to perform speech recognition on the preprocessed audio data, convert the input audio data into the corresponding text sequence, and obtain the classroom teaching text sequence At the same time, the timestamp of each sentence is recorded; among them, the teaching text sentence represents the text content corresponding to the jth sentence, and the start and end times are and 8. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 1 is characterized in that: The process of comparing and matching the courseware text with the teaching text to obtain the result of the teacher's reliance on PPT during the teaching process includes: Perform PPT page turning detection on the courseware video, cut the teaching text according to the PPT page turning time, and obtain the corresponding teaching text content of each PPT page; Compare the PPT courseware text and the corresponding teaching text to obtain the number of repeated characters. If the repeated characters are greater than 75% of the total sentence length, it is considered that the sentence is reading the PPT; The percentage of verbatim content and total content in each page of PPT was counted to obtain the degree of teacher's reliance on the courseware during the teaching process.

9. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 8 is characterized in that: The process of performing PPT page turning detection on the courseware video includes: Based on the video screen of the courseware, the page turning of the PPT is recognized, the key frame of each page of the PPT is obtained, and the key frame image of each page of the PPT is saved; Calculate the absolute difference of the pixels in the two key frame images, and compare the absolute difference with the dynamic similarity threshold to determine whether the key frame images are the same, identify the page turning of the PPT, and record the page turning time point, and save it in the local file together with the PPT key frame image.

10. The teacher courseware dependency evaluation method based on audio and video analysis according to claim 9 is characterized in that: The process of calculating the absolute difference of pixels in the two key frame images and comparing the absolute difference with the dynamic similarity threshold includes: Use the OpenCV library to read the video file, obtain the frame rate of the courseware video, and sample the video frames at 3-second intervals; For the current frame f obtained by sampling n , if the previous frame f n-1 Does not exist, that is, the current frame is the first frame f n =f0, and sample the next image; if there is a previous frame image f n-1 , then the current frame f n and the previous frame f n-1 Convert to grayscale image for comparison, the comparison formula is: Among them, A ij f n The gray value of the i-th row and j-th column, B ij f n-1 Gray value of the i-th row and j-th column; Then, by calculating the relationship between S and the dynamic similarity threshold T, it is determined whether to turn the page; if S > T, it is considered that the PPT has been turned, and the current frame is saved and denoted as h i , and the timestamp t at this time n ; if S < T, it is considered that the PPT has not been turned. Finally, the key frame set H = {h0, h1,..., h i}, where h i represents the key frame image of the i-th page of the PPT, and at the same time records the time when the PPT is turned and the start and end times and duration of each page of the PPT; The formula expression of the dynamic similarity threshold T is: T=W*H*C Among them, W is the width of the image, H is the height, and C is the number of channels.

Citation Information

Cited By

  • Teaching management method and system for intelligent teaching platform

    CN120410808A

  • A teaching management method and system for a smart teaching platform

    CN120410808B