AI multi-dimensional teaching behavior analysis method and system based on classroom video
Through the AI multi-dimensional teaching behavior analysis method, the problems of insufficient integrity and correlation of teaching behavior analysis in the existing technology, video quality and noise are solved, high-quality teaching behavior analysis and feedback are achieved, and teaching quality and evaluation accuracy are improved. .
Patent Information
- Application Number
- CN202411134389.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The existing technology lacks the integrity and relevance of teaching behavior in classroom teaching video analysis, and faces problems such as low video quality, high audio noise, and complex and changeable teacher behavior, which affects the accuracy and reliability of the analysis.
Using AI multi-dimensional teaching behavior analysis method, through the acquisition of adaptive segmentation of classroom video frames and audio frames, image super-resolution reconstruction of deep convolutional neural networks, adaptive noise reduction and spectral subtraction enhancement, combined with object detection, pose estimation, Deep embedded clustering and heterogeneous graph neural networks, construct teaching behavior analysis graphs, and generate timing semantic representations and optimal teaching modes.
The video and audio quality is improved, noise interference is eliminated, the teacher's posture and sight movement information is obtained, comprehensive teaching feedback and guidance is generated, and teaching quality and evaluation accuracy is improved.
Smart Images

Figure CN118658128B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an AI multi-dimensional teaching behavior analysis method and system based on classroom videos. Background Art
[0002] With the rapid development of educational informatization, a large amount of video data is generated in the classroom teaching process. The video data contains rich teaching behavior information, such as teachers' voice, body movements, and gaze. Traditional teaching evaluation methods mainly rely on manual observation and subjective judgment, which have problems such as low efficiency and strong subjectivity.
[0003] In the existing technology, there have been some attempts to use computer vision and machine learning methods to analyze teaching behavior, but most of them analyze teaching behavior in a single dimension and lack consideration of the integrity and relevance of teaching behavior. In addition, when processing teaching videos, existing methods face challenges such as low video quality, high audio noise, and complex and changeable teacher behavior, which affect the accuracy and reliability of the analysis;
[0004] Therefore, a solution is urgently needed to solve the problems existing in the prior art. Summary of the invention
[0005] The embodiments of the present invention provide an AI multi-dimensional teaching behavior analysis method and system based on classroom videos, which can at least solve some of the problems existing in the prior art.
[0006] A first aspect of an embodiment of the present invention provides an AI multi-dimensional teaching behavior analysis method based on classroom videos, comprising:
[0007] Acquire a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video;
[0008] The standard classroom teaching video is added to a pre-set target detection model and a single target tracking model respectively. The target detection model performs target recognition through an attention mechanism and a feature pyramid structure to obtain a target bounding box. The single target tracking model tracks the target teacher's position in each frame to obtain a spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through a posture estimation network in the target bounding box, a three-dimensional skeleton model is constructed and posture feature information is obtained. The audio track of the standard classroom teaching video is subjected to speaker separation through a deep embedded clustering algorithm, separated speech segments are generated, and the speech feature vector is obtained through learning through a gated recurrent unit network. The line of sight movement of the target teacher in the standard classroom teaching is modeled through a three-dimensional convolutional neural network to obtain gaze point distribution information.
[0009] Based on a heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the speech feature vector and the gaze point distribution information are used as nodes, and the correlation between different nodes is used as an edge to construct a teaching behavior analysis graph. At each time step, the nodes are iteratively updated in time by a gated recurrent unit. The attention weight of each node and the intrinsic correlation between nodes are determined by the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a temporal semantic representation. A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation, and the optimal teaching model is generated with the standard specification of the teaching behavior as a condition. The optimal teaching model is added to the teaching behavior analysis graph to compare with the current teaching behavior, and is micro-deconstructed in combination with the graph reasoning algorithm to obtain a teaching behavior analysis report.
[0010] In an optional embodiment,
[0011] Obtain a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames respectively into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them in combination with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video, including:
[0012] Acquire a classroom teaching video and respectively determine the start frame and stop frame of the video and audio in the classroom teaching video to obtain the video frame and the audio frame; for the video frame, determine the voice activity probability in the current video frame through a long short-term memory neural network, determine the voice activity frame in combination with a preset voice activity threshold, perform anchor point detection based on the voice activity frame in combination with a sliding window, generate multiple anchor points, and divide the video frame into multiple video segments according to the anchor points;
[0013] For the audio frame, the logarithmic Mel-frequency spectrum corresponding to the current audio frame is added to the improved convolutional neural network, and feature extraction is performed through the convolution layer and the pooling layer to obtain the corresponding emotion intensity value and generate an emotion intensity sequence, and the intensity change of adjacent emotion intensity values in the emotion intensity sequence is calculated by the first-order difference algorithm, and the intensity change is compared with a preset mutation threshold. If the intensity change is greater than the mutation threshold, it is marked as an emotion mutation point, and the calculation and screening are repeated to obtain an emotion mutation sequence. Based on the time interval of adjacent emotion mutation points in the emotion mutation sequence, the audio frame is divided into multiple audio segments;
[0014] For each frame in the video clip, feature extraction and nonlinear mapping are performed through a super-resolution convolutional neural network model, and image reconstruction is performed according to the mapping result to obtain a high-resolution image. For the audio clip, the audio signal is decomposed into multiple sub-bands through wavelet packet decomposition, and the Gaussian noise standard deviation is calculated on each sub-band and a denoising threshold is determined, soft threshold denoising is performed based on the denoising threshold, and the denoised sub-bands are reconstructed by wavelet packets to obtain an initial denoised frequency spectrum, and the initial denoised frequency spectrum is short-time Fourier transformed to generate an initial spectrogram and the pure spectrum of the teacher's voice is calculated based on the spectral subtraction formula of the prior signal-to-noise ratio, and the pure spectrum is inversely short-time Fourier transformed to obtain an enhanced voice signal;
[0015] For each video clip and audio clip, the time offset is determined and blank frames are added for synchronization alignment, and the aligned video clips and audio clips are alternately spliced according to the original time sequence to obtain the standard classroom teaching video.
[0016] In an optional embodiment,
[0017] The pure spectrum of the teacher's speech is calculated based on the spectral subtraction formula of the prior signal-to-noise ratio as shown in the following formula:
[0018] ;
[0019] in, represents the pure spectrum of the teacher's speech, oh is the frequency index, indicating the frequency component currently being analyzed. t is the time index, indicating the time frame of the current analysis, represents the power spectrum of the noisy speech spectrum, represents the corrected prior signal-to-noise ratio, Represents the power spectrum of the noise spectrum estimate.
[0020] In an optional embodiment,
[0021] The standard classroom teaching video is added to the pre-set target detection model and single target tracking model respectively. The target detection model performs target recognition through the attention mechanism and feature pyramid structure to obtain the target bounding box. The single target tracking model tracks the target teacher's position in each frame to obtain the spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through the posture estimation network in the target bounding box, and a three-dimensional skeleton model is constructed to obtain posture feature information. The audio track of the standard classroom teaching video is separated by the deep embedded clustering algorithm, and the separated speech segments are generated and learned through the gated recurrent unit network to obtain the speech feature vector. The line of sight movement of the target teacher in the standard classroom teaching is modeled by the three-dimensional convolutional neural network, and the gaze point distribution information is obtained, including:
[0022] The standard classroom teaching video is added as input to a pre-set target detection model and a single target tracking model. The target detection model extracts a multi-scale feature map corresponding to each video frame in the standard classroom teaching video through a backbone network, and adjusts the weights corresponding to different regions in combination with a spatial attention module to obtain a spatial attention weight map and multiply the multi-scale feature map element by element to obtain an attention-enhanced feature representation and upsample the outputs of different stages of the backbone network from top to bottom based on a feature pyramid structure to obtain a fused multi-scale feature map. For the fused multi-scale feature map, a region candidate network is used to generate a region candidate box for the target teacher on the feature map of each scale, and the region candidate box is non-maximum suppressed and screened according to a threshold to obtain the target bounding box;
[0023] The single target tracking model adds a template candidate frame in the first frame of the standard classroom teaching video according to the target image, uses the image in the template candidate frame as a tracking template, adds the target image and the tracking template to two branches based on the twin network setting and performs feature extraction, generates a tracking feature map and performs a cross-correlation operation, calculates a similarity score map, performs a soft normalization operation on the similarity score map, determines the center coordinates of the target image, and repeats the operation in each frame to obtain the spatiotemporal behavior trajectory;
[0024] In the target bounding box, feature extraction is performed through a posture estimation network to generate a posture feature map and set three parallel branches, namely a heat map branch, a horizontal coordinate branch and a vertical coordinate branch. The heat map branch predicts the heat map of each key point through a one-dimensional convolution, and the horizontal coordinate branch and the vertical coordinate branch predict the horizontal coordinate and the vertical coordinate of each key point. Based on the prediction results obtained by the three parallel branches, the pixel coordinates of each key point are obtained, the pixel coordinates are estimated by a soft threshold random sampling consistency algorithm, and the depth information of the current key point is restored by the triangulation principle. The three-dimensional key point coordinates corresponding to each key point are obtained based on the depth information, and the three-dimensional key point coordinates are connected to obtain the three-dimensional skeleton model and the corresponding posture feature information;
[0025] For the audio track of the standard classroom teaching video, a short-time Fourier transform is performed through a preset frame shift to generate an audio track spectrogram, the audio track spectrogram is added to a deep embedding network, the speaker embedding vector of each frame is extracted, the speaker embedding vector is clustered, the number of speakers is estimated and assigned to different categories, the speaker embedding vector is masked and de-embedded according to the clustering result, and the separated speech segment corresponding to each speaker is obtained, the separated speech segment is learned through a gated recurrent unit, the separated speech features and the corresponding weight matrix are determined, and the speech feature vector is obtained in combination with an attention weighting mechanism;
[0026] Based on the standard classroom teaching video, a facial image sequence of the target teacher is obtained and added to the three-dimensional convolutional neural network. Feature extraction is performed according to the preset time convolution kernel and space convolution kernel, and a line of sight feature vector is output. The line of sight feature vector is predicted through a fully connected layer and a soft normalization function to obtain the gaze point coordinates of the target teacher in each frame. A gaze point coordinate sequence is generated based on the gaze point coordinates, and the gaze point distribution information is obtained in combination with statistical operations.
[0027] In an optional embodiment,
[0028] The similarity score graph calculated in the single-target model is shown in the following formula:
[0029] ;
[0030] in, S * ( m , n ) indicates the similarity score graph in ( m , n ) position, i represents the feature channel index, D represents the number of feature channels, u and vrepresents the coordinate index, Y i ( u , v ) indicates the i The expected output of a channel at position (u, v), Indicates the tracking feature map at position ( u , v ), Indicates the tracking feature map at position ( u , v ), l 1 represents the L2 regularization parameter, l 2 represents the regularization parameter of the spatial penalty term, W i ( u , v ) indicates that the spatial penalty weight is at position ( u , v ), Indicates the tracking feature map at position ( m+u , n+v) The value of .
[0031] In an optional embodiment,
[0032] Based on the heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information are used as nodes, and the correlation between different nodes is used as the edge to construct a teaching behavior analysis graph. At each time step, the nodes are iteratively updated in time series through the gated recurrent unit. The attention weight of each node and the intrinsic correlation between nodes are determined by the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a time series semantic representation. A multi-layer conditional generative adversarial network is constructed based on the time series semantic representation, and the standard specification of teaching behavior is used as a condition to generate the optimal teaching mode. The optimal teaching mode is added to the teaching behavior analysis graph for comparison with the current teaching behavior, and the micro-deconstruction is performed in combination with the graph reasoning algorithm to obtain a teaching behavior analysis report including:
[0033] Based on a heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information are used as nodes, the nodes in each time step are combined to obtain a node subgraph, the correlation between different nodes in the same time step is used as an undirected edge, and the changes between nodes of the same type between node subgraphs at different time steps are used as directed edges to construct the teaching behavior analysis graph;
[0034] In each time step, the nodes are updated sequentially and iteratively through the gated recurrent unit, the hidden state of each node is updated and the node change rule is determined. Based on the node change rule, the attention weight of each node is determined through the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network. The hidden state of each node is weighted summed based on the attention weight to obtain the subgraph representation corresponding to each node subgraph, which is spliced according to the time sequence to obtain the temporal semantic representation;
[0035] A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation and the generator and discriminator are initialized. Random noise and standard specifications of teaching behavior are used as inputs of the generator to generate a training teaching model. The training teaching model is added as input to the discriminator and a discrimination result is generated. The training is repeated until a preset maximum number of iterations is reached to obtain a trained multi-layer conditional generative adversarial network. Noise is sampled from the current random noise distribution and added to the generator. The optimal teaching behavior is generated and added to the teaching behavior analysis graph as a new node. The optimal teaching model is obtained through forward propagation. The optimal teaching model is compared with the current teaching behavior. The relationship path between nodes is performed through a graph reasoning algorithm, and the teaching behavior analysis report is obtained based on the relationship path.
[0036] In an optional embodiment,
[0037] The nodes are updated sequentially through the gated recurrent unit, and the hidden state of each node is updated as shown in the following formula:
[0038] ;
[0039] in, h i,t Represents the current time node i The hidden state of h i,t-1 Indicates the last moment t- 1 node i The hidden state of s () represents the activation function, W z represents the update gate weight matrix, x i,t Indicates at the current moment t node i The input features of Indicates the current time t Candidate hidden states of .
[0040] A second aspect of an embodiment of the present invention provides an AI multi-dimensional teaching behavior analysis system based on classroom videos, comprising:
[0041] The first unit is used to obtain a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them in combination with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video;
[0042] The second unit is used to add the standard classroom teaching video to a preset target detection model and a single target tracking model respectively. The target detection model performs target recognition through an attention mechanism and a feature pyramid structure to obtain a target bounding box. The single target tracking model tracks the target teacher's position in each frame to obtain a spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through a posture estimation network in the target bounding box, a three-dimensional skeleton model is constructed and posture feature information is obtained. The audio track of the standard classroom teaching video is subjected to speaker separation through a deep embedded clustering algorithm, separated speech segments are generated, and learning is performed through a gated recurrent unit network to obtain a speech feature vector. The line of sight movement of the target teacher in the standard classroom teaching is modeled through a three-dimensional convolutional neural network to obtain gaze point distribution information.
[0043] The third unit is used to construct a teaching behavior analysis graph based on a heterogeneous graph neural network, with the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information as nodes, and the correlation between different nodes as edges. At each time step, the nodes are iteratively updated in time by a gated recurrent unit, and the attention weight of each node and the intrinsic correlation between nodes are determined by the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a temporal semantic representation. A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation, and an optimal teaching model is generated with the standard specification of teaching behavior as a condition. The optimal teaching model is added to the teaching behavior analysis graph for comparison with the current teaching behavior, and micro-deconstruction is performed in combination with the graph reasoning algorithm to obtain a teaching behavior analysis report.
[0044] According to a third aspect of the embodiments of the present invention,
[0045] An electronic device is provided, comprising:
[0046] processor;
[0047] a memory for storing processor-executable instructions;
[0048] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.
[0049] A fourth aspect of the embodiments of the present invention is:
[0050] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.
[0051] In the present invention, by adaptively segmenting the classroom teaching video, and performing super-resolution reconstruction and adaptive noise reduction enhancement on the video clips and audio clips respectively, a standardized classroom teaching video can be obtained, the quality of the video and audio can be improved, noise interference can be eliminated, and a clearer and more accurate data basis can be provided for subsequent analysis. By using a posture estimation network in a target bounding box to extract the three-dimensional key point coordinates of the target teacher, and constructing a three-dimensional skeleton model, the posture feature information of the teacher can be obtained, which can reflect the non-verbal behaviors such as the teacher's body language and gestures, and provide important visual clues for analyzing teaching behaviors. By using a three-dimensional convolutional neural network to model the sight movement of the target teacher in standard classroom teaching, the gaze point distribution information can be obtained, and important visual attention information is provided for analyzing teaching behaviors. By constructing a multi-layer conditional generative adversarial network and taking the standard specifications of teaching behaviors as conditions, an optimal teaching mode can be generated. The idea of adversarial training can be used to learn and generate an optimal teaching mode that meets the standard specifications from a large amount of teaching behavior data, and provide reference and guidance for teachers. In summary, the present invention can provide teachers with comprehensive teaching feedback and guidance, promote the improvement of teaching quality, and provide data support for teaching evaluation and management. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 Schematic diagram of the process of an AI multi-dimensional teaching behavior analysis method based on classroom videos according to an embodiment of the present invention;
[0053] Figure 2 It is a structural diagram of an AI multi-dimensional teaching behavior analysis system based on classroom videos according to an embodiment of the present invention. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0056] Figure 1 FIG. 1 is a flow chart of an AI multi-dimensional teaching behavior analysis method based on classroom video according to an embodiment of the present invention. Figure 1 As shown, the method includes:
[0057] S1. Obtain a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them in combination with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video;
[0058] The deep convolutional neural network is a type of artificial neural network used to process image data, including multiple convolutional layers, pooling layers and fully connected layers. The image super-resolution reconstruction is a technology that aims to restore a high-resolution image from a low-resolution image. The spectral subtraction is a speech signal denoising method that is mainly used to reduce noise in speech signals. The adaptive denoising is a method for dynamically adjusting noise elimination strategies, which automatically adjusts denoising parameters according to signal characteristics and noise environment.
[0059] In an optional embodiment,
[0060] Obtain a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames respectively into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them in combination with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video, including:
[0061] Acquire a classroom teaching video and respectively determine the start frame and stop frame of the video and audio in the classroom teaching video to obtain the video frame and the audio frame; for the video frame, determine the voice activity probability in the current video frame through a long short-term memory neural network, determine the voice activity frame in combination with a preset voice activity threshold, perform anchor point detection based on the voice activity frame in combination with a sliding window, generate multiple anchor points, and divide the video frame into multiple video segments according to the anchor points;
[0062] For the audio frame, the logarithmic Mel-frequency spectrum corresponding to the current audio frame is added to the improved convolutional neural network, and feature extraction is performed through the convolution layer and the pooling layer to obtain the corresponding emotion intensity value and generate an emotion intensity sequence, and the intensity change of adjacent emotion intensity values in the emotion intensity sequence is calculated by the first-order difference algorithm, and the intensity change is compared with a preset mutation threshold. If the intensity change is greater than the mutation threshold, it is marked as an emotion mutation point, and the calculation and screening are repeated to obtain an emotion mutation sequence. Based on the time interval of adjacent emotion mutation points in the emotion mutation sequence, the audio frame is divided into multiple audio segments;
[0063] For each frame in the video clip, feature extraction and nonlinear mapping are performed through a super-resolution convolutional neural network model, and image reconstruction is performed according to the mapping result to obtain a high-resolution image. For the audio clip, the audio signal is decomposed into multiple sub-bands through wavelet packet decomposition, and the Gaussian noise standard deviation is calculated on each sub-band and a denoising threshold is determined, soft threshold denoising is performed based on the denoising threshold, and the denoised sub-bands are reconstructed by wavelet packets to obtain an initial denoised frequency spectrum, and the initial denoised frequency spectrum is short-time Fourier transformed to generate an initial spectrogram and the pure spectrum of the teacher's voice is calculated based on the spectral subtraction formula of the prior signal-to-noise ratio, and the pure spectrum is inversely short-time Fourier transformed to obtain an enhanced voice signal;
[0064] For each video clip and audio clip, the time offset is determined and blank frames are added for synchronization alignment, and the aligned video clips and audio clips are alternately spliced according to the original time sequence to obtain the standard classroom teaching video.
[0065] The speech activity probability refers to the probability that a signal belongs to speech within a given time period. The speech activity threshold is a critical value used to determine whether a signal contains speech. The speech activity frame refers to a time period into which an audio signal is segmented. During this time period, the signal is marked as containing speech or not containing speech. The anchor point detection is usually used to identify key points or feature points in signal processing. These points play an important role in the change or processing of the signal. The logarithmic Mel spectrogram is a graphical way to represent the spectrum of an audio signal. The emotion intensity value refers to the intensity of the emotion expressed in speech or text, which can be obtained by analyzing the sound of the speech. The method comprises the following steps: the first-order difference algorithm is used to calculate the change amount of time series data, and the difference between the current value and the previous value is calculated to detect the change trend of the data. The mutation threshold is used to detect the mutation point or change point in the signal. The emotional mutation point refers to the time point when the emotional state in the speech signal changes significantly. The Gaussian noise standard deviation represents the intensity or amplitude of the Gaussian noise. The pure spectrum refers to the spectrum without noise interference, which represents the ideal spectrum of the signal. The inverse short-time Fourier transform is the process of converting the frequency domain data after the short-time Fourier transform back to the time domain signal.
[0066] Obtain classroom teaching videos, and determine the start frame and stop frame of the video and audio to obtain video frames and audio frames. For the obtained video frames, determine the probability of speech activity in the current video frame through a long short-term memory neural network, and input the video frames into a pre-trained long short-term memory neural network model. The model can learn the temporal features in the video frames and predict the probability of speech activity. Combined with a pre-set speech activity threshold, the video frames with a speech activity probability greater than the threshold are marked as speech activity frames. Based on the marked speech activity frames, anchor point detection is performed in combination with a sliding window to generate multiple anchor points. A sliding window of a fixed size is set to slide on the speech activity frame. In each window, the anchor point is determined by a specific anchor point detection algorithm (such as edge detection, key frame extraction, etc.), and the video frame is divided into multiple video segments according to the detected anchor points.
[0067] For the obtained audio frame, convert the audio frame into a logarithmic Mel spectrum map as the input of the improved convolutional neural network, extract features of the spectrum map through the convolution layer and the pooling layer to obtain a feature vector representing the emotion intensity, input the feature vector into the fully connected layer and the activation function to obtain the corresponding emotion intensity value, repeat the calculation to generate a complete emotion intensity sequence, perform a first-order difference operation on the emotion intensity sequence to obtain the intensity change between adjacent emotion intensity values, compare the calculated intensity change with a preset mutation threshold, if the intensity change is greater than the mutation threshold, mark the corresponding position as an emotion mutation point, repeat the screening to obtain the emotion mutation sequence, calculate the time interval between adjacent emotion mutation points in the emotion mutation sequence, divide the audio frame into multiple audio segments according to the time interval, and each segment corresponds to a time period of emotional stability;
[0068] Each frame in the video clip is input into the pre-trained super-resolution convolutional neural network model, the input image is feature extracted through the convolution layer, and the features are mapped to the high-resolution space through nonlinear mapping, and the image is reconstructed according to the mapping result to obtain the corresponding high-resolution image, the audio clip is decomposed into multiple frequency subbands by using the wavelet packet decomposition algorithm, the standard deviation of Gaussian noise is estimated on each subband, the corresponding denoising threshold is determined according to the estimated standard deviation, the soft threshold function is applied to each subband, the coefficients below the threshold are set to zero, the denoised subband is reconstructed by wavelet packets, and the initial denoised frequency is synthesized, the initial denoised frequency is subjected to short-time Fourier transform for time-frequency analysis, and the initial spectrogram is obtained to reflect the time-frequency characteristics of the audio, the noise spectrum is estimated by using the prior signal-to-noise ratio, and the estimated noise spectrum is subtracted from the initial spectrogram by the spectral subtraction formula to obtain the pure spectrum of the teacher's voice, the pure spectrum is inversely transformed by short-time Fourier inverse transform, and it is converted from the frequency domain back to the time domain to obtain the denoised and enhanced teacher's voice signal;
[0069] The time offset between the video clip and the corresponding audio clip is calculated. According to the offset, blank frames are added at the beginning or end position of the video clip or audio clip to align them in time. The aligned video clips and audio clips are spliced alternately in the original time sequence. The splicing order of the video clips and audio clips is consistent with the order in the original classroom teaching video. After the splicing is completed, a standard classroom teaching video with synchronized alignment and improved audio and video quality is obtained.
[0070] For example, suppose a 45-minute classroom teaching video is obtained, the video resolution is 1920x1080, the frame rate is 30 frames / second, the audio sampling rate is 44.1kHz, the depth is 16 bits, the video start frame is 0, the video stop frame is 81000 (45 minutes × 60 seconds × 30 frames / second), the audio start frame is 0, the audio stop frame is 396900 (45 minutes × 60 seconds × 44100 samples / second), the voice activity probability of each frame is calculated by the long short-term memory neural network, the voice activity threshold is set to 0.6, and the voice activity frame is anchored. , using a sliding window of 5 seconds and a step size of 1 second, we get the anchor points: [0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, ...], and divide the video frames into segments according to the anchor points: [0-150], [151-300], [301-450], [451-600]…, convert the audio frames into logarithmic Mel spectrograms and add them to the improved convolutional neural network. We extract features through convolutional and pooling layers and get the emotion intensity sequence: [0.2, 0.3, 0.4, 0.6, 0.8, 0.5 , 0.9, ...], the first-order difference algorithm is used to calculate the change in adjacent emotion intensity values, and the mutation threshold is set to 0.3, and the emotion mutation point moments are obtained: [3, 7, 15, 20, 28, 35, 40, ...], the audio frames are divided into segments according to the emotion mutation points: [0-132300], [132301-308700], [308701-661500], ..., the super-resolution convolutional neural network model is applied to the frames in each video segment, and the 1920x1080 frames are reconstructed into 3840x through feature extraction and nonlinear mapping. A high-resolution image of 2160 is obtained, and a 6-level wavelet packet decomposition is performed on each audio clip, and the standard deviation of Gaussian noise of each subband is calculated. Soft threshold denoising is performed according to the denoising threshold, and wavelet packet reconstruction is performed to obtain an initial denoised frequency. The initial denoised frequency is short-time Fourier transformed, and the prior signal-to-noise ratio and pure spectrum are calculated. The pure spectrum is inversely short-time Fourier transformed to obtain an enhanced speech signal, and the time offset of the video clip and the audio clip is calculated. Blank frames are added for alignment, and the aligned video clips and audio clips are alternately spliced in the original time order to obtain the standard classroom teaching video.
[0071] In this embodiment, by predicting the probability of speech activity for video frames and combining it with an anchor point detection algorithm, the video can be accurately divided into speech activity segments and non-speech activity segments, which is helpful for subsequent analysis and processing, such as focusing on the teacher's speech segments, or separating the student's question and answer segments. The logarithmic Mel spectrum of the audio frame is subjected to feature extraction and emotion intensity prediction to obtain a continuous emotion intensity sequence. The mutation point of emotion intensity can be detected through first-order difference and threshold comparison, and the audio can be divided into emotionally stable segments, which is helpful for analyzing the teacher's emotional changes during the teaching process, as well as the correlation with the teaching content and the student's reaction. Feature extraction and non-speech activity prediction are performed on each frame in the video segment. Linear mapping can reconstruct the original low-resolution video frames into high-resolution images, which can significantly improve the visual quality of the video, make the teacher's facial expressions, gestures, blackboard content and other details clearer, and provide a better visual basis for subsequent analysis. Through wavelet packet decomposition and soft threshold denoising, the background noise in the audio clip can be effectively suppressed, which is helpful for automatic speech recognition and speech content analysis, as well as improving students' auditory experience. In summary, this embodiment provides an important data basis for intelligent teaching analysis and evaluation, helps teachers optimize teaching strategies, provides personalized teaching feedback, and provides educational researchers with richer and more accurate analysis perspectives, ultimately promoting the improvement of teaching quality and the intelligent development of education.
[0072] In an optional embodiment,
[0073] The pure spectrum of the teacher's speech is calculated based on the spectral subtraction formula of the prior signal-to-noise ratio as shown in the following formula:
[0074] ;
[0075] in, represents the pure spectrum of the teacher's speech, oh is the frequency index, indicating the frequency component currently being analyzed. t is the time index, indicating the time frame of the current analysis, represents the power spectrum of the noisy speech spectrum, represents the corrected prior signal-to-noise ratio, Represents the power spectrum of the noise spectrum estimate.
[0076] In this embodiment, by introducing a corrected prior signal-to-noise ratio, the intensity of noise suppression can be adaptively adjusted, which helps to improve the clarity and signal-to-noise ratio of the teacher's voice. By introducing the prior signal-to-noise ratio as a weighting factor, the prior signal-to-noise ratio is smaller on the frequency component dominated by speech, so that less subtraction is performed on the noisy speech spectrum, which retains the details of the speech and helps to maintain the naturalness and intelligibility of the teacher's voice. By updating the noise spectrum estimation in real time, the current noise characteristics can be more accurately reflected, thereby achieving more effective noise suppression. In summary, this embodiment can retain the detailed information of the speech while effectively suppressing background noise, improve the quality and clarity of the speech, help improve students' auditory experience, improve the accuracy of intelligent teaching analysis, and promote the development of intelligent education.
[0077] S2. The standard classroom teaching video is added to the pre-set target detection model and single target tracking model respectively. The target detection model performs target recognition through the attention mechanism and feature pyramid structure to obtain the target bounding box. The single target tracking model tracks the target teacher by obtaining the position of the target teacher in each frame to obtain the spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through the posture estimation network in the target bounding box, and a three-dimensional skeleton model is constructed to obtain posture feature information. The audio track of the standard classroom teaching video is separated by the deep embedded clustering algorithm, and separated speech segments are generated and learned through the gated recurrent unit network to obtain the speech feature vector. The line of sight movement of the target teacher in the standard classroom teaching is modeled by the three-dimensional convolutional neural network to obtain the gaze point distribution information;
[0078] The single target tracking model is used to track the motion trajectory of a single target in a video sequence. The feature pyramid structure is an architecture for multi-scale feature extraction, which is usually used in target detection and image segmentation tasks. The spatiotemporal behavior trajectory describes the motion path of an object in space and time, combines time series and spatial information, and is used to analyze and predict the motion behavior of an object. The posture estimation network is used to estimate the posture or key points of a person's body in an image. The three-dimensional key point coordinates refer to the positions of key points in three-dimensional space. The three-dimensional skeleton model is a three-dimensional model that represents the skeleton structure of an object or a human body, and describes the posture and motion of an object through a set of skeleton bones and joints. The deep embedded clustering algorithm combines deep learning and traditional clustering techniques, and aims to improve the clustering effect by learning the deep representation of data. The three-dimensional convolutional neural network is a structure of an extended convolutional neural network, which is suitable for processing three-dimensional data (such as video or volume data). The gaze point distribution information describes the spatial distribution of the gaze points of the human eye when observing a scene.
[0079] In an optional embodiment,
[0080] The standard classroom teaching video is added to the pre-set target detection model and single target tracking model respectively. The target detection model performs target recognition through the attention mechanism and feature pyramid structure to obtain the target bounding box. The single target tracking model tracks the target teacher's position in each frame to obtain the spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through the posture estimation network in the target bounding box, and a three-dimensional skeleton model is constructed to obtain posture feature information. The audio track of the standard classroom teaching video is separated by the deep embedded clustering algorithm, and the separated speech segments are generated and learned through the gated recurrent unit network to obtain the speech feature vector. The line of sight movement of the target teacher in the standard classroom teaching is modeled by the three-dimensional convolutional neural network, and the gaze point distribution information is obtained, including:
[0081] The standard classroom teaching video is added as input to a pre-set target detection model and a single target tracking model. The target detection model extracts a multi-scale feature map corresponding to each video frame in the standard classroom teaching video through a backbone network, and adjusts the weights corresponding to different regions in combination with a spatial attention module to obtain a spatial attention weight map and multiply the multi-scale feature map element by element to obtain an attention-enhanced feature representation and upsample the outputs of different stages of the backbone network from top to bottom based on a feature pyramid structure to obtain a fused multi-scale feature map. For the fused multi-scale feature map, a region candidate network is used to generate a region candidate box for the target teacher on the feature map of each scale, and the region candidate box is non-maximum suppressed and screened according to a threshold to obtain the target bounding box;
[0082] The single target tracking model adds a template candidate frame in the first frame of the standard classroom teaching video according to the target image, uses the image in the template candidate frame as a tracking template, adds the target image and the tracking template to two branches based on the twin network setting and performs feature extraction, generates a tracking feature map and performs a cross-correlation operation, calculates a similarity score map, performs a soft normalization operation on the similarity score map, determines the center coordinates of the target image, and repeats the operation in each frame to obtain the spatiotemporal behavior trajectory;
[0083] In the target bounding box, feature extraction is performed through a posture estimation network to generate a posture feature map and set three parallel branches, namely a heat map branch, a horizontal coordinate branch and a vertical coordinate branch. The heat map branch predicts the heat map of each key point through a one-dimensional convolution, and the horizontal coordinate branch and the vertical coordinate branch predict the horizontal coordinate and the vertical coordinate of each key point. Based on the prediction results obtained by the three parallel branches, the pixel coordinates of each key point are obtained, the pixel coordinates are estimated by a soft threshold random sampling consistency algorithm, and the depth information of the current key point is restored by the triangulation principle. The three-dimensional key point coordinates corresponding to each key point are obtained based on the depth information, and the three-dimensional key point coordinates are connected to obtain the three-dimensional skeleton model and the corresponding posture feature information;
[0084] For the audio track of the standard classroom teaching video, a short-time Fourier transform is performed through a preset frame shift to generate an audio track spectrogram, the audio track spectrogram is added to a deep embedding network, the speaker embedding vector of each frame is extracted, the speaker embedding vector is clustered, the number of speakers is estimated and assigned to different categories, the speaker embedding vector is masked and de-embedded according to the clustering result, and the separated speech segment corresponding to each speaker is obtained, the separated speech segment is learned through a gated recurrent unit, the separated speech features and the corresponding weight matrix are determined, and the speech feature vector is obtained in combination with an attention weighting mechanism;
[0085] Based on the standard classroom teaching video, a facial image sequence of the target teacher is obtained and added to the three-dimensional convolutional neural network. Feature extraction is performed according to the preset time convolution kernel and space convolution kernel, and a line of sight feature vector is output. The line of sight feature vector is predicted through a fully connected layer and a soft normalization function to obtain the gaze point coordinates of the target teacher in each frame. A gaze point coordinate sequence is generated based on the gaze point coordinates, and the gaze point distribution information is obtained in combination with statistical operations.
[0086] For example, suppose there is a 10-minute standard classroom teaching video with a resolution of 1920x1080 and a frame rate of 30 frames per second. The input video frame size is 1920x1080. The multi-scale feature maps extracted by the backbone network are 480x270, 240x135, and 120x68 in size. The weight map generated by the spatial attention module is the same size as the corresponding feature map. Assuming that the weight value range is [0, 1], the attention-enhanced feature representation is obtained by element-by-element multiplication. The feature pyramid structure uses 1x1 convolution and upsampling operations to fuse feature maps of different scales into a 480x270 feature map. The region proposal network generates 1000 candidate boxes on the fused feature map. The non-maximum suppression threshold is set to 0.7 and the screening threshold is set to 0.5. Finally, the bounding box of the target teacher is obtained, such as [100, 200, 300, 400], indicating that the target is located in the region from (100, 200) to (300, 400).
[0087] In the first frame, the template candidate box of the target teacher is manually annotated with a size of 200x200. Both branches of the twin network use AlexNet as the backbone network, extract 256-dimensional feature vectors, and perform cross-correlation operations on the feature vectors of the two branches to obtain a similarity score map of size 33x33. The similarity score map is converted into a probability distribution map through a soft normalization function, and the position with the maximum probability is taken as the target center coordinate. Assuming that the target center coordinate in the current frame is (500, 400), the next frame will search and match with this coordinate as the center. In a 10-minute video, there are 30 frames per second, a total of 18,000 target center coordinates, forming the spatiotemporal trajectory of the target teacher. The target bounding box size is 400x600, and the input posture estimation network HRNet-32 is used. The heat map branch outputs 17 key points. The heat map of each heat map is 100x150 in size. The horizontal and vertical coordinate branches output the horizontal and vertical coordinates of 17 key points respectively, with a range of [0, 1]. Combining the prediction results of the three branches, the pixel coordinates of each key point are obtained, such as the left shoulder (120, 80), the right elbow (200, 150), etc., and the camera internal and external parameters are estimated. Assuming that the camera focal length is 1000 and the main point coordinates are (500, 400), the depth value of each key point is calculated according to the principle of triangulation, such as the left shoulder depth is 2 meters, the right elbow depth is 1.5 meters, etc. The pixel coordinates and depth values are combined to obtain the three-dimensional coordinates of each key point, such as the left shoulder (0.2, -0.1, 2), the right elbow (0.5, 0.3, 1.5), and the three-dimensional key points are connected according to the predefined skeleton structure to construct a three-dimensional skeleton model of the target teacher;
[0088] Perform short-time Fourier transform on the 10-minute audio track with a frame length of 25 milliseconds and a frame shift of 10 milliseconds to generate a 600x257 audio track spectrogram. Use a convolutional neural network to extract the 256-dimensional speaker embedding vector of each frame, and obtain a total of 60,000 embedding vectors. Use k-means clustering to divide the embedding vectors into three categories, corresponding to three speakers (teacher, student 1, and student 2). Generate a 60,000x3 mask matrix based on the clustering results, mask and de-embed the speaker embedding vectors, and obtain separated speech segments of the three speakers, each containing 20,000 frames, each with 257 dimensions. Use a 2-layer GRU to learn the separated speech segments of each speaker, with a hidden layer dimension of 128 and an output speech feature dimension of 64. Use the attention weighted mechanism to fuse the speech features of the three speakers to obtain the final speech feature vector with a dimension of 64.
[0089] The facial area of the target teacher is detected and cropped from the video with a size of 256x256, and a total of 18,000 facial images are obtained. A three-dimensional convolutional neural network is used to extract features from the facial image sequence. The temporal convolution kernel size is 3, and the spatial convolution kernel size is 3x3. The network outputs a 512-dimensional line of sight feature vector, which is mapped to a 2-dimensional gaze point coordinate in the range of [-1, 1] through a fully connected layer. Assume that the gaze point coordinate in a certain frame is (0.2, -0.3), indicating that the gaze point is located to the upper right of the center point. In a 10-minute video, there are 30 frames per second, with a total of 18,000 gaze point coordinates, forming a gaze point coordinate sequence. The gaze point coordinate sequence is statistically analyzed, and a gaze point distribution heat map is drawn to obtain the gaze distribution information of the target teacher.
[0090] The region candidate box refers to a series of rectangular regions generated in the image that may contain targets. The target bounding box is a rectangular box used to enclose the target object in the image. The bounding box defines the position and size of the target and is usually described by four parameters. The template candidate box is a method for target detection that generates a candidate box by matching a template image with the target region. The twin network is a deep learning network structure consisting of two or more subnetworks that share the same weights. The cross-correlation operation is an operation used in signal processing to calculate the correlation between two signals or images. The similarity score map is an image generated by calculating the similarity between different regions in the image and the template. The soft normalization It is a normalization technique used to normalize data but allows some flexibility to better adapt to the characteristics of the data. The heat map branch is a branch in the deep learning network, which is used to generate a heat map to represent the predicted probability distribution. The soft threshold random sampling consensus algorithm is an improved random sampling consensus algorithm that handles outliers in the data by introducing a soft threshold. The triangulation principle is used to calculate the position of a three-dimensional point from two images from different perspectives. The depth information represents the distance from each point in the scene to the camera. The frame shift refers to the process of moving the analysis window or processing frame in the video sequence. The de-embedding operation in machine learning usually refers to the process of extracting original features from deep features.
[0091] The standard classroom teaching video is added as input to the pre-set target detection model and single target tracking model. The weights corresponding to different regions are adjusted in combination with the spatial attention module to obtain the spatial attention weight map. The spatial attention weight map is multiplied element by element with the multi-scale feature map to obtain the attention-enhanced feature representation. The outputs of different stages of the backbone network are upsampled from top to bottom based on the feature pyramid structure to obtain a fused multi-scale feature map. For the fused multi-scale feature map, the region candidate box of the target teacher is generated on the feature map of each scale through the region candidate network. The region candidate box is non-maximum suppressed and screened according to the threshold to obtain the target bounding box.
[0092] The single target tracking model adds a template candidate box to the first frame of the standard classroom teaching video according to the target image, uses the image in the template candidate box as the tracking template, adds the target image and the tracking template to the two branches based on the twin network setting, and performs feature extraction to generate a tracking feature map. The tracking feature map is cross-correlated to calculate the similarity score map, and the similarity score map is soft-normalized to determine the center coordinates of the target image. The operation is repeated in each frame to obtain the spatiotemporal behavior trajectory.
[0093] Three parallel branches are set, namely, the heat map branch, the horizontal coordinate branch and the vertical coordinate branch. The heat map branch predicts the heat map of each key point through one-dimensional convolution. The horizontal coordinate branch and the vertical coordinate branch predict the horizontal coordinate and vertical coordinate of each key point respectively. Based on the prediction results obtained by the three parallel branches, the pixel coordinates of each key point are obtained. The pixel coordinates of each key point are estimated by the soft threshold random sampling consistency algorithm. The depth information of the current key point is restored by the triangulation principle. The three-dimensional key point coordinates corresponding to each key point are obtained based on the depth information. The three-dimensional key point coordinates are connected to obtain a three-dimensional skeleton model and the corresponding posture feature information;
[0094] For the audio tracks of standard classroom teaching videos, a short-time Fourier transform is performed through a preset frame shift to generate an audio track spectrogram, which is added to the deep embedding network to extract the speaker embedding vector of each frame, cluster the speaker embedding vectors, estimate the number of speakers and assign them to different categories, mask and de-embed the speaker embedding vectors according to the clustering results, and obtain the separated speech segments corresponding to each speaker. The separated speech segments are learned through gated recurrent units to determine the separated speech features and the corresponding weight matrix, and the speech feature vector is obtained in combination with the attention weighting mechanism;
[0095] Feature extraction is performed according to the preset temporal convolution kernel and spatial convolution kernel, and the gaze feature vector is output. The gaze feature vector is predicted through a fully connected layer and a soft normalization function to obtain the gaze point coordinates of the target teacher in each frame. A gaze point coordinate sequence is generated based on the gaze point coordinates, and the gaze point distribution information is obtained by combining statistical operations.
[0096] In this embodiment, by introducing the spatial attention mechanism and the feature pyramid structure, the target detection model can effectively process feature information of different scales and regions, improve the detection accuracy, and analyze the teacher image in the target bounding box through the posture estimation network, and predict the pixel coordinates of the teacher's key points, which can provide an important data basis for analyzing the teacher's body language, teaching movements and interactive behaviors. By masking and de-embedding the speaker embedding vector, the separated voice fragments corresponding to each speaker can be obtained, which provides important voice information for analyzing the teacher's voice expression, speech rhythm and teacher-student interaction. By analyzing the teacher's facial image sequence through a three-dimensional convolutional neural network, the coordinates of the teacher's gaze point in each frame can be predicted. The line of sight attention information can reflect the teacher's focus, visual interaction mode and classroom control ability, and provide objective data support for evaluating teaching effectiveness. In summary, this embodiment provides an important data basis for intelligent education applications, promotes the objective evaluation of the teaching process, the optimization of teaching strategies and the improvement of teachers' professional quality, and also provides new perspectives and tools for educational researchers, which helps to deeply understand the inherent laws and influencing factors of the teaching process and promote the development of educational science.
[0097] In an optional embodiment,
[0098] The similarity score graph calculated in the single-target model is shown in the following formula:
[0099] ;
[0100] in, S * ( m , n ) indicates the similarity score graph in ( m , n ) position, i represents the feature channel index, D represents the number of feature channels, u and v represents the coordinate index, Y i ( u , v ) indicates the i The expected output of a channel at position (u, v), Indicates the tracking feature map at position ( u , v ), Indicates the tracking feature map at position ( u , v ), l 1 represents the L2 regularization parameter, l 2 represents the regularization parameter of the spatial penalty term, Wi ( u , v ) indicates that the spatial penalty weight is at position ( u , v ), Indicates the tracking feature map at position ( m+u , n+v) The value of .
[0101] In this embodiment, the feature matching-based method can effectively capture the appearance changes of the target and achieve stable target tracking. By multiplying the expected output with the tracking feature map element by element, the calculation of the similarity score map can be guided to make it more consistent with the prior hypothesis and improve the robustness of tracking. The introduction of the regularization term can improve the robustness and accuracy of the tracking model. By iteratively calculating the similarity score map in continuous video frames, real-time tracking of the target can be achieved and the motion trajectory of the target can be captured. In summary, this embodiment is of great significance for achieving stable, accurate and real-time single target tracking, and provides important data support for subsequent behavior analysis and interaction understanding.
[0102] S3. Based on the heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the speech feature vector and the gaze point distribution information are used as nodes, and the correlation between different nodes is used as the edge to construct a teaching behavior analysis graph. At each time step, the nodes are iteratively updated in time by the gated recurrent unit. The attention weight of each node and the intrinsic correlation between nodes are determined by the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a temporal semantic representation. A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation, and the optimal teaching model is generated with the standard specification of teaching behavior as a condition. The optimal teaching model is added to the teaching behavior analysis graph for comparison with the current teaching behavior, and micro-deconstruction is performed in combination with the graph reasoning algorithm to obtain a teaching behavior analysis report.
[0103] The teaching behavior analysis graph is a tool for analyzing and visualizing various behaviors in the teaching process. The self-attention pooling mechanism is a technology for feature extraction and aggregation, which is mainly used to process sequence data (such as text or time series). The temporal semantic representation refers to the representation method of semantic information in time series data. The conditional generative adversarial network is an extension of the generative adversarial network, which allows the generator and discriminator to be trained according to additional conditions. The graph reasoning algorithm is used to reason in the knowledge graph to obtain implicit information or relationships. The micro deconstruction refers to the detailed decomposition and analysis of the system or phenomenon, which usually involves detail analysis, local observation and process modeling.
[0104] In an optional embodiment,
[0105] Based on the heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information are used as nodes, and the correlation between different nodes is used as the edge to construct a teaching behavior analysis graph. At each time step, the nodes are iteratively updated in time series through the gated recurrent unit. The attention weight of each node and the intrinsic correlation between nodes are determined by the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a time series semantic representation. A multi-layer conditional generative adversarial network is constructed based on the time series semantic representation, and the standard specification of teaching behavior is used as a condition to generate the optimal teaching mode. The optimal teaching mode is added to the teaching behavior analysis graph for comparison with the current teaching behavior, and the micro-deconstruction is performed in combination with the graph reasoning algorithm to obtain a teaching behavior analysis report including:
[0106] Based on a heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information are used as nodes, the nodes in each time step are combined to obtain a node subgraph, the correlation between different nodes in the same time step is used as an undirected edge, and the changes between nodes of the same type between node subgraphs at different time steps are used as directed edges to construct the teaching behavior analysis graph;
[0107] In each time step, the nodes are updated sequentially and iteratively through the gated recurrent unit, the hidden state of each node is updated and the node change rule is determined. Based on the node change rule, the attention weight of each node is determined through the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network. The hidden state of each node is weighted summed based on the attention weight to obtain the subgraph representation corresponding to each node subgraph, which is spliced according to the time sequence to obtain the temporal semantic representation;
[0108] A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation and the generator and discriminator are initialized. Random noise and standard specifications of teaching behavior are used as inputs of the generator to generate a training teaching model. The training teaching model is added as input to the discriminator and a discrimination result is generated. The training is repeated until a preset maximum number of iterations is reached to obtain a trained multi-layer conditional generative adversarial network. Noise is sampled from the current random noise distribution and added to the generator. The optimal teaching behavior is generated and added to the teaching behavior analysis graph as a new node. The optimal teaching model is obtained through forward propagation. The optimal teaching model is compared with the current teaching behavior. The relationship path between nodes is performed through a graph reasoning algorithm, and the teaching behavior analysis report is obtained based on the relationship path.
[0109] The undirected edge is a type of edge in a graph, which connects two nodes and has no directionality. The directed edge is another type of edge in a graph, which connects two nodes and has directionality. For example, if there is a directed edge between node A and node B, the edge from A to B is different from the edge from B to A. The time series iterative update refers to the process of predicting or calculating future states by repeatedly updating states in time series data or dynamic systems. The subgraph representation refers to describing the structure and properties of a subgraph by extracting and representing its features. The graph reasoning algorithm is used to perform reasoning in a knowledge graph or graph data to obtain implicit information or relationships. The relationship path is a path between two nodes in a graph, which consists of a series of edges and intermediate nodes.
[0110] Taking the spatiotemporal behavior trajectory, posture feature information, speech feature vector and gaze point distribution information as nodes, the nodes in each time step are combined to obtain a node subgraph. The correlation between different nodes in the same time step is used as an undirected edge, and the changes between nodes of the same type between node subgraphs at different time steps are used as directed edges. Based on the heterogeneous graph neural network, a teaching behavior analysis graph is constructed. In each time step, the nodes are iteratively updated in time sequence through the gated recurrent unit, the hidden state of each node is updated, and the node change rule is determined.
[0111] Based on the node change rules, the attention weight of each node is determined by setting the self-attention pooling mechanism in the last layer of the heterogeneous graph neural network. The hidden state of each node is weighted summed based on the attention weight to obtain the subgraph representation corresponding to each node subgraph. The subgraph representations are spliced according to the time sequence to obtain the temporal semantic representation.
[0112] A multi-layer conditional generative adversarial network is constructed based on temporal semantic representation, the generator and the discriminator are initialized, random noise and standard specifications of teaching behavior are used as inputs of the generator, the generator generates a training teaching model, the training teaching model is added as input to the discriminator, the discriminator generates a discrimination result, and the training is repeated until the preset maximum number of iterations is reached, and a trained multi-layer conditional generative adversarial network is obtained;
[0113] Noise sampling is performed from the current random noise distribution, and the sampled noise is added to the generator. The generator generates the optimal teaching behavior, which is added to the teaching behavior analysis graph as a new node. The optimal teaching mode is obtained through forward propagation, and the optimal teaching mode is compared with the current teaching behavior. The relationship path between nodes is performed through the graph reasoning algorithm, and a teaching behavior analysis report is obtained based on the relationship path.
[0114] For example, suppose a 45-minute classroom teaching video is analyzed, the video frame rate is 30 frames per second, and there are 81,000 frames in total. Through the previous steps, the following information is obtained: spatiotemporal behavior trajectory: the center coordinate sequence of the target teacher in 81,000 frames, the dimension is 81,000x2, posture feature information: the three-dimensional coordinate sequence of the key points of the target teacher in 81,000 frames, the dimension is 81,000x17x3, speech feature vector: the speech feature sequence of the target teacher in the 45-minute audio track, the dimension is 2700x64, gaze point distribution information: the gaze point coordinate sequence of the target teacher in 81,000 frames, the dimension is 81,000x2;
[0115] The spatiotemporal behavior trajectory, posture feature information, speech feature vector and gaze point distribution information are used as the nodes of the graph. The nodes are combined every 1 second (30 frames) to obtain 2700 node subgraphs, each of which contains 4 nodes. In the same subgraph, the Pearson correlation coefficient of different nodes is calculated as the undirected edge weight between nodes. Between different subgraphs, the Euclidean distance of similar nodes is calculated as the directed edge weight to represent the temporal change of the nodes. A 2-layer GRU is used to update each node in time series. The hidden layer dimension is 128, and the total iteration is 2700 steps. It is assumed that the hidden state of the spatiotemporal behavior trajectory node is HT1 after the tth step update, and the dimension is 128. It is assumed that the hidden state of the posture feature information node is HT2 after the tth step update, and the dimension is 128. , assuming that the hidden state of the speech feature vector node is HT3 after the tth step update, the dimension is 128, assuming that the hidden state of the gaze point distribution information node is HT4 after the tth step update, the dimension is 128, and the self-attention pooling mechanism is used in the last layer of the heterogeneous graph neural network to calculate the attention weight of each node. Assuming that the attention weight of the spatiotemporal behavior trajectory node is A1, the attention weight of the posture feature information node is A2, assuming that the attention weight of the speech feature vector node is A3, and the attention weight of the gaze point distribution information node is A4, the weighted sum of the node hidden states in each subgraph is performed to obtain the subgraph representation, and the 2700 subgraph representations are spliced in chronological order to obtain the temporal semantic representation Z with a dimension of 2700x128;
[0116] Taking the temporal semantic representation Z and the random noise vector e (dimension is 100) as conditions, a multi-layer conditional generative adversarial network is trained. The generator G takes e and Z as input, passes through 3 fully connected layers (dimensions are 256, 512, 1024 respectively) and 1 deconvolution layer, and outputs the teaching mode M'. The discriminator D takes M' and Z as input, passes through 3 fully connected layers (dimensions are 1024, 512, 256 respectively) and 1 sigmoid layer, and outputs the true or false probability p. The loss function of the generator and the discriminator is calculated using the standard specification M (manually defined) of teaching behavior, and gradient descent optimization is performed. It is assumed that after 10,000 training times, the generator can generate a realistic teaching mode M', which is highly similar to the standard specification M.
[0117] 100 noise vectors are sampled from the current noise distribution and added to the trained generator G to generate 100 teaching modes. The optimal one M* is selected from the generated teaching modes and added as a new node to the teaching behavior analysis graph. The forward propagation algorithm is applied to the updated graph to obtain the temporal semantic representation Z* that integrates the optimal teaching mode M. Z* is decoded into the corresponding spatiotemporal behavior trajectory, posture features, voice features, and gaze point distribution as the optimal teaching behavior. The graph reasoning algorithm (such as random walk) is used to calculate the relationship path between the current teaching behavior and the optimal teaching behavior in the graph. A teaching behavior analysis report is generated based on the relationship path, pointing out the advantages, disadvantages, and improvement suggestions of the current teaching behavior.
[0118] In this embodiment, by taking the spatiotemporal behavior trajectory, posture feature information, speech feature vector and gaze point distribution information as nodes, and constructing edges according to the relationship between time steps and nodes, a heterogeneous graph structure is formed, which provides a basis for comprehensive analysis of teaching behavior. By using gated recurrent units to iteratively update the nodes in each time step, the temporal evolution of teaching behavior can be captured. The attention mechanism can dynamically adjust the importance of different nodes according to the hidden state and context information of the nodes, highlight the key teaching behavior features, and improve the accuracy of the analysis. The temporal semantic representation can capture the evolution law and semantic information of teaching behavior in the time dimension, and provide rich feature representation for subsequent teaching behavior analysis and pattern generation. Through adversarial training of the generator and the discriminator, the generated teaching behavior can be continuously optimized, and finally the optimal teaching behavior can be obtained, which can provide reference and guidance for teachers and help optimize teaching strategies. In summary, this embodiment can help teachers optimize teaching strategies, improve teaching quality, promote teacher professional development, and provide data support for education management and decision-making.
[0119] In an optional embodiment,
[0120] The nodes are updated sequentially through the gated recurrent unit, and the hidden state of each node is updated as shown in the following formula:
[0121] ;
[0122] in, h i,t Represents the current time node i The hidden state of h i,t-1 Indicates the last moment t- 1 node i The hidden state of s () represents the activation function, W z represents the update gate weight matrix, x i,t Indicates at the current moment t node i The input features of Indicates the current time t Candidate hidden states of .
[0123] In this embodiment, by iteratively updating the hidden state, the temporal dependency of the nodes can be established, and the dynamic changes of the nodes can be modeled. By introducing the update gate, the update of the node state can be adaptively controlled to balance the fusion of historical information and new information. When the update gate is opened, the information of the candidate hidden state will flow into the hidden state of the node. When the update gate is closed, the information of the candidate hidden state will be ignored. By retaining the historical state, the state of the node can be flexibly updated while retaining important historical information. Long-term dependencies can be transmitted and retained through multiple iterative updates, and short-term dependencies can be updated through the input features of the current moment. Complex dependencies can be established in the time dimension to capture the timing characteristics of teaching behavior. In summary, this embodiment can capture the dynamic characteristics of the teaching process and improve the accuracy and comprehensiveness of teaching behavior analysis.
[0124] Figure 2 Schematic diagram of the structure of the AI multi-dimensional teaching behavior analysis system based on classroom video according to an embodiment of the present invention. Figure 2 As shown, the system comprises:
[0125] The first unit is used to obtain a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them in combination with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video;
[0126] The second unit is used to add the standard classroom teaching video to a preset target detection model and a single target tracking model respectively. The target detection model performs target recognition through an attention mechanism and a feature pyramid structure to obtain a target bounding box. The single target tracking model tracks the target teacher's position in each frame to obtain a spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through a posture estimation network in the target bounding box, a three-dimensional skeleton model is constructed and posture feature information is obtained. The audio track of the standard classroom teaching video is subjected to speaker separation through a deep embedded clustering algorithm, separated speech segments are generated, and learning is performed through a gated recurrent unit network to obtain a speech feature vector. The line of sight movement of the target teacher in the standard classroom teaching is modeled through a three-dimensional convolutional neural network to obtain gaze point distribution information.
[0127] The third unit is used to construct a teaching behavior analysis graph based on a heterogeneous graph neural network, with the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information as nodes, and the correlation between different nodes as edges. At each time step, the nodes are iteratively updated in time by a gated recurrent unit, and the attention weight of each node and the intrinsic correlation between nodes are determined by the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a temporal semantic representation. A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation, and an optimal teaching model is generated with the standard specification of teaching behavior as a condition. The optimal teaching model is added to the teaching behavior analysis graph for comparison with the current teaching behavior, and micro-deconstruction is performed in combination with the graph reasoning algorithm to obtain a teaching behavior analysis report.
[0128] According to a third aspect of the embodiments of the present invention,
[0129] An electronic device is provided, comprising:
[0130] processor;
[0131] a memory for storing processor-executable instructions;
[0132] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.
[0133] A fourth aspect of the embodiments of the present invention is:
[0134] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.
[0135] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.
[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. The AI multi-dimensional teaching behavior analysis method based on classroom videos is characterized by: include: Acquire a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video; The standard classroom teaching video is added to a pre-set target detection model and a single target tracking model respectively. The target detection model performs target recognition through an attention mechanism and a feature pyramid structure to obtain a target bounding box. The single target tracking model tracks the target teacher's position in each frame to obtain a spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through a posture estimation network in the target bounding box, a three-dimensional skeleton model is constructed and posture feature information is obtained. The audio track of the standard classroom teaching video is subjected to speaker separation through a deep embedded clustering algorithm, separated speech segments are generated, and the speech feature vector is obtained through learning through a gated recurrent unit network. The line of sight movement of the target teacher in the standard classroom teaching is modeled through a three-dimensional convolutional neural network to obtain gaze point distribution information. Based on a heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information are used as nodes, and the correlation between different nodes is used as an edge to construct a teaching behavior analysis graph. At each time step, the nodes are iteratively updated in time by a gated recurrent unit. The attention weight of each node and the intrinsic correlation between nodes are determined by a self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a temporal semantic representation. A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation, and the standard specification of teaching behavior is used as a condition to generate an optimal teaching mode. The optimal teaching mode is added to the teaching behavior analysis graph to compare with the current teaching behavior, and a microscopic deconstruction is performed in combination with a graph reasoning algorithm to obtain a teaching behavior analysis report. Based on the heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information are used as nodes, and the correlation between different nodes is used as the edge to construct a teaching behavior analysis graph. At each time step, the nodes are iteratively updated in time series through the gated recurrent unit. The attention weight of each node and the intrinsic correlation between nodes are determined by the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a time series semantic representation. A multi-layer conditional generative adversarial network is constructed based on the time series semantic representation, and the standard specification of teaching behavior is used as a condition to generate the optimal teaching mode. The optimal teaching mode is added to the teaching behavior analysis graph for comparison with the current teaching behavior, and the micro-deconstruction is performed in combination with the graph reasoning algorithm to obtain a teaching behavior analysis report including: Based on a heterogeneous graph neural network, the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information are used as nodes, the nodes in each time step are combined to obtain a node subgraph, the correlation between different nodes in the same time step is used as an undirected edge, and the changes between nodes of the same type between node subgraphs at different time steps are used as directed edges to construct the teaching behavior analysis graph; In each time step, the nodes are updated sequentially and iteratively through the gated recurrent unit, the hidden state of each node is updated and the node change rule is determined. Based on the node change rule, the attention weight of each node is determined through the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network. The hidden state of each node is weighted summed based on the attention weight to obtain the subgraph representation corresponding to each node subgraph, which is spliced according to the time sequence to obtain the temporal semantic representation; A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation and the generator and discriminator are initialized. Random noise and standard specifications of teaching behavior are used as inputs of the generator to generate a training teaching model. The training teaching model is added as input to the discriminator and a discrimination result is generated. The training is repeated until a preset maximum number of iterations is reached to obtain a trained multi-layer conditional generative adversarial network. Noise is sampled from the current random noise distribution and added to the generator. The optimal teaching behavior is generated and added to the teaching behavior analysis graph as a new node. The optimal teaching model is obtained through forward propagation. The optimal teaching model is compared with the current teaching behavior. The relationship path between nodes is performed through a graph reasoning algorithm, and the teaching behavior analysis report is obtained based on the relationship path.
2. The method according to claim 1, characterized in that Obtain a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames respectively into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them in combination with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video, including: Acquire a classroom teaching video and respectively determine the start frame and stop frame of the video and audio in the classroom teaching video to obtain the video frame and the audio frame; for the video frame, determine the voice activity probability in the current video frame through a long short-term memory neural network, determine the voice activity frame in combination with a preset voice activity threshold, perform anchor point detection based on the voice activity frame in combination with a sliding window, generate multiple anchor points, and divide the video frame into multiple video segments according to the anchor points; For the audio frame, the logarithmic Mel-frequency spectrum corresponding to the current audio frame is added to the improved convolutional neural network, and feature extraction is performed through the convolution layer and the pooling layer to obtain the corresponding emotion intensity value and generate an emotion intensity sequence, and the intensity change of adjacent emotion intensity values in the emotion intensity sequence is calculated by the first-order difference algorithm, and the intensity change is compared with a preset mutation threshold. If the intensity change is greater than the mutation threshold, it is marked as an emotion mutation point, and the calculation and screening are repeated to obtain an emotion mutation sequence. Based on the time interval of adjacent emotion mutation points in the emotion mutation sequence, the audio frame is divided into multiple audio segments; For each frame in the video clip, feature extraction and nonlinear mapping are performed through a super-resolution convolutional neural network model, and image reconstruction is performed according to the mapping result to obtain a high-resolution image. For the audio clip, the audio signal is decomposed into multiple sub-bands through wavelet packet decomposition, and the Gaussian noise standard deviation is calculated on each sub-band and a denoising threshold is determined, soft threshold denoising is performed based on the denoising threshold, and the denoised sub-bands are reconstructed by wavelet packets to obtain an initial denoised frequency spectrum, and the initial denoised frequency spectrum is short-time Fourier transformed to generate an initial spectrogram and the pure spectrum of the teacher's voice is calculated based on the spectral subtraction formula of the prior signal-to-noise ratio, and the pure spectrum is inversely short-time Fourier transformed to obtain an enhanced voice signal; For each video clip and audio clip, the time offset is determined and blank frames are added for synchronization alignment, and the aligned video clips and audio clips are alternately spliced according to the original time sequence to obtain the standard classroom teaching video.
3. The method according to claim 2, characterized in that The pure spectrum of the teacher's speech is calculated based on the spectral subtraction formula of the prior signal-to-noise ratio as shown in the following formula: ; in, represents the pure spectrum of the teacher's speech, ω is the frequency index, indicating the frequency component currently being analyzed. t is the time index, indicating the time frame of the current analysis, represents the power spectrum of the noisy speech spectrum, represents the corrected prior signal-to-noise ratio, Represents the power spectrum of the noise spectrum estimate.
4. The method according to claim 1, characterized in that The standard classroom teaching video is added to the pre-set target detection model and single target tracking model respectively. The target detection model performs target recognition through the attention mechanism and feature pyramid structure to obtain the target bounding box. The single target tracking model tracks the target teacher's position in each frame to obtain the spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through the posture estimation network in the target bounding box, and a three-dimensional skeleton model is constructed to obtain posture feature information. The audio track of the standard classroom teaching video is separated by the deep embedded clustering algorithm, and the separated speech segments are generated and learned through the gated recurrent unit network to obtain the speech feature vector. The line of sight movement of the target teacher in the standard classroom teaching is modeled by the three-dimensional convolutional neural network, and the gaze point distribution information is obtained, including: The standard classroom teaching video is added as input to a pre-set target detection model and a single target tracking model. The target detection model extracts a multi-scale feature map corresponding to each video frame in the standard classroom teaching video through a backbone network, and adjusts the weights corresponding to different regions in combination with a spatial attention module to obtain a spatial attention weight map and multiply the multi-scale feature map element by element to obtain an attention-enhanced feature representation and upsample the outputs of different stages of the backbone network from top to bottom based on a feature pyramid structure to obtain a fused multi-scale feature map. For the fused multi-scale feature map, a region candidate network is used to generate a region candidate box for the target teacher on the feature map of each scale, and the region candidate box is non-maximum suppressed and screened according to a threshold to obtain the target bounding box; The single target tracking model adds a template candidate frame in the first frame of the standard classroom teaching video according to the target image, uses the image in the template candidate frame as a tracking template, adds the target image and the tracking template to two branches based on the twin network setting and performs feature extraction, generates a tracking feature map and performs a cross-correlation operation, calculates a similarity score map, performs a soft normalization operation on the similarity score map, determines the center coordinates of the target image, and repeats the operation in each frame to obtain the spatiotemporal behavior trajectory; In the target bounding box, feature extraction is performed through a posture estimation network to generate a posture feature map and set three parallel branches, namely a heat map branch, a horizontal coordinate branch and a vertical coordinate branch. The heat map branch predicts the heat map of each key point through a one-dimensional convolution, and the horizontal coordinate branch and the vertical coordinate branch predict the horizontal coordinate and the vertical coordinate of each key point. Based on the prediction results obtained by the three parallel branches, the pixel coordinates of each key point are obtained, the pixel coordinates are estimated by a soft threshold random sampling consistency algorithm, and the depth information of the current key point is restored by the triangulation principle. The three-dimensional key point coordinates corresponding to each key point are obtained based on the depth information, and the three-dimensional key point coordinates are connected to obtain the three-dimensional skeleton model and the corresponding posture feature information; For the audio track of the standard classroom teaching video, a short-time Fourier transform is performed through a preset frame shift to generate an audio track spectrogram, the audio track spectrogram is added to a deep embedding network, the speaker embedding vector of each frame is extracted, the speaker embedding vector is clustered, the number of speakers is estimated and assigned to different categories, the speaker embedding vector is masked and de-embedded according to the clustering result, and the separated speech segment corresponding to each speaker is obtained, the separated speech segment is learned through a gated recurrent unit, the separated speech features and the corresponding weight matrix are determined, and the speech feature vector is obtained in combination with an attention weighting mechanism; Based on the standard classroom teaching video, a facial image sequence of the target teacher is obtained and added to the three-dimensional convolutional neural network. Feature extraction is performed according to the preset time convolution kernel and space convolution kernel, and a line of sight feature vector is output. The line of sight feature vector is predicted through a fully connected layer and a soft normalization function to obtain the gaze point coordinates of the target teacher in each frame. A gaze point coordinate sequence is generated based on the gaze point coordinates, and the gaze point distribution information is obtained in combination with statistical operations.
5. The method according to claim 4, characterized in that The similarity score graph calculated in the single-target model is shown in the following formula: ; in, S * ( m , n ) indicates the similarity score graph in ( m , n ) position, i represents the feature channel index, D represents the number of feature channels, u and v represents the coordinate index, Y i ( u , v ) indicates the i The expected output of a channel at position (u, v), Indicates the tracking feature map at position ( u , v ), Indicates the tracking feature map at position ( u , v ), λ 1 represents the L2 regularization parameter, λ 2 represents the regularization parameter of the spatial penalty term, W i ( u , v ) indicates that the spatial penalty weight is at position ( u , v ), Indicates the tracking feature map at position ( m+u , n+v) The value of .
6. The method according to claim 1, characterized in that The nodes are updated sequentially through the gated recurrent unit, and the hidden state of each node is updated as shown in the following formula: ; in, h i,t Represents the current time node i The hidden state of h i,t-1 Indicates the last moment t- 1 node i The hidden state of σ () represents the activation function, W z represents the update gate weight matrix, x i,t Indicates at the current moment t node i The input features of Indicates the current time t Candidate hidden states of .
7. An AI multi-dimensional teaching behavior analysis system based on classroom videos, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to obtain a classroom teaching video, extract video frames and audio frames of the classroom teaching video, adaptively segment the audio frames and the video frames into multiple video segments and multiple audio segments, perform image super-resolution reconstruction based on a deep convolutional neural network on the video segments, adaptively reduce noise on the audio segments and enhance them in combination with an adaptive enhancement algorithm using spectral subtraction, and combine the reconstructed video segments and the enhanced audio segments to obtain a standard classroom teaching video; The second unit is used to add the standard classroom teaching video to a preset target detection model and a single target tracking model respectively. The target detection model performs target recognition through an attention mechanism and a feature pyramid structure to obtain a target bounding box. The single target tracking model tracks the target teacher's position in each frame to obtain a spatiotemporal behavior trajectory. The three-dimensional key point coordinates of the target teacher are extracted through a posture estimation network in the target bounding box, a three-dimensional skeleton model is constructed and posture feature information is obtained. The audio track of the standard classroom teaching video is subjected to speaker separation through a deep embedded clustering algorithm, separated speech segments are generated, and learning is performed through a gated recurrent unit network to obtain a speech feature vector. The line of sight movement of the target teacher in the standard classroom teaching is modeled through a three-dimensional convolutional neural network to obtain gaze point distribution information. The third unit is used to construct a teaching behavior analysis graph based on a heterogeneous graph neural network, with the spatiotemporal behavior trajectory, the posture feature information, the voice feature vector and the gaze point distribution information as nodes, and the correlation between different nodes as edges. At each time step, the nodes are iteratively updated in time by a gated recurrent unit, and the attention weight of each node and the intrinsic correlation between nodes are determined by the self-attention pooling mechanism set in the last layer of the heterogeneous graph neural network to generate a temporal semantic representation. A multi-layer conditional generative adversarial network is constructed based on the temporal semantic representation, and an optimal teaching model is generated with the standard specification of teaching behavior as a condition. The optimal teaching model is added to the teaching behavior analysis graph for comparison with the current teaching behavior, and micro-deconstruction is performed in combination with the graph reasoning algorithm to obtain a teaching behavior analysis report.
8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Video intelligent analysis system based on deep learning
CN110232346A
Teacher-student behavior analysis system based on classroom videos
CN111709358A
Method for analyzing classroom interaction behaviors based on audio and video
CN114998968A
Real-time visual monitoring module of computer teaching device
CN117033141A
Teacher-student teaching interaction behavior analysis method and system based on speaker marking
CN117095684A
Cited By
Teacher teaching video sequence behavior analysis system based on large model
CN120318738A
Method for quickly calculating image correlation
CN120780858A
