A Method and System for Analyzing Teacher-Student Interaction Behavior Based on Speaker Marking

Abstract

This invention belongs to the field of classroom speech analysis technology and discloses a method and system for analyzing teacher-student interaction behavior based on speaker tagging. The method includes: speaker feature extraction to obtain MFCC features; introducing a time-delay neural network to extract x-vector feature vectors; inputting the two types of features into a preset neural network to extract deep features; fusing frame-level deep MFCC features and deep x-vector features based on a multi-head attention mechanism to obtain teacher-student fused deep features; optimizing the model using a loss function; and outputting frame-level prediction results of classroom speaker identities through the model. This invention not only achieves more accurate classroom teacher-student identity prediction but also provides a new method for S-T analysis, solving the problem of low efficiency in traditional S-T analysis, thereby better helping teachers analyze classroom behavior and improve teaching methods in a timely manner.

Description

Technical Field

[0001] This invention belongs to the field of speech analysis technology, and in particular relates to a method and system for analyzing teacher-student interaction behavior based on speaker tagging. Background Technology

[0002] Since ancient times, the classroom has been the main battleground of education. A good teaching model and a relaxed and pleasant teaching atmosphere can bring students twice the result with half the effort.

[0003] ST analysis is a relatively complete and scientific method for analyzing classroom teaching. It can measure and evaluate the "process and method" of teaching. This analysis method divides classroom teaching behavior into two categories: student behavior and teacher behavior. Student behavior is represented by "S" (student), and teacher behavior is represented by "T" (teacher).

[0004] In recent years, with the deepening of educational informatization, online learning models, born from the integration of traditional education and information technology, have gradually entered the public eye, and more and more online classrooms are emerging. However, speech is the main carrier of classroom teaching. Currently, the mainstream ST analysis method still relies on manual labeling and expert recognition, which is inefficient, inaccurate, and does not fully utilize all classroom speech information. By automatically identifying and labeling the classroom speech of teachers and students, simple ST analysis of classroom activities can be performed, revealing classroom interaction behaviors and the classroom atmosphere. This eliminates a large amount of tedious manual labeling work and helps teachers reflect on and improve their classroom behavior.

[0005] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0006] Manual labeling and expert identification: Current mainstream ST analysis methods rely on manual labeling and expert identification to determine student and teacher behavior. This method is inefficient and prone to inaccuracies. Manual labeling is time-consuming and resource-intensive, and expert subjective judgment may also contain errors.

[0007] Inadequate information utilization: Existing methods do not fully utilize all classroom audio information. Classroom interaction is not limited to student and teacher speech; it also includes nonverbal information such as speech rate, tone, and emotion, which are equally important for analyzing the effectiveness and atmosphere of classroom teaching. Existing technologies fail to fully utilize this information for comprehensive ST analysis.

[0008] Therefore, the key technologies for solving the above problems include:

[0009] Automatic identification and labeling: Develop automatic identification and labeling technologies, utilizing technologies such as speech recognition and natural language processing, to automatically identify and label student and teacher behaviors in the classroom. This will improve efficiency and reduce the workload of manual labeling, while also reducing errors caused by subjective human judgment.

[0010] Multimodal information analysis: In addition to voice information, other multimodal information, such as video and images, should also be considered. By comprehensively analyzing multiple information sources, including voice, language, non-verbal communication, and emotion, the effectiveness and atmosphere of classroom teaching can be evaluated more comprehensively and accurately. This may require the integration of related technologies such as computer vision and emotion recognition.

[0011] Machine Learning and Deep Learning: By applying machine learning and deep learning methods, and through training and model optimization on large-scale data, the accuracy and robustness of ST analysis can be improved. This will help the system better understand and interpret student and teacher behavior in the classroom, and provide more accurate assessments and feedback.

[0012] Intelligent Teaching System: This system applies the aforementioned technologies to intelligent teaching, enabling real-time monitoring, analysis, and feedback of student and teacher behavior in the classroom. It provides teachers with detailed ST (Student-Teacher) analysis reports and personalized improvement suggestions to help them optimize teaching methods and enhance teaching effectiveness.

[0013] In summary, addressing the shortcomings of existing technologies requires developing automatic identification and labeling technologies, multimodal information analysis, machine learning, and deep learning methods, and applying them to intelligent teaching systems to improve the efficiency and accuracy of ST analysis and promote the improvement of teaching quality. Summary of the Invention

[0014] To address the problems existing in the prior art, this invention provides a method and system for analyzing teacher-student interaction behavior based on speaker tagging.

[0015] This invention is implemented as follows: a method for analyzing teacher-student interaction behavior based on speaker tagging, comprising:

[0016] Step 1: Speaker feature extraction. Use relevant tools to collect frame-level voiceprint features (MFCCs) of students and teachers from simulated classroom audio, which serve as feature vector representations of teachers and students. Combine this with the frame-level labels of the audio to obtain the corresponding label representation vectors of teachers and students.

[0017] Step 2: Introduce a time-delay neural network to further extract features from teachers and students, and obtain the x-vector feature vector;

[0018] Step 3: Introduce a transformer encoder network to encode the features, perform deep representation learning on the MFCC features and x-vector features respectively, introduce a multi-head attention mechanism, assign different weights to different features, and obtain the teacher and student fused features through feature fusion respectively.

[0019] Step four: Combine the fused feature vector with the label representation vectors of teachers and students to predict identities, and construct the corresponding loss function for training;

[0020] Step 5: Apply the trained parameters to predict the audio in the test set, and post-process the prediction results to distinguish between teacher and student behaviors, analyze the classroom model accordingly, and visualize the analysis results.

[0021] Furthermore, step one specifically includes:

[0022] (1.1) Dataset preprocessing:

[0023] The LibriSpeech dataset was used as the simulation dataset, which included approximately 50 hours of audio data from 251 speakers (divided into one teacher and 250 students). The data was divided into training and testing sets in a 9:1 ratio. The audio was mixed into a long audio segment using the corresponding audio mixing algorithm. Each audio segment contained audio data from the teacher and several students, and noise, room impulse response, and corresponding time intervals were added to synthesize 1000 audio segments with a duration of approximately 2 to 3 minutes.

[0024] (1.2) Extraction of speaker-level MFCC features:

[0025] Using long audio as input, a 23-dimensional logarithmically scaled Mel filter bank is first extracted, with a frame length of 25ms and a frame shift of 10ms. Then, each of these filters is concatenated with those from the previous seven frames and the next seven frames, and the samples are double-sampled with a factor of 10. For every 100ms, 345 (=23×15) dimensional acoustic features are extracted.

[0026] Furthermore, in step two, a time-delay neural network is introduced into the MFCC feature vector extracted in step one. For the feature representation of each frame, the contextual dependency of the features is increased by considering the contextual information of the preceding and following frames, and a deep neural network (DNN) is used to learn the intermediate layer representation.

[0027] Furthermore, step three specifically includes:

[0028] MultiHead(Q,K,V)=Concat(head1,…,head h W O

[0029]

[0030] Where MultiHead(·) represents multi-head attention mechanism, Q represents query vector, K represents query vector, V represents content vector, Concat(·) represents concatenation, h represents the number of heads, and W represents the number of heads. O The training parameters, head1, ..., head1, represent the parameters to be linearly transformed after concatenation. h This represents the attention mechanism for each head, wherehead i Let W represent the attention matrix parameters for the i-th head, and Attention(·) represent a single attention mechanism. i Q , W i V Let Q, K, and V represent the training parameters for the i-th linear transformation, respectively.

[0031] Furthermore, step four specifically includes:

[0032] First, perform a layer normalization process on the fusion vector:

[0033]

[0034] Then, the frame-by-frame posterior probability z is obtained through a linear layer and the sigmoid activation function. t And calculate the corresponding loss:

[0035]

[0036]

[0037] Where z t It is the prediction vector output by the model, where each value represents the probability that the frame belongs to each speaker, and CE is the cross-entropy function between the label and the output.

[0038] Furthermore, step five specifically includes:

[0039] (1) After the model training is completed, the test audio is processed frame by frame. The prediction results of each time frame are obtained by inputting into the model. The results are filtered by setting a threshold, and speakers that do not exist in the audio are deleted. The label corresponding to the index with the highest remaining probability in each frame is selected as the prediction result of that frame. Finally, the speaker corresponding to each frame is obtained.

[0040] (2) Process the results and define the speaker identity corresponding to speaker 1 as teacher (determined by the label), and the identities corresponding to other speakers as students. In the results, distinguish when the teacher and students speak and analyze how they interact.

[0041] (3) Calculate the teacher behavior occupancy rate and student behavior occupancy rate, as well as the teacher-student behavior conversion rate (T->S or S->T is defined as one conversion) by statistically analyzing the number of teacher behaviors and student behaviors, and use this to analyze the teaching model and classroom atmosphere.

[0042] Another objective of this invention is to provide a teacher-student interaction behavior analysis system based on speaker tagging, comprising:

[0043] Speaker feature extraction module 1 is used to extract preliminary features (MFCC) and further feature representations (x-vector) of teacher and student speech based on audio.

[0044] The deep feature representation and fusion module 2 learns the deep representation of the speech features of teachers and students through the introduced neural network, and applies a multi-head attention mechanism to fuse the features, thereby obtaining the fused features required for the final prediction.

[0045] Speaker identity prediction module 3 inputs the obtained fused features into the final prediction module, combines them with the label representation vectors of teachers and students, calculates the corresponding loss according to the loss function, deletes speakers that do not exist in the audio based on the prediction results, retains speakers that actually exist in the audio, and obtains the final prediction results for each frame.

[0046] The teacher-student interaction behavior analysis module 4 uses the prediction results obtained above to distinguish whether each frame represents a teacher or a student, and combines the traditional ST analysis method to analyze classroom interaction and classroom teaching atmosphere.

[0047] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the teacher-student interaction behavior analysis method based on speaker tagging.

[0048] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the teacher-student interaction behavior analysis method based on speaker tagging.

[0049] Another objective of this invention is to provide an information data processing terminal for implementing the teacher-student interaction behavior analysis system based on speaker tagging.

[0050] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0051] First, regarding the technical problems existing in the above-mentioned prior art, the creative technical effects resulting from solving these problems are described in detail below:

[0052] (1) This invention constructs a multi-scale student and teacher speech representation, and the performance of the multi-scale speech representation is better than that of the single-scale feature.

[0053] (2) This invention constructs an end-to-end teaching behavior labeling model. The original ST analysis requires a lot of manual statistics on the behavior of teachers and students in the classroom, which requires a lot of manpower and resources. However, this system can directly label the behavior of students and teachers from the classroom audio and use it for subsequent analysis.

[0054] (3) This invention solves the problem that the number of speakers in each audio message is not fixed in real-world scenarios, making it more universal.

[0055] (4) The present invention performs visualization analysis on the results and presents the teacher-student classroom interaction mode in a visual form. The results are presented in the form of teacher-student discourse classroom time axis diagram, teacher-student discourse ratio diagram, teaching mode discrimination diagram, etc.

[0056] Secondly, the present invention provides a method for analyzing teacher-student interaction behavior based on speaker tagging. Starting from classroom audio, it performs deep learning representation and fusion on different features of classroom speakers. From the perspective of speaker tagging, it tags classroom speakers in the audio and records their speaking order, thereby realizing automated analysis of classroom teacher-student interaction behavior. It gets rid of the constraints of traditional ST analysis that requires a lot of manual tagging and provides an end-to-end teacher-student interaction behavior analysis model.

[0057] Third, the expected benefits and commercial value of the technical solution of this invention after transformation are as follows: The teacher-student interaction behavior analysis method based on multi-scale speaker tagging proposed in this invention constructs multi-scale student and teacher speech representations and builds an end-to-end teaching behavior tagging model. It solves the problem that traditional ST analysis requires a lot of manpower and resources, and solves the problem that the number of speakers in each audio clip is not fixed in real scenarios. It is of great benefit to classroom teaching behavior analysis, and helps teachers to reflect on and improve their teaching behavior, which has great commercial value.

[0058] The technical solution of this invention fills a technical gap in the industry both at home and abroad: existing end-to-end speaker labeling methods directly extract MFCC features as input without considering deeper speaker representation. This method extracts x-vector features with stronger representation capabilities and fuses MFCC features, and assigns corresponding weights as input to achieve more accurate prediction of speaker identity.

[0059] Does the technical solution of this invention solve a technical problem that people have long desired to solve but have never been able to successfully solve? The technical solution of this invention processes classroom audio by designing a multi-scale end-to-end speaker tagging model. While improving the prediction effect, it solves the problems of difficulty and time consumption in traditional ST analysis tagging. It enables rapid and efficient analysis of classroom teacher-student interaction behavior, thereby helping teachers better understand the teaching mode and classroom atmosphere, adjust strategies in a timely manner, and teach according to students' aptitude.

[0060] Fourth, the significant technological advancements in speaker-labeled teacher-student interaction behavior analysis at each step are as follows:

[0061] Step 1: By collecting frame-level voiceprint features (MFCCs) of students and teachers, this step achieves low-level feature extraction from the raw audio data. Unlike traditional speaker recognition, this method provides a feature vector for each participant in a simulated classroom environment, which lays the foundation for subsequent analysis.

[0062] 2) Step Two: The introduction of a Time Delay Neural Network (TDNN) enables the system to take into account the temporal information in the audio. As a more profound voiceprint feature, the x-vector can capture the speaker's long-term information and complement the short-term MFCC features, enhancing the speaker's feature description ability.

[0063] 3) Step Three: By introducing a transformer encoder network, this method incorporates deep representation learning during the feature extraction stage. This differs from traditional methods, which often perform classification immediately after feature extraction. The application of a multi-head attention mechanism allows the system to assign different weights to different features, thereby improving the feature fusion effect and enhancing the model's discriminative ability.

[0064] 4) Step four: By combining the fused feature vector and the label representation vector, the model not only focuses on the discriminativeness of the features during training, but also considers the correspondence with the actual labels, thereby enhancing the model's generalization ability and accuracy.

[0065] 5) Step Five: The post-processing process makes the prediction results more robust, filtering out possible outliers. Furthermore, by visualizing the analysis results, educators and researchers can more intuitively understand and evaluate teacher-student interaction patterns in the classroom, providing valuable feedback for educational practice.

[0066] In summary, this technical solution provides a deeply integrated approach to analyzing speaker-speaker interactions. It combines deep learning with traditional audio processing techniques, improving the accuracy and efficiency of speaker recognition and behavior analysis. Attached Figure Description

[0067] Figure 1 This is a flowchart of the teacher-student interaction behavior analysis method based on speaker tagging provided in an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the teacher-student interaction behavior analysis method based on speaker tagging provided in this invention example;

[0069] Figure 3 This is a model structure diagram used for teacher-student interaction behavior analysis based on speaker tagging provided in this invention example;

[0070] Figure 4 This is a partial prediction result diagram of an example of teacher-student interaction behavior analysis based on speaker tagging provided by the present invention.

[0071] Figure 5 This is a diagram showing the proportion of teacher-student speech in a teacher-student interaction behavior analysis system based on speaker tagging provided in this invention.

[0072] Figure 6 This is a classroom mode analysis diagram of the teacher-student interaction behavior analysis system based on speaker tagging provided in this invention example;

[0073] Figure 7 This is a comparison chart of the error rates in application examples of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0075] like Figure 1 As shown, the teacher-student interaction behavior analysis method based on speaker tagging provided in this embodiment of the invention includes the following steps:

[0076] S101, Speaker feature extraction: Use relevant tools to collect frame-level voiceprint features (MFCC) of students and teachers from simulated classroom audio, which are used as feature vector representations of teachers and students, and combined with the frame-level labels of the audio to obtain the corresponding label representation vectors of teachers and students.

[0077] S102, introduce a time-delay neural network to further extract the features of teachers and students, and obtain the x-vector feature vector;

[0078] S103 introduces a transformer encoder network to encode features, performs deep representation learning on MFCC features and x-vector features respectively, introduces a multi-head attention mechanism to assign different weights to different features, and obtains the teacher and student fused features through feature fusion respectively.

[0079] S104 combines the fused feature vector with the label representation vectors of teachers and students to predict identities and constructs a corresponding loss function for training.

[0080] S105 applies the trained parameters to predict the audio of the test set, and performs post-processing on the prediction results to distinguish between teacher behavior and student behavior, analyze the classroom mode accordingly, and visualize the analysis results.

[0081] The principle diagram of the teacher-student interaction behavior analysis method based on speaker tagging provided in this embodiment of the invention is as follows: Figure 2 As shown in Table 1. The symbols used in the embodiments of this invention are also shown in Table 1.

[0082] Table 1. Symbols involved in the embodiments of the present invention.

[0083]

[0084]

[0085] Step one provided in this embodiment of the invention specifically includes:

[0086] (1.1) Dataset preprocessing:

[0087] We used the LibriSpeech dataset as the simulated dataset for our experiment, which includes approximately 50 hours of audio data from 251 speakers (divided into one teacher and 250 students), and was divided into training and test sets in a 9:1 ratio.

[0088] Since the audio data contained in this dataset are all short, single-person audio data, which does not match the characteristics of a classroom environment with long durations and multiple speakers interacting, a corresponding audio mixing algorithm was used to mix these audios into a long audio segment. Each audio segment contains audio data of the teacher and several students. In order to simulate the classroom environment, noise, room impulse response, and corresponding time intervals were added to it. 1000 audio segments with a duration of about 2 to 3 minutes were synthesized and used as our training set and validation set, respectively.

[0089] (1.2) Extraction of speaker-level MFCC features:

[0090] Using these long audio clips as input, a 23-dimensional logarithmically scaled Mel filter bank is first extracted, with a frame length of 25ms and a frame shift of 10ms. Each of these filters is then concatenated with those from the preceding and following seven frames, and subsequently subsampled by a factor of 10. The result is a 345 (23 × 15) dimensional acoustic feature extraction per 100ms.

[0091] Since the duration of different audio segments in the dataset varies greatly and the audio duration is relatively long, in order to meet the input requirements of the network, we divide the continuous audio into several 50-second speech segments with a sampling rate of 16000Hz, and discard the insufficient parts; then we extract features from the segmented audio segments.

[0092] And by combining the speaker's label to which the audio belongs, the speaker's representation vector I is obtained. t :

[0093] I t =[0,0,....1,0..],I t ∈C

[0094] If the speaker at time step t is i, then I t The i-th dimension is 1, and the rest are 0.

[0095] Step two provided in this embodiment of the invention specifically includes:

[0096] For the MFCC feature vectors extracted in step one, a time-delay neural network is introduced. For the feature representation of each frame, the contextual dependency of the features is increased by considering the contextual information of the preceding and following frames. To obtain more discriminative speaker-related features, a deep neural network (DNN) is used to learn the intermediate layer representations. The structure of the DNN is shown in Table 2.

[0097] Table 2. Structure of the DNN used in this invention

[0098]

[0099]

[0100] The first five layers of the network are frame-level layers. The first layer is based on a TDNN network. Assuming the current frame is t, it considers the features of the two frames before and after the current frame. Since speech is continuous, five consecutive frames can generally be considered to belong to the same speaker, and they share a common label. The statistical pooling layer aggregates the outputs of these five frames and calculates their mean and standard deviation. This process aggregates information across the time dimension and propagates this information through the segment layer, finally reaching the softmax output layer to obtain the classification results for N speakers. After training, x-vector embeddings are extracted from segment 6, eliminating the need for segment 7 and the softmax layer. The x-vector has already been proven effective in speaker recognition tasks in Deep Neural Network Embeddings for Text-Independent Speaker Verification, demonstrating stronger temporal modeling and speaker variation modeling capabilities compared to general features.

[0101] Step three provided in this embodiment of the invention specifically includes:

[0102] (3.1) Introducing the transformer network (structure diagram as follows) Figure 3 As shown), deep representation learning is performed on the obtained MFCC vector and x-vector vector respectively. Taking MFCC features as an example, the extraction method of x-vector is similar:

[0103] First, project the input vector onto a D-dimensional vector:

[0104]

[0105] The resulting D-dimensional vector input sequence is then fed into the Encoder, which outputs the corresponding D-dimensional vector:

[0106]

[0107] in This represents the p-th encoder block.

[0108] (3.2) Structural features of the encoder:

[0109] Input sequence Transformed into R T×D The matrix is ​​then subjected to layer normalization:

[0110]

[0111] Next, we move to the multi-head self-attention mechanism module, which uses the dot product of the query vectors to calculate the pairwise similarity matrix. and according to Scaling is performed, and the softmax function is applied to form the attention weight matrix. Use the attention weight matrix to compute the context content vector. Finally, the output projection vector O is used. (p) The context vectors of all heads are concatenated and projected to obtain a self-fused feature vector. After the self-attention layer, residual connections and layer normalization are applied to alleviate the vanishing and exploding gradient problems, resulting in the final feature vector input to the next encoder.

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] in These are the query matrix, the query object matrix, and the content vector matrix for the h-th head, respectively.

[0118] A positional feedforward layer is introduced after each attention sublayer. This layer consists of two fully connected layers and an activation function, introducing non-linearity to better capture complex patterns and relationships in the input sequence, thereby enabling the learning of higher-level feature representations.

[0119]

[0120] W1, W2, b1, b2 are the linear projection matrices and biases of the first and second linear layers, respectively.

[0121] Next, the output of the encoder block for each time frame is computed by applying residual connections:

[0122]

[0123] The same process can be applied to the x-vector features to obtain...

[0124] (3.3) A single feature can only capture a limited number of speaker characteristics, often lacking comprehensiveness. To more accurately represent speaker characteristics, the embeddings of these two features are fused to form the final feature representation for decision-making. Since not all frame-level features contribute equally to speaker recognition, a multi-head attention mechanism is used for feature fusion, assigning different weights to different features. Relatively important features are given higher weights, thus contributing more to the final recognition result.

[0125] MultiHead(Q,K,V)=Concat(head1,…,head h W O

[0126]

[0127] Where MultiHead(·) represents multi-head attention mechanism, Q represents query vector, K represents query vector, V represents content vector, Concat(·) represents concatenation, h represents the number of heads, and W represents the number of heads. O The training parameters, head1, ..., head1, represent the parameters to be linearly transformed after concatenation. h This represents the attention mechanism for each head, wherehead i Let W represent the attention matrix parameters for the i-th head, and Attention(·) represent a single attention mechanism. i Q , W i V Let Q, K, and V represent the training parameters for the i-th linear transformation, respectively.

[0128] Step four provided in this embodiment of the invention specifically includes:

[0129] By combining the fused feature vector and the speaker label representation vector, and applying the appropriate loss function, the model is trained and speaker identity is predicted.

[0130] First, perform a layer normalization process on the fusion vector:

[0131]

[0132] in This represents the fused feature vector at time step t of layer p. This represents the fused vector features across all time steps after normalization of layer p;

[0133] Then, the frame-by-frame posterior probability z is obtained through a linear layer and the sigmoid activation function. tAnd calculate the corresponding loss:

[0134]

[0135]

[0136] Where z t is the predicted vector output by the model, where each value represents the probability that the frame belongs to each speaker. W3 and b3 are the linear projection matrix and bias of the last layer, respectively. CE is the cross-entropy function between the label and the output.

[0137] Step five provided in this embodiment of the invention specifically includes:

[0138] After training the above model, the test audio is processed frame by frame using the above method. The prediction result of each time frame is obtained by inputting it into the model. If the probability of the speaker corresponding to each frame is greater than 0.5, we consider that the speaker to be in the audio. Then, the label corresponding to the index with the highest remaining probability in each frame is selected as the prediction result of that frame. Finally, the speaker corresponding to each frame is obtained.

[0139] The results are then processed. We define the speaker identity corresponding to speaker 1 as teacher (determined by the label), and the identities corresponding to other speakers as students. This allows us to distinguish when teachers and students speak and analyze how they interact in the results.

[0140] The teacher-student teaching interaction behavior analysis system based on multi-scale end-to-end speaker tagging provided in this embodiment of the invention includes:

[0141] Speaker feature extraction module 1 is used to extract preliminary features (MFCC) and further feature representations (x-vector) of teacher and student speech based on audio.

[0142] The deep feature representation and fusion module 2 learns the deep representation of the speech features of teachers and students through the introduced neural network, and applies a multi-head attention mechanism to fuse the features, thereby obtaining the fused features required for the final prediction.

[0143] Speaker identity prediction module 3 inputs the obtained fused features into the final prediction module, combines them with the label representation vectors of teachers and students, calculates the corresponding loss according to the loss function, deletes speakers that do not exist in the audio based on the prediction results, retains speakers that actually exist in the audio, and obtains the final prediction results for each frame.

[0144] The teacher-student interaction behavior analysis module 4 uses the prediction results obtained above to distinguish whether each frame represents a teacher or a student, and combines the traditional ST analysis method to analyze classroom interaction and classroom teaching atmosphere.

[0145] The main development environment used in this invention experiment includes: Windows 10, GTX 1080Ti, Chainer 5.4.0, and Python 3.7. The specific hyperparameter settings of the model are shown in Table 3 below.

[0146] Table 3. Hyperparameter settings of the experimental model

[0147]

[0148]

[0149] The method for analyzing teacher-student teaching interaction behavior based on multi-scale end-to-end speaker tagging provided in the application embodiment of the present invention is applied to a computer device. The computer device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of the method for analyzing teacher-student teaching interaction behavior based on multi-scale end-to-end speaker tagging.

[0150] The teacher-student teaching interaction behavior analysis method based on multi-scale end-to-end speaker tagging provided in the application embodiment of the present invention is applied to an information data processing terminal, which is used to implement the teacher-student teaching interaction behavior analysis system based on multi-scale end-to-end speaker tagging.

[0151] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0152] This invention compares a multi-scale end-to-end speaker labeling method with a single-scale speaker labeling method, using the area under the curve (DER) as the comparison metric. The DER (diarization error rate), also known as the speaker clustering error rate, is a common metric used to evaluate speaker segmentation and clustering performance. It compares the reference label segment with the predicted label segment and calculates the prediction error rate. A smaller DER indicates better model performance and a lower prediction error rate. To ensure a fair comparison, the hyperparameters of the corresponding modules in both models were set to the same value. The comparison results are shown in Table 4. The training process of the models on the dataset is as follows: Figure 7 As shown.

[0153] Table 4 Comparison of experimental results from different methods

[0154]

[0155] Table 5 shows some prediction results from embodiments of the present invention. Taking the prediction result of a certain audio as an example, the speaker identity at each time step is output:

[0156] Table 5. Partial prediction results of examples of the present invention.

[0157] T T T T T T S2 S2 S2 S2 S2 T T T T T T T T T T T S3 S3 S3 S3 S3 S3 S3 S3 S3 S3 T T T T T T T T T S4 S4 S4 S4 S4 S4 S4 T T T T T T T T T S5 S5 S5 S5 S5 S5 S5 S5 S5 T T T T T T T T T T S6 S6 S6 S6 S6 S6 S6 S6 T T T T T T T T T T S7 S7 S7 S7 S7 S7 S7 T T T T T T T T T T T T T T T T S8 S8 S8 S8 S8 S8 S8 T T T T T T T T

[0158] By comparing the predicted results with the image, and according to the preset frame times, the speaking times and order of the teacher and each student are obtained as follows: Figure 4 Each student and teacher is represented by a different color, with the horizontal axis representing time and the vertical axis representing different speakers.

[0159] Next, we will use ST analysis to analyze the interactive behavior in this class:

[0160] First, we statistically analyze the behavior of teachers and students. Since classroom discourse is continuous, we use the duration of teacher and student speaking as the statistical indicator. Then, we calculate the duration of classroom silence (i.e., the duration of no one speaking), thus obtaining a graph showing the proportion of teacher and student discourse, as shown below. Figure 5 .

[0161] Next, calculate the percentage of teacher-student behavior (R). t = Teacher speaking time / Total audio duration). In this example, we have a total of 195 seconds of audio, of which the teacher speaks for 107 seconds and the students speak for 65 seconds. The calculated teacher behavior percentage R is... t =0.54. This shows that the teacher plays a major role in the classroom, and the teacher does most of the talking in class.

[0162] Teacher-student behavior conversion rate ch=g / N, where g represents the number of teacher-student behavior conversions, and N represents the total number of teacher-student behaviors. For example: if the result of the S-T analysis of this classroom is "SS->TTTT->S->T->S->T", then the total number of behaviors N is 10, and the number of teacher-student behavior conversions is 5.

[0163] In this example, the total number of teacher-student behaviors is 1950, and the number of teacher-student behavior conversions is 858. The teacher-student behavior conversion rate of this class is calculated as ch=0.44. It can thus be obtained that the teacher and students interact frequently in this class, and the classroom atmosphere is relatively active.

[0164] Finally, classroom pattern analysis is performed, drawing on Liu Lixin's analysis method: classroom patterns are divided into five types, namely: conversational type, balanced type, block type, practice type, and lecture type. Where R t ≥0.7 and ch<0.4 is defined as lecture type, R t ≤0.3 and ch<0.4 is defined as practice type, 0.3<R t <0.7 and ch≥0.4 is conversational type, 0.3<R t <0.7 and ch≤0.2 is block type, 0.3<R t <0.7 and 0.2<ch<0.4 is balanced type.

[0165] Figure 5 , Figure 6 are the analysis results of the teacher-student speech proportion diagram and the classroom pattern diagram in this example respectively. As can be seen from the diagrams, this class is a conversational classroom, the interaction between teachers and students is relatively frequent, and the classroom atmosphere is relatively active.

[0166] Under the multi-scale student and teacher speech representation, compared with the original single-scale representation, the identity marking error rate of teachers and students is reduced by about 5%. The experimental results are shown in Figure 7 .

[0167] In conclusion, the teacher-student teaching interaction behavior analysis and system based on multi-scale end-to-end speaker diarization provided by the present invention not only has improved performance compared with the traditional end-to-end speaker diarization method, but also provides a new idea, that is, automatic marking of classroom speech can be used to analyze the teacher-student classroom interaction behavior, thereby helping teachers to reflect on classroom teaching behaviors, and also saving a large amount of human and material resources.

[0168] Based on the aforementioned teacher-student interaction behavior analysis method based on speaker diarization, the following are two specific embodiments and their implementation schemes:

[0169] Example 1: Real-time teacher-student interaction analysis system

[0170] 1) Hardware equipment: A multi-microphone array system is deployed in various corners of the classroom to ensure comprehensive capture of audio information during class.

[0171] 2) Preprocessing: Noise removal and enhancement are performed on the captured audio to ensure the quality of the voiceprint features.

[0172] 3) Feature extraction: Perform feature extraction and fusion according to the steps above.

[0173] 4) Interaction analysis: Based on the pre-trained model, predict in real time whether the speaker is a teacher or a student, and label and classify their interaction patterns.

[0174] 5) Real-time feedback: A visual dashboard displays real-time teacher-student interaction patterns and statistics in the classroom, such as the number of times students ask questions and the time spent explaining by the teacher.

[0175] 6) Data storage: Store the interactive data of each lesson for future analysis and review.

[0176] Example 2: Teacher-Student Interaction Assessment Tool

[0177] 1) Input module: Allows users to upload classroom audio files.

[0178] 2) Audio processing: The uploaded audio is segmented and preprocessed.

[0179] 3) Feature extraction and analysis: Perform feature extraction and teacher and student behavior analysis according to the steps of the above method.

[0180] 4) Evaluation Report Generation: Based on the analysis results, a detailed evaluation report on teacher-student interaction will be generated. This report includes, but is not limited to: teacher lecture time, student interaction time, number of questions asked, peak interaction times, etc.

[0181] 5) Suggestions module: Based on the analysis results, suggestions are provided to teachers, such as the need to increase student participation and to give students more opportunities to ask questions.

[0182] 6) Historical comparison: Allows users to view past assessment reports and compare teacher-student interaction patterns in different classes or time periods, so as to facilitate self-reflection and improvement.

[0183] These two examples provide educators with a practical tool to help them better understand classroom interactions and make corresponding improvements.

[0184] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for analyzing teacher-student interaction behavior based on speaker tagging, characterized in that, The analytical method includes: Step 1: Speaker feature extraction. Collect frame-level voiceprint features (MFCCs) of students and teachers from the simulated classroom audio, and use them as feature vector representations of teachers and students. Combine the frame-level labels of the audio to obtain the corresponding label representation vectors of teachers and students. Step 2: Introduce a time-delay neural network to further extract features from teachers and students, obtaining x-vector features; Step 3: Introduce a transformer encoder network to encode the features, perform deep representation learning on the MFCC features and x-vector features respectively, introduce a multi-head attention mechanism, assign different weights to different features, and obtain the teacher and student fused features through feature fusion respectively. Step 4: Combine the fused features with the label representation vectors of teachers and students to predict identities, and construct the corresponding loss function for training. Step 5: Apply the trained parameters to predict the audio in the test set, and post-process the prediction results to distinguish between teacher behavior and student behavior, analyze the classroom model accordingly, and visualize the analysis results. In step two, a time-delay neural network is introduced into the MFCC features extracted in step one. For the feature representation of each frame, the contextual dependency of the features is increased by considering the contextual information of the frames before and after it, and a deep neural network is used to learn the intermediate layer representation. Step five specifically includes: (1) After the model training is completed, the test audio is processed frame by frame. The prediction results of each time frame are obtained by inputting into the model. The results are filtered by setting a threshold, and speakers that do not exist in the audio are deleted. The label corresponding to the index with the highest remaining probability in each frame is selected as the prediction result of that frame. Finally, the speaker corresponding to each frame is obtained. (2) Process the results, define the speaker identity corresponding to speaker 1 as teacher, and the identities corresponding to other speakers as students, and divide the results to determine when the teacher and students speak and analyze how they interact. (3) The teacher behavior rate and student behavior rate, as well as the teacher-student behavior conversion rate, are calculated by counting the number of teacher behaviors and student behaviors, and the teaching model and classroom atmosphere are analyzed accordingly.

2. The method for analyzing teacher-student interaction behavior based on speaker tagging according to claim 1, characterized in that, Step one specifically includes: (1.1) Dataset preprocessing: The LibriSpeech dataset was used as the simulation dataset, which includes 50 hours of audio data from 251 speakers. The dataset was divided into training and testing sets in a 9:1 ratio. The audio was mixed into a long audio segment using the corresponding audio mixing algorithm. Each audio segment contained audio data from the teacher and several students. Noise, room impulse response, and corresponding time intervals were added to the audio segments to synthesize 1,000 audio segments with a duration of 2 to 3 minutes. (1.2) Extraction of speaker-level MFCC features: Using long audio as input, we first extract a 23-dimensional logarithmically scaled Mel filter bank with a frame length of 25ms and a frame shift of 10ms. Then, we concatenate each of these filters with those from the previous seven frames and the next seven frames, and perform secondary sampling with a factor of 10. For every 100ms, we extract 345-dimensional acoustic features.

3. The method for analyzing teacher-student interaction behavior based on speaker tagging according to claim 1, characterized in that, Step three specifically involves: ； ； in, This indicates a multi-head attention mechanism. Represents the query vector. Represents the vector being searched. Represents a content vector. Indicates splicing, express The number of heads, This represents the training parameters for the linear transformation after concatenation. This represents the attention mechanism of each head. Indicates the first indivual Attention matrix parameters, Represents a single attention mechanism. They represent , , Conduct the first Training parameters for the linear transformation.

4. The method for analyzing teacher-student interaction behavior based on speaker tagging according to claim 1, characterized in that, Step four specifically includes: First, perform a layer normalization process on the fusion vector: ； This represents the fused vector features across all time steps after normalization of layer p; Then, the posterior probability for each frame is obtained through a linear layer and the sigmoid activation function. And calculate the corresponding loss: ； ； in This is the prediction vector output by the model, where each value represents the probability that the frame belongs to each speaker. , These are the linear projection matrix and bias of the last layer, respectively, and CE is the cross-entropy function between the label and the output.

5. A teacher-student interaction behavior analysis system based on speaker tagging, implementing the analysis method as described in any one of claims 1-4, characterized in that, The teacher-student interaction behavior analysis system based on speaker tagging includes: Speaker feature extraction module 1 is used to extract preliminary features (MFCC) and further feature representations (x-vector) of teacher and student speech based on audio. The deep feature representation and fusion module 2 learns the deep representation of the speech features of teachers and students through the introduced neural network, and applies a multi-head attention mechanism to fuse the features, thereby obtaining the fused features required for the final prediction. Speaker identity prediction module 3 inputs the obtained fused features into the final prediction module, combines them with the label representation vectors of teachers and students, calculates the corresponding loss according to the loss function, deletes speakers that do not exist in the audio based on the prediction results, retains speakers that actually exist in the audio, and obtains the final prediction results for each frame. The teacher-student interaction behavior analysis module 4 uses the prediction results obtained above to distinguish whether each frame represents a teacher or a student, and combines the traditional ST analysis method to analyze classroom interaction and classroom teaching atmosphere.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the teacher-student interaction behavior analysis method based on speaker tagging as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the speaker-marked teacher-student interaction behavior analysis method as described in any one of claims 1-4.

Images