Multimodal classroom teacher speech behavior analysis method, system and device based on text-driven

Abstract

This invention discloses a text-driven, multimodal classroom teacher speech behavior analysis method, system, and device. This method constructs a multimodal learning model capable of classifying classroom teacher speech behavior, improving the model's classification performance. This method fuses different modalities using a text-driven approach and employs a cross-modal attention mechanism to focus on the contributions of different modalities. It also extracts features based on the data characteristics of different modalities and selects appropriate representations, enriching the feature representation of each modality. Furthermore, it improves the loss function to optimize model parameters, improving the classification accuracy of imbalanced data.

Description

Technical Field

[0001] The present invention belongs to the field of data mining technology, and relates to a method, system and device for analyzing the speech behavior of teachers in a multimodal classroom based on text driving, and specifically relates to a method, system and device for analyzing the speech behavior of teachers in a multimodal classroom based on text driving. Background Art

[0002] The development of smart classroom devices has made it possible to collect and analyze classroom data, primarily in the form of video, audio, and text. In classrooms, teacher-student interactions are primarily verbal. High-quality verbal interaction can stimulate deep thinking in students, and analyzing teacher verbal behavior can effectively help teachers improve teaching quality. Traditional analysis of teacher teaching behavior relies primarily on manual observation, which is subject to high subjectivity and low efficiency. Therefore, intelligent methods are needed to improve the effectiveness of teacher verbal behavior analysis.

[0003] Multimodality refers to the fusion of two or more modalities. Multimodal learning can establish associations between data by processing different types of data such as video, audio, and text, and explore the potential shared information of the data, ultimately achieving semantic unification between multiple modalities and obtaining better learning results. A multimodal learning model refers to a neural network model that can complete multimodal learning tasks. Multimodal learning models can complement information from different modalities, and compared to single-modal learning models, multimodal learning models obtain richer and more comprehensive information. Due to the different sources of information in multimodal data and the different contributions of various modalities in different application scenarios, multimodal learning models still face some challenges in representation and fusion, including how to convert data in multimodal data into vector modal representations and how to align data during the fusion process while retaining modality-specific information. Summary of the Invention

[0004] The purpose of the present invention is to propose a multimodal classroom teacher speech behavior analysis method, system and equipment based on text-driven, using the semantic features of text and the acoustic features of audio to construct a text-driven cross-modal attention fusion model to improve the accuracy of teacher speech behavior classification.

[0005] The technical solution adopted by the method of the present invention is: a multimodal classroom teacher speech behavior analysis method based on text-driven, comprising the following steps:

[0006] Step 1: Get the classroom video and extract the audio from the video;

[0007] Step 2: transcribe the audio to text, recording the pause time and speaker;

[0008] Step 3: Divide the audio into audio segments according to the voice pauses recorded in step 2;

[0009] Step 4: audio data processing, screening and retaining the teacher's audio segment and text;

[0010] Step 5: extracting audio features and text features of the audio segment in step 4;

[0011] In step 6, the audio features and text features obtained in step 5 are input into the constructed text-driven multimodal classroom teacher speech behavior classification model, and a loss function is constructed to optimize the classification results.

[0012] Furthermore, in step 2, the API of iFLYTEK is called to transcribe the audio into text and record it in a CSV file. The record format is start time, end time, speaker, and text content.

[0013] Furthermore, in step 3, the audio in step 1 is divided into audio segments of unequal lengths according to the start and end times recorded in step 2.

[0014] Furthermore, in step 4, the audio segment and text of the student are removed according to the speaker recorded in step 2, and the audio segment and text containing noise and silence are removed according to the recorded text content.

[0015] Furthermore, in step 5, first, the Mel-frequency cepstral coefficient MFCC features and acoustic low-level descriptor LLD features of the audio segment are extracted; secondly, the text is input into the Chinese pre-trained model Bert to obtain word vector features.

[0016] Furthermore, the specific implementation process of step 6 is as follows:

[0017] First, the audio feature MFCC is input into the convolution module of the Conformer to obtain the convolved audio feature MFCC, and the audio feature LLD is input into the BiLSTM module to obtain the processed audio feature LLD. Then, the convolved audio feature MFCC and the processed audio feature LLD are respectively input into the one-dimensional convolution to obtain the final audio feature MFCC and LLD;

[0018] Then the Bert word vector feature is input into the one-dimensional convolution to obtain the final text feature;

[0019] Finally, the final audio features MFCC and LLD are fused with the text features in a text-centric attention fusion process. The features fused by the attention mechanism are superimposed and input into the maximum pooling layer and the fully connected layer through the activation function to predict the category of the teacher's speech behavior in the classroom. The final loss function consists of three parts: the cross-entropy loss function, the loss function of the single modality pair, and the loss function between the single modality and the fused modality.

[0020] Furthermore, the specific implementation formula for fusing the final audio features MFCC and LLD with the text features in a text-centric attention fusion process is as follows;

[0021]

[0022]

[0023] Among them, in Attention m→t middle, Similarly, in Attention l→t middle, W q 、W k and W v are the different weights of query, key and value respectively, L is the linear transformation of the input vector, d t is the input text feature X t The dimension, d k is the dimension of the query vector / key vector, and n is the hidden representation dimension in the model;

[0024] The specific implementation method of predicting the classification results of teachers' speech behaviors in the classroom is as follows:

[0025] Add=Attention m→t +Attention l→t

[0026] x=f(Add)=max(0,Add)

[0027] y i,j,k =max p,q (x i+p,j+p,k )

[0028]

[0029] Among them, f(Add) represents the Relu activation function; y i,j,k It is the calculation of the maximum pooling layer, which is calculated as follows: i,j,k is the value of the kth channel in the i-th row, j-th column of the output matrix, x i+p,j+p,k It is the maximum value of the kth channel in the rectangular area with (i+p,j+q) as the upper left corner and (i+p+P,j+q+Q) as the lower right corner in the input matrix. P and Q are the window sizes of the pooling layer. p and q refer to the offset of the upper left corner coordinates of the pooling selection area relative to the coordinates of the output matrix. is the result of the calculation of the fully connected layer, where W A is the weight matrix and b is the bias.

[0030] Furthermore, the cross entropy loss function The calculation formula is as follows:

[0031]

[0032] Among them, m is the size of each batch, n is the number of categories, the true distribution is y, and the network output distribution is

[0033] Loss function for unimodal pairs The calculation formula is as follows:

[0034]

[0035]

[0036] Where I represents the lower bound of MI that maximizes two modal pairs. In each modal pair, one modality is considered as x and the other modality is considered as y. t,m The text mode is regarded as x, and the MFCC mode of the audio is regarded as y. The formula of input I is calculated; similarly, I t,l The text mode is considered as x and the LLD mode of the audio is considered as y during calculation; m,l When calculating, the MFCC mode of the audio is regarded as x, and the LLD mode of the audio is regarded as y; p(y|x) is the true probability distribution, q(y|x) is the estimated probability distribution, E p(x,y) and E p(y) is the expected value in the probability distribution; KL(·) represents the KL divergence;

[0037] Loss function between single modality and fusion modality The calculation formula is as follows:

[0038]

[0039]

[0040] It is noise contrast estimation. Its core idea is to compare the representation of one modality with other representations of the modality in the same batch, and regard all other representations of the modality in the same batch as negative samples. Among them, Z represents the result of multimodal fusion, that is, the vector obtained by superimposing the features after the fusion of the attention mechanism. represents the negative samples generated by a single modality before fusion, represents the positive sample generated by a single modality before fusion, i and j represent the i-th and j-th samples; E X Indicates expected value, exp() indicates exponential operation; and Represents the noise contrast estimation loss of text, MFCC and LLD respectively, in the calculation When X p =X t , similarly, in the calculation When X p =X m , in the calculation When X p =X l ;

[0041] The final loss function is as follows:

[0042]

[0043] Among them, α and β are adjustable hyperparameter weights.

[0044] Based on the same inventive concept, the present invention also designs a text-driven multimodal classroom teacher speech behavior classification system, which includes the following modules:

[0045] Audio extraction module, used to obtain classroom videos and extract audio from them;

[0046] Audio transcription module, used to transcribe audio into text, record speech pause time and speaker;

[0047] An audio segmentation module, used to segment the audio into audio segments according to the recorded voice pauses;

[0048] A processing module is used to process the audio data and filter and retain the teacher's audio segment and text;

[0049] A feature extraction module extracts audio features and text features of the audio segments in the processing module;

[0050] The prediction module is used to input the obtained audio features and text features into the constructed text-driven multimodal classroom teacher speech behavior classification model, and construct a loss function to optimize the classification results.

[0051] Based on the same inventive concept, the present invention also provides an electronic device, comprising:

[0052] one or more processors;

[0053] A storage device is used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement multimodal classroom teacher speech behavior analysis based on text-driven main drive.

[0054] Based on the same inventive concept, the present invention also designs a computer-readable medium on which a computer program is stored. When the program is executed by a processor, it realizes multimodal classroom teacher speech behavior analysis based on text-driven main drive.

[0055] The advantages of the present invention are:

[0056] (1) The network model trained with the entire training set can achieve significant results in classifying classroom teacher speech behaviors, and the classification effect is better than that of single-modal text or audio.

[0057] (2) Extract features and select appropriate representation methods based on the data characteristics of different modalities. In the extraction of audio features, both fine-grained frame-level features and coarse-grained speech-level features are considered.

[0058] (3) It integrates different modalities in a text-driven manner, considers the contribution of different modal data, and uses a cross-modal attention mechanism to pay more attention to the text modality with rich information.

[0059] (4) By optimizing the loss function, the overall effect of the classification model is improved, and the improvement in classification effect is more significant when the sample size is small.

[0060] This method acquires audio and text data from classroom videos and constructs a text-driven, multimodal model for classifying teacher speech behavior. This model leverages the complementary nature of multimodal information to focus on the contributions of information from different modalities, improving classification performance. Regarding multimodal representation, features are extracted and appropriate representation methods are selected based on the characteristics of the data from different modalities. Audio feature extraction considers both fine-grained frame-level features and coarse-grained utterance-level features. Text feature extraction uses the Chinese pre-trained model BERT to obtain word embeddings, enriching the feature representation of modalities. Furthermore, model optimization improves the loss function, enhancing classification accuracy for imbalanced data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0062] Figure 2 This is a model structure diagram of an embodiment of the present invention;

[0063] Figure 3 MFCC feature extraction flow chart in an embodiment of the present invention;

[0064] Figure 4 This is a diagram of the Conformer convolution structure in an embodiment of the present invention;

[0065] Figure 5 This is a BiLSTM structure diagram in an embodiment of the present invention;

[0066] Figure 6 This is a diagram of the cross-modal attention structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0068] As attached Figure 1 、 Figure 2 As shown, the present invention provides a text-driven multimodal classroom teacher speech behavior analysis method, which includes the following steps:

[0069] Step 1: Get the classroom video and extract the audio from the video;

[0070] In this embodiment, the specific implementation of step 1 includes the following sub-steps:

[0071] Step 1.1: Get and read the classroom video;

[0072] Step 1.2: Separate the audio from the video and save it in WAV format;

[0073] Step 2: Transcribe the audio to text, recording the pause time and speaker;

[0074] In this embodiment, the specific implementation of step 2 includes the following sub-steps:

[0075] Step 2.1: Call iFlytek's speech transcription API

[0076] Step 2.2: Set the output format and save it in a CSV file. The record format is start time, end time, speaker, and text content.

[0077] Step 3: Divide the audio into audio segments according to the voice pauses recorded in step 2;

[0078] In this embodiment, the specific implementation of step 3 includes the following sub-steps:

[0079] Step 3.1: Read the start time and end time data columns of the CSV file generated in step 2.2;

[0080] Step 3.2: Automatically split the audio separated in step 1.2 into audio segments of unequal lengths, rename and save;

[0081] Step 4: Audio data processing, screening and retaining the teacher's audio segment and text;

[0082] In this embodiment, the specific implementation of step 4 includes the following sub-steps:

[0083] Step 4.1: Read the speaker and text content data columns of the CSV file generated in step 2.2;

[0084] Step 4.2: Delete the audio and text content where the speaker is the student;

[0085] Step 4.3: Delete the audio and text contents that are empty or garbled;

[0086] Step 5: Extract audio features and text features of the audio segment in step 4;

[0087] In this embodiment, the specific implementation of step 5 includes the following sub-steps:

[0088] Step 5.1: Read the teacher audio filtered in step 4, as shown in the attached Figure 3 The figure shows the MFCC feature extraction flow chart, setting parameters to extract audio feature MFCC;

[0089] Step 5.2: Read the teacher audio filtered in step 4 and use the openSMILM tool to extract the audio features LLD. The specific features are 1582 features calculated from 34 low-level descriptors (LLDs) in the Interspeech2010 challenge feature set.

[0090] Step 5.3: Read the text content data column of the CSV file generated in step 2.2 and input it into the Chinese pre-trained model Bert to obtain word vector features;

[0091] Step 6: Input the audio features and text features obtained in step 5 into the constructed text-driven multimodal classroom teacher speech behavior classification model, and construct a loss function to optimize the classification results.

[0092] In this embodiment, the specific implementation of step 6 includes the following sub-steps:

[0093] Step 6.1: As attached Figure 2 The MFCC audio features X that will be extracted are shown as follows MFCC and LLD Audio Feature X LLD And the Bert word vector feature X of the text T As input;

[0094] Specifically, for each input token, the Bert word vector model of text features will be converted into a d-dimensional vector, that is, t = [t1, t2, ..., t n ], where t irepresents the embedding vector corresponding to the i-th token, and n represents the number of tokens in the input sequence. The size of n affects the computational complexity of the BERT model. In the original implementation of BERT, the maximum length of the input sequence is set to 512 tokens. Assuming that the hidden layer size of BERT is h, the embedding vector of each token is obtained by adding the word embedding vector and the position embedding vector:

[0095] e i =w i,embedding +p i

[0096] Among them, e i represents the embedding vector of the i-th token, w i,embedding is the word embedding vector corresponding to the i-th token, p i Embedding vector for the position corresponding to the i-th token.

[0097] The Bert model uses multiple Transformer encoder layers to encode the input sequence. The output vector of the encoder corresponding to the lth layer is:

[0098] H l =[h 1,l ;h 2,l ;...;h n,l ]∈R n×h

[0099] Among them, h 1,l Represents the output vector of the i-th token corresponding to the l-th layer encoder. In the Bert model, each Transformer encoder consists of two sub-layers: a multi-head self-attention mechanism Attention and a feed-forward fully connected layer FFN.

[0100]

[0101] in, Represents the matrix corresponding to the query vector, key, and value respectively, and n is the hidden representation dimension in the model; d k is the dimension of the query vector / key vector, d v is the dimension of the value vector, and the softmax function is used to calculate the weights.

[0102] FFN(x)=max(0,xW1+b1)W2+b2

[0103] Where x represents the input vector, W1 and b1 represent the weight and bias of the first fully connected layer, W2 and b2 represent the weight and bias of the second fully connected layer, and max(0, ) represents the ReLU activation function.

[0104] Step 6.2: X MFCC Input attachment Figure 4 The Conformer convolution module shown in the figure obtains X M The structure of the Conformer convolution module is composed of layer normalization, point-by-point convolution, Glu activation function, 1D depth-separable convolution, batch normalization and Swish activation function. It processes the input through these modules to extract the corresponding features.

[0105] Layer Normalization normalizes the input to improve the stability of training. The formula is:

[0106]

[0107] Where x is the input, μ and σ are the mean and standard deviation of the input, respectively, and ∈ is a small constant to prevent the denominator from being zero. γ and β are learnable parameter vectors used to scale and translate the normalized feature vector. The solution for γ and β is typically obtained through gradient descent optimization during training.

[0108] Pointwise Convolution is an operation that performs convolution operations only in the channel dimension and is commonly used in convolutional neural networks. Its formula is:

[0109]

[0110] Among them, x i,j,k.c is the value of the i-th row, j-th column, k-th element, and c-th channel in the input tensor, w c and b are the weight and bias values ​​on the cth channel respectively, and C represents the number of channels of the input tensor. Output variable y i,j,k It is the result of the weighted sum of all channels of the input tensor at position (i, j, k) plus the bias value. Through the Pointwise Convolution operation, the nonlinear ability and expressiveness of the neural network can be increased without increasing the computational complexity.

[0111] The Glu Activation formula is as follows:

[0112]

[0113] in, Represents element-by-element multiplication, and σ is the sigmoid function. Both x and y are tensors, and their shapes must be the same.

[0114] The formula for 1D depthwise convolution is as follows:

[0115]

[0116] Among them, y i is the i-th element of the output sequence, w k is the weight of the kth position in the convolution kernel, x is the input sequence, K is the size of the convolution kernel, and s is the stride.

[0117] The formula for Batch Normalization is as follows:

[0118]

[0119] Where x represents the input data; E[x] represents the mean of x; Var[x] represents the variance of x; ∈ is a constant to avoid division by zero (usually 10 -5 ); γ and β are scale parameters and displacement parameters, respectively, which are learnable.

[0120] The Swish Activation (Swish activation function) formula is as follows, where β is a learnable parameter.

[0121] f(x)=x·sigmoid(β·x)

[0122] Step 6.3: X LLD Input attachment Figure 5 The BiLSTM module shown in the figure obtains X L ; BiLSTM is a network based on LSTM, which can process the input sequence forward and backward at the same time. Assume that the BiLSTM input is a sequence X of length n, and the BiLSTM output is a sequence H of length n, where each time t has a forward state and backward state The specific calculation is as follows:

[0123]

[0124]

[0125]

[0126] where x t represents the input of BiLSTM at time t, and Represent the outputs of the forward state and the backward state at time t-1 and t+1 respectively, and H represents the output vector at time t.

[0127] Step 6.4: Then we transform the text feature vector X T , audio feature vector XM and X L Input the convolution kernel size of 3 into the one-dimensional convolution Conv1D module to obtain the convolution text feature vector X t , audio feature vector X m and X l , the formula of the one-dimensional convolution Conv1D module is as follows:

[0128]

[0129] Among them, y i is the i-th element of the output sequence, w j is the jth weight of the convolution kernel, x i+j is the i+jth element of the input sequence, b is the bias term, and k represents the length or size of the convolution kernel, also known as the convolution window size. By performing calculations on different inputs and weights, the features of the input sequence can be effectively extracted.

[0130] Step 6.5: As attached Figure 6 The audio feature X m and X l and text features X t Perform text-centered attention fusion processing, the formula is as follows:

[0131]

[0132]

[0133] Among them, in Attention m→t middle, Similarly, in Attention l→t middle, W q 、W k and W v are the different weights of query, key and value respectively, L is the linear transformation of the input vector, d t is the input text feature X t The dimension, d k is the dimension of the query vector / key vector, and n is the hidden representation dimension in the model.

[0134] Step 6.6: The features fused by the attention mechanism are superimposed and input into the maximum pooling layer and the fully connected layer through the ReLU activation function to predict the classification results of the teacher's speech behavior in the classroom;

[0135] Add=Attention m→y +Attention l→t

[0136] x=f(Add)=max(0,Add)

[0137] y i,j,k =max p,q (x i+p,j+p,k )

[0138]

[0139] Among them, f(Add) represents the Relu activation function; y i,j,k It is the calculation of the maximum pooling layer, which is calculated as follows: i,j,k is the value of the kth channel in the i-th row, j-th column of the output matrix, x i+p,j+p,k It is the maximum value of the kth channel in the rectangular area with (i+p,j+q) as the upper left corner and (i+p+P,j+q+Q) as the lower right corner in the input matrix. P and Q are the window sizes of the pooling layer, and p and q refer to the offset of the coordinates of the upper left corner of the pooling selection area relative to the coordinates of the output matrix; is the result of the calculation of the fully connected layer, where W A is the weight matrix and b is the bias.

[0140] Step 6.7: The model loss function consists of three parts: the cross entropy loss function, the loss function of the single modality pair, and the loss function between the single modality and the fusion modality;

[0141] Cross Entropy Loss Function The calculation formula is as follows:

[0142]

[0143] Among them, m is the size of each batch, n is the number of categories, the true distribution is y, and the network output distribution is

[0144] Loss function for unimodal pairs The calculation formula is as follows:

[0145]

[0146]

[0147] Where I represents the lower bound of maximizing the MI (Mutual Information) of two modal pairs. In each modal pair, we regard one modality as x and the other modality as y. For example: I t,m The text mode is regarded as x, and the MFCC mode of the audio is regarded as y. The formula of input I is calculated; similarly, I t,lThe text mode is considered as x and the LLD mode of the audio is considered as y during calculation; m,l When calculating, the MFCC mode of the audio is regarded as x, and the LLD mode of the audio is regarded as y. p(y|x) is the true probability distribution, q(y|x) is the estimated probability distribution, and E p(x,y) and E p(y) is the expected value in the probability distribution; KL(·) represents the Kullback–Leibler divergence (KLD).

[0148] Loss function between single modality and fusion modality The calculation formula is as follows:

[0149]

[0150]

[0151] It is the noise contrast estimation. Its core idea is to compare the representation of one modality with other representations of the same modality in the same batch, and regard all other representations of the same modality in the same batch as negative samples. Among them, Z represents the result of multimodal fusion, that is, the vector obtained by superimposing the features after the attention mechanism fusion. Represents the negative samples generated by the single modality before fusion (obtained after con1D convolution). These negative samples are samples that the model does not want to recognize during training and are used to evaluate the performance of real samples. represents the positive sample generated by a single modality before fusion, i and j represent the i-th and j-th samples; E X Indicates expected value, exp() indicates exponential operation; and Represents the noise contrast estimation loss of text, MFCC and LLD respectively, in the calculation When X p =X t , similarly, in the calculation When X p =X m , in the calculation When X p =X l .

[0152] The final loss function is as follows:

[0153]

[0154] Among them, α and β are adjustable hyperparameter weights, which are usually values ​​less than 1.

[0155] Based on the same inventive concept, the present invention also designs a text-driven multimodal classroom teacher speech behavior analysis model structure, including:

[0156] The preprocessing module performs data preprocessing on the acquired classroom videos, namely extracting audio, transcribing text, and screening and retaining the teacher's audio segment;

[0157] The audio feature processing module extracts audio features from the preprocessed audio. Specifically, the audio feature MFCCs are input into the Conformer's convolution module to obtain the convolved audio feature MFCCs. The audio feature LLDs are input into the BiLSTM module to obtain the processed audio feature LLDs. The processed audio feature MFCCs and LLDs are then input into a one-dimensional convolution to obtain the final audio feature vector.

[0158] The text feature processing module extracts word vector features from the pre-processed text. This means the text is input into the Chinese pre-trained model Bert to obtain word vector features, and then the Bert word vector features are input into a one-dimensional convolution to obtain the final text feature vector.

[0159] The fusion module obtains the MFCC vector of the fused text feature by fusing the text vector and the audio feature MFCC through cross-modal attention, obtains the LLD vector of the fused text feature by fusing the text vector and the audio feature LLD through cross-modal attention, and then superimposes the MFCC vector of the fused text feature with the LLD vector of the fused text feature to finally obtain the multimodal feature representation vector of the fused text and audio.

[0160] The prediction module inputs the multimodal feature representation vector of the fused text and audio into the maximum pooling layer and the fully connected layer through the activation function to predict the final classification result.

[0161] Based on the same inventive concept, the present invention also provides an electronic device, comprising:

[0162] one or more processors;

[0163] A storage device is used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement multimodal classroom teacher speech behavior analysis based on text-driven main drive.

[0164] Based on the same inventive concept, the present invention also designs a computer-readable medium on which a computer program is stored. When the program is executed by a processor, it realizes multimodal classroom teacher speech behavior analysis based on text-driven main drive.

[0165] The method of the present invention is to analyze the speech behavior of classroom teachers through a multimodal learning model. First, classroom videos are obtained, audio is separated from the video, the audio is transcribed into text in the required format, and the teacher audio and the corresponding text are retained according to the recorded speaker and text content; secondly, features are extracted according to the data characteristics of different modes and a suitable representation method is selected. In the extraction of audio features, both fine-grained frame-level features and coarse-grained speech-level features are considered, and in the extraction of text features, the Chinese pre-training model Bert is selected to obtain word vector features; then a multimodal classroom teacher speech behavior classification model based on text-based driving is constructed. The model is centered on text and uses an attention mechanism to fuse text features and audio features across modalities. The fused features are sequentially entered into the maximum pooling layer and the fully connected layer through the activation function, and finally the prediction results of the teacher's speech behavior are obtained. In addition, during the training process of the model, the loss function is optimized. On the basis of the cross entropy loss function, the loss function of the single modal pair and the loss function between the single modal and the fusion modal are added to improve the accuracy of the model.

[0166] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A multimodal classroom teacher speech behavior analysis method based on text-driven, characterized by: The following steps are involved: Step 1: Get the classroom video and extract the audio from the video; Step 2: transcribe the audio to text, recording the pause time and speaker; Step 3: Divide the audio into audio segments according to the voice pauses recorded in step 2; Step 4: audio data processing, screening and retaining the teacher's audio segment and text; Step 5: extracting audio features and text features of the audio segment in step 4; In step 5, first, extract the Mel frequency cepstral coefficient MFCC feature X of the audio segment MFCC and acoustic low-level descriptor LLD feature X LLD ; Secondly, the text is input into the Chinese pre-training model Bert to obtain the word vector feature X T ; Step 6: Input the audio features and text features obtained in step 5 into the constructed text-driven multimodal classroom teacher speech behavior classification model, and construct a loss function to optimize the classification results; The specific implementation process of step 6 is as follows: First, the audio feature MFCC is input into the convolution module of Conformer to obtain the convolved audio feature MFCC, which is recorded as X M , input the audio feature LLD into the BiLSTM module to obtain the processed audio feature LLD, recorded as X L , then the audio feature MFCC obtained after convolution and the processed audio feature LLD are input into the one-dimensional convolution to obtain the final audio feature MFCC and LLD, recorded as the audio feature vector X m and X l ; Then the Bert word vector feature is input into the one-dimensional convolution to obtain the final text feature, which is recorded as the text feature vector X t ; Finally, the final audio features MFCC and LLD are fused with the text features in a text-centric attention fusion process. The features fused by the attention mechanism are superimposed and input into the maximum pooling layer and the fully connected layer through the activation function to predict the category of the teacher's speech behavior in the classroom. The final loss function consists of three parts: the cross-entropy loss function, the loss function of the single modality pair, and the loss function between the single modality and the fused modality.

2. The text-driven multimodal classroom teacher speech behavior analysis method according to claim 1 is characterized by: In step 2, call iFLYTEK's API to transcribe the audio into text and record it in a CSV file. The record format is start time, end time, speaker, and text content.

3. The text-driven multimodal classroom teacher speech behavior analysis method according to claim 2 is characterized by: In step 3, the audio in step 1 is divided into audio segments of unequal lengths according to the start and end times recorded in step 2.

4. The text-driven multimodal classroom teacher speech behavior analysis method according to claim 2 is characterized by: In step 4, the student's audio segment and text are removed based on the speaker recorded in step 2, and the noisy and silent audio segment and text are removed based on the recorded text content.

5. The text-driven multimodal classroom teacher speech behavior analysis method according to claim 1 is characterized by: The specific implementation formula for fusing the final audio features MFCC and LLD with the text features in a text-centric attention fusion process is as follows; Among them, in Attention m→t middle, Similarly, in Attention l→t middle, W q 、W k and W v are the different weights of query, key and value respectively, L is the linear transformation of the input vector, d t is the input text feature X t The dimension, d k is the dimension of the query vector / key vector, and n is the hidden representation dimension in the model; The specific implementation method of predicting the classification results of teachers' speech behaviors in the classroom is as follows: Add=Attention m→t +Attention l→t x=f(Add)=max(0,Add) and i,j,k =max p,q (x i+p,j+p,k ) Among them, f(Add) represents the Relu activation function; y i,j,k It is the calculation of the maximum pooling layer, which is calculated as follows: i,j,k is the value of the kth channel in the i-th row, j-th column of the output matrix, x i+p,j+p,k It is the maximum value of the kth channel in the rectangular area with (i+p,j+q) as the upper left corner and (i+p+P,j+q+Q) as the lower right corner in the input matrix. P and Q are the window sizes of the pooling layer. p and q refer to the offset of the upper left corner coordinates of the pooling selection area relative to the coordinates of the output matrix. is the result of the calculation of the fully connected layer, where W A is the weight matrix and b is the bias.

6. The text-driven multimodal classroom teacher speech behavior analysis method according to claim 1 is characterized by: Cross Entropy Loss Function The calculation formula is as follows: Among them, m is the size of each batch, n is the number of categories, the true distribution is y, and the network output distribution is Loss function for unimodal pairs The calculation formula is as follows: Where I represents the lower bound of MI that maximizes two modal pairs. In each modal pair, one modality is considered as x and the other modality is considered as y. t,m The text mode is regarded as x, and the MFCC mode of the audio is regarded as y. The formula of input I is calculated; similarly, I t,l The text mode is considered as x and the LLD mode of the audio is considered as y during calculation; m,l When calculating, the MFCC mode of the audio is regarded as x, and the LLD mode of the audio is regarded as y; p(y|x) is the true probability distribution, q(y|x) is the estimated probability distribution, E p(x,y) and E p(y) is the expected value in the probability distribution; KL(·) represents the KL divergence; Loss function between single modality and fusion modality The calculation formula is as follows: It is noise contrast estimation. Its core idea is to compare the representation of one modality with other representations of the modality in the same batch, and regard all other representations of the modality in the same batch as negative samples. Among them, Z represents the result of multimodal fusion, that is, the vector obtained by superimposing the features after the fusion of the attention mechanism. represents the negative samples generated by a single modality before fusion, represents the positive sample generated by a single modality before fusion, i and j represent the i-th and j-th samples; E X Indicates expected value, exp() indicates exponential operation; and Represents the noise contrast estimation loss of text, MFCC and LLD respectively, in the calculation When X p =X t , similarly, in the calculation When X p =X m , in the calculation When X p =X l ; The final loss function is as follows: Among them, α and β are adjustable hyperparameter weights.

7. A text-driven multimodal classroom teacher speech behavior analysis system, characterized by: Includes the following modules: Audio extraction module, used to obtain classroom videos and extract audio from them; Audio transcription module, used to transcribe audio into text, record speech pause time and speaker; An audio segmentation module, used to segment the audio into audio segments according to the recorded voice pauses; A processing module is used to process the audio data and filter and retain the teacher's audio segment and text; A feature extraction module extracts audio features and text features of the audio segments in the processing module; First, extract the Mel frequency cepstral coefficient MFCC feature X of the audio segment MFCC and acoustic low-level descriptor LLD feature X LLD ; Secondly, the text is input into the Chinese pre-training model Bert to obtain the word vector feature X T ; The prediction module is used to input the obtained audio and text features into the constructed text-driven multimodal classroom teacher speech behavior classification model and construct a loss function to optimize the classification results. The specific implementation process is as follows: First, the audio feature MFCC is input into the convolution module of Conformer to obtain the convolved audio feature MFCC, which is recorded as X M , input the audio feature LLD into the BiLSTM module to obtain the processed audio feature LLD, recorded as X L , then the audio feature MFCC obtained after convolution and the processed audio feature LLD are input into the one-dimensional convolution to obtain the final audio feature MFCC and LLD, recorded as the audio feature vector X m and X l ; Then the Bert word vector feature is input into the one-dimensional convolution to obtain the final text feature, which is recorded as the text feature vector X t ; Finally, the final audio features MFCC and LLD are fused with the text features in a text-centric attention fusion process. The features fused by the attention mechanism are superimposed and input into the maximum pooling layer and the fully connected layer through the activation function to predict the category of the teacher's speech behavior in the classroom. The final loss function consists of three parts: the cross-entropy loss function, the loss function of the single modality pair, and the loss function between the single modality and the fused modality.

8. A multimodal classroom teacher speech behavior analysis device based on text-driven, characterized by: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, causes the one or more processors to implement the method according to any one of claims 1 to 6.

Images