A speech recognition method and system applied to classroom assessment

Abstract

The application relates to a speech recognition method and system applied to classroom evaluation, which comprises the following steps: when receiving an evaluation instruction sent by a teacher terminal, receiving audio from the evaluation instruction and extracting audio features; inputting the audio features into a Conformer speech recognition model, wherein the Conformer speech recognition model comprises a Conformer coding module and a decoding module; after the audio features are subjected to re-feature extraction in the Conformer coding module, high-level features are obtained based on a Transformer decoding module in the decoding module and a decoding module of an improved Transformer decoder in the decoding module; the high-level features are decoded based on a self-attention mechanism of the Transformer decoding module in the decoding module and a cross-attention mechanism of the decoding module of the improved Transformer decoder in the decoding module, and corresponding recognition results are output. The application has the effect of more effectively improving the speech recognition model by using cold fusion.

Description

Technical Field

[0001] This application relates to the technical field of classroom assessment, and in particular to a speech recognition method and system for classroom assessment. Background Technology

[0002] In the current smart classroom environment, in order to improve the effectiveness and efficiency of language teaching, automatic speech recognition technology is used to assess students' language pronunciation and expression abilities.

[0003] However, existing speech recognition models, when handling language assessment in classroom environments, often focus only on optimizing the acoustic model, neglecting the impact of the language model on acoustic performance. Due to the variability of the classroom environment, the diversity of student pronunciation, and the complexity of language expression, a single acoustic model is insufficient to achieve ideal assessment results.

[0004] Furthermore, current methods for fusing acoustic and language models in speech recognition include shallow fusion, deep fusion, and cold fusion. Shallow fusion is the mainstream method, which, while improving recognition accuracy to some extent, relies on traditional N-gram statistical models and fails to fully utilize the potential of deep learning technology. During model training and fusion, the lack of optimization strategies and network structures tailored to specific classroom scenarios means that the performance of speech recognition models in classroom language evaluation still needs improvement. Summary of the Invention

[0005] To more effectively improve speech recognition models using cold fusion, this application provides a speech recognition method and system for classroom assessment.

[0006] The first objective of this invention is achieved through the following technical solution:

[0007] A speech recognition method for classroom assessment includes the following steps:

[0008] When an assessment instruction is received from the teacher's terminal, audio is received from the assessment instruction, and audio features are extracted.

[0009] Input audio features into the Conformer speech recognition model, which includes a Conformer encoding module and a decoding module;

[0010] After the audio features are extracted again in the Conformer encoding module, the high-level features are obtained based on the Transformer decoding module in the decoding module and the decoding module of the improved Transformer decoder in the decoding module.

[0011] Based on the self-attention mechanism of the Transformer decoding module in the decoding module and the cross-attention mechanism of the decoding module of the improved Transformer decoder in the decoding module, high-level features are decoded and the corresponding recognition results are output.

[0012] In a preferred embodiment of this application, the input audio features are fed into a Conformer speech recognition model, which includes a Conformer encoding module and a decoding module. This includes:

[0013] In the Conformer encoding module, based on residual connections, the connections from input to output are sequentially: first feedforward module, multi-head self-attention module, convolution module, second feedforward module, and layer norm module.

[0014] The multi-head self-attention module adopts a multi-head attention approach with residual connection, layer normalization, and positional encoding. The positional encoding multi-head attention adopts a sinusoidal encoding scheme for relative positions.

[0015] The convolution module employs normalization, activation functions, pointwise convolution, and one-dimensional depthwise separable convolution.

[0016] The first feedforward module and the second feedforward module include a linear transformation and a swish activation function.

[0017] In a preferred embodiment of this application, the input audio features are fed into a Conformer speech recognition model, which includes a Conformer encoding module and a decoding module. It also includes:

[0018] The decoding module includes a Transformer decoding module and a decoding module for an improved Transformer decoder;

[0019] In the Transformer decoding module, based on residual connections, the connections from input to output are sequentially: input sequence module, masked multi-head attention module, first standard addition module, multi-head attention module, second standard addition module, feedforward module, third standard addition module, linear transformation module, and Softmax function module.

[0020] In a preferred embodiment of this application, the Transformer decoding module, based on residual connections, comprises, in sequence from input to output, an input sequence module, a masked multi-head attention module, a first standard addition module, a multi-head attention module, a second standard addition module, a feedforward module, a third standard addition module, a linear transformation module, and a Softmax function module. This includes:

[0021] The Transformer decoding module includes a first self-attention module and a second self-attention module. The decoding module of the improved Transformer decoder includes a third self-attention module and a cross-attention module.

[0022] In the first self-attention module, the second self-attention module, and the third self-attention module, the connection from input to output is, in sequence, the input sequence module, the self-attention language module, the linear transformation module, and the Softmax function module;

[0023] In the self-attention language module, the connections from input to output are sequentially: masked multi-head attention module, fourth standard addition module, feedforward module, and fifth standard addition module.

[0024] Given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module feed each token in the sequence, after embedding and adding positional encoding, into M identical self-attention language modules. The high-level features are then fused with the output of the Conformer encoding module and the predicted sequence is output after linear transformation and the Softmax function.

[0025] In a preferred embodiment of this application, given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module feed each token in the sequence, after embedding and adding positional encoding, into M identical self-attention language modules. These modules are then fused with the output of the Conformer encoding module to obtain high-level features. After linear transformation and a Softmax function, the predicted sequence is output. This includes:

[0026] When the input sequence contains only 5 tokens, QK T The QK represents the attention score obtained from the input token. T Each line represents the attention score obtained by the i-th token and all other tokens except the i-th token;

[0027] In the Mask matrix, a green value of 1 indicates that no masking is needed, while a black value of 0 indicates that masking is used.

[0028] When the Mask matrix and the QK T After performing bitwise multiplication, we obtain Mask QK. T The Mask QK T Each row contains only the corresponding token and the information preceding the corresponding token, and then the data is weighted and summed using the Softmax function.

[0029] In a preferred embodiment of this application, given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module feed each token in the sequence, after embedding and adding positional encoding, into M identical self-attention language modules. These modules are then fused with the output of the Conformer encoding module to obtain high-level features. After linear transformation and a Softmax function, the predicted sequence is output. The application also includes:

[0030] The self-attention language module includes a self-attention mechanism and a neural network;

[0031] When using the self-attention mechanism, the weight of each element in the sequence with respect to all elements in the sequence is calculated, and a weighted sum is performed on each element in the sequence based on the weights between the elements.

[0032] SA: The weighted summation is divided into the following 4 steps:

[0033] SA1: Calculate the query vector, key vector, and value vector for each element in the sequence:

[0034] Given an input sequence vector X = [x1, x2, ..., x...] n Weight matrix W Q W K W V For each element x in the sequence i This requires performing linear transformations with the three weight matrices to obtain the final query vector, key vector, and value vector for each element.

[0035] q i =W Q x i

[0036] k i =W K x i

[0037] v i =W V x i

[0038] q i ,k i ,v i These represent the query vector, key vector, and value vector corresponding to the i-th element, respectively.

[0039] SA2: The attention score is obtained by performing a dot product of each query vector with each key vector.

[0040] For each element, the corresponding query vector q i Sum value vector k jFirst, take the dot product of the two to obtain the attention score, which represents the degree of attention element i pays to element j. Then, divide the attention score by a scaling factor to obtain the final attention score.

[0041]

[0042] SA3: The attention weights are obtained by using the Softmax function on the attention score of each element.

[0043] The attention scores s of element i and other elements are obtained. i =[s i1 ,s i2 ,…,s ij …,s in After that, the Softmax function is used to adjust the attention score s. i This is converted into attention weights, where each attention weight is a number between 0 and 1 that sums to 1, representing the degree of attention element i receives from each of the n elements.

[0044] w i =softmax(s i )=[w i1 ,w i2 ,…,w ij …,w in ]

[0045] SA4: The attention weights of elements are weighted and summed with the value vectors corresponding to each element to obtain the attention weight w for element i. i Next, each attention weight is multiplied by its corresponding value vector, and then the multiplied vectors are summed.

[0046]

[0047] z i Includes the required information for each element in the sequence;

[0048] For the entire sequence X, the corresponding self-attention calculation method is as follows:

[0049] Q = W Q X

[0050] K = W K X

[0051] V = W V X

[0052]

[0053] In a preferred embodiment of this application, given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module feed each token in the sequence, after embedding and adding positional encoding, into M identical self-attention language modules. These modules are then fused with the output of the Conformer encoding module to obtain high-level features. After linear transformation and a Softmax function, the predicted sequence is output. The application also includes:

[0054] The third self-attention module is trained using a cold fusion method based on a public text dataset and a speech model;

[0055] Given the input token sequence of the decoder, after passing through the third self-attention module, information extraction is performed once, and a high-level feature that extracts important information from the input token sequence is output.

[0056] Based on the cross-attention module, the token sequence after one information extraction is fused with the high-level features and then processed.

[0057] In a preferred embodiment of this application, the token sequence after one information extraction is fused with the high-level features based on the cross-attention module for processing. This includes:

[0058] The high-level features are used as input to the cross-attention query vector in the cross-attention module;

[0059] The cross-attention module includes a cross-attention mechanism;

[0060] The output of the token sequence after one information extraction is used as the input of the key vector and value vector. After N operations such as cross-attention mechanism, the recognition result of the next token sequence is output.

[0061] The second objective of this invention is achieved through the following technical solution:

[0062] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aforementioned speech recognition method for classroom assessment.

[0063] The third objective of this invention is achieved through the following technical solution:

[0064] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned speech recognition method for classroom assessment.

[0065] In summary, this application includes at least one of the following beneficial technical effects:

[0066] 1. Based on the Transformer decoding module and the improved Transformer decoder decoding module, the first step is to deeply fuse the token sequence after embedding and positional encoding with the output of the Transformer encoding module, thereby obtaining high-level features rich in contextual information. Using these high-level features as input, the Transformer decoding module and the improved Transformer decoder decoding module further perform N-1 iterations. Each iteration refines the features, allowing the model to gradually refine its understanding of the sequence through continuous information transmission and interaction. After N-1 iterations, the final output features are transformed into the probability distribution of the token at the next time step through linear transformation and the Softmax function. This step is crucial to the decoding process, enabling the model to predict the most likely next token given the context.

[0067] 2. High-level features are cleverly used as input to the cross-attention query vector in the cross-attention module. This design significantly improves the information extraction and integration capabilities of the decoding module in the improved Transformer decoder when processing sequence data. The core of the cross-attention module is the cross-attention mechanism, which can effectively interact and fuse features from different sources, thereby uncovering deeper semantic information.

[0068] Specifically, the improved Transformer decoder's decoding module first uses the token sequence output after one information extraction as input to the key and value vectors in the cross-attention module. This step ensures that the improved Transformer decoder's decoding module can fully utilize the extracted sequence information in subsequent processing, providing rich context for the query vector. In this way, the cross-attention module can more accurately locate and focus on key elements in the sequence, thereby improving recognition accuracy.

[0069] Subsequently, the improved Transformer decoder's decoding module deeply mines and refines the information in the sequence through N iterations of cross-attention mechanism. Each iteration is a meticulous processing of the sequence features, enabling the model to gradually optimize its understanding of each token in the sequence through continuous attention weighting. This recursive processing method not only enhances the model's grasp of the global structure of the sequence but also improves its ability to recognize local details.

[0070] Finally, after N complex calculations including cross-attention, the improved Transformer decoder's decoding module outputs the recognition result for the next token sequence. This result is based on a comprehensive consideration of all relevant information, including the sequence's historical information, the current token's contextual information, and external information fused in through the cross-attention mechanism. Therefore, this recognition result has high reliability and accuracy, providing strong support for sequence processing tasks such as speech recognition. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating an embodiment of a speech recognition method for classroom assessment according to this application.

[0072] Figure 2 This is a flowchart of S20 in an embodiment of a speech recognition method for classroom assessment according to this application;

[0073] Figure 3 This is another implementation flowchart of S20 and an implementation flowchart of S206 in Embodiment 1 of a speech recognition method for classroom assessment in this application;

[0074] Figure 4 This is a flowchart of the first implementation of S4 in an embodiment of a speech recognition method for classroom assessment in this application;

[0075] Figure 5 This is a second implementation flowchart of S4 in Embodiment 1 of a speech recognition method for classroom assessment in this application;

[0076] Figure 6 This is a flowchart of the third implementation of S4 in Embodiment 1 and an implementation flowchart of S48 in Embodiment 2 of the speech recognition method for classroom assessment in this application.

[0077] Figure 7 This is a schematic block diagram of a computer device according to this application. Detailed Implementation

[0078] The following is in conjunction with the appendix Figure 1-6 This application will be described in further detail.

[0079] In Example 1, as Figure 1 As shown, this application discloses a speech recognition method for classroom assessment, which specifically includes the following steps:

[0080] S10: When an assessment instruction is received from the teacher's terminal, audio is received from the assessment instruction, and audio features are extracted;

[0081] In this embodiment, the teacher terminal refers to a PC set up in the classroom or a mobile terminal carried by the teacher for operation. The assessment instruction refers to the instruction issued by the teacher through the teacher terminal for collecting, processing, and assessing audio. Audio features refer to a series of parameters extracted from the captured audio, representing various attributes of the audio. Audio features include pitch features, spectrum features, and energy features.

[0082] Specifically, when teachers assess students' language, such as Mandarin, they issue instructions through a PC logged in in the classroom or a mobile app carried by the teacher to collect, process, and assess audio, extracting the audio's pitch, spectral, and energy characteristics.

[0083] S20: Input audio features into the Conformer speech recognition model. The Conformer speech recognition model includes a Conformer encoding module and a decoding module.

[0084] In this embodiment, audio features are input into the Conformer speech recognition model. The Conformer encoding module generates a portion of the audio feature sequence including contextual information by processing the input audio feature sequence. The decoding module generates a portion of the output sequence based on the feature representation provided by the Conformer encoding module and predicts the next element in the sequence.

[0085] S30: After the audio features are extracted again in the Conformer encoding module, the high-level features are obtained based on the Transformer decoding module in the decoding module and the decoding module of the improved Transformer decoder in the decoding module.

[0086] In this embodiment, high-level features refer to the audio features that have undergone in-depth processing by the Transformer decoding module and the improved Transformer decoder decoding module within the decoding module. These features are further abstracted and transformed to form feature representations that better represent the high-level semantic and structural information of the audio signal. These high-level features can then be used by the decoding module to generate the final recognition result, such as a text sequence.

[0087] S40: Based on the self-attention mechanism of the Transformer decoding module in the decoding module and the cross-attention mechanism of the decoding module of the improved Transformer decoder in the decoding module, high-level features are decoded and the corresponding recognition results are output.

[0088] In this embodiment, decoding refers to converting high-level features into readable text.

[0089] Reference Figure 2 S20. Includes:

[0090] S201: In the Conformer encoding module, based on residual connections, the connections from input to output are sequentially: first feedforward module, multi-head self-attention module, convolution module, second feedforward module, and layer norm module.

[0091] S202: The multi-head self-attention module adopts a multi-head attention method with residual connection, layer normalization and position encoding. The position-encoded multi-head attention adopts a sinusoidal encoding scheme for relative position.

[0092] S203: The convolution module uses normalization, activation function, pointwise convolution and one-dimensional depthwise separable convolution.

[0093] S204: The first feedforward module and the second feedforward module include a linear transformation and a swish activation function.

[0094] In this embodiment, the multi-head self-attention module significantly improves its ability to process sequence data by integrating residual connections, layer normalization, and positional encoding. The sinusoidal encoding scheme used in the positional encoding effectively generalizes to the length of the input sequence, ensuring good performance of the multi-head attention module when processing sequences of different lengths. The convolutional module employs normalization, activation functions, pointwise convolution, and one-dimensional depthwise separable convolution techniques. These two convolutional network designs aim to precisely capture local features in the input sequence, enabling the Conformer encoding module to identify subtle changes and key patterns in the speech signal. The application of normalization allows for more stable and efficient training of the network. The first and second feedforward modules mainly consist of linear layers and swish activation functions; this combination greatly enhances the model's ability to express features. Linear layers ensure feature transfer and transformation, while the swish activation function, due to its smooth and non-monotonic characteristics, better preserves the nonlinear information of the features. Overall, the synergistic effect of the multi-head self-attention module, convolution module, first feedforward module, and second feedforward module enables the entire network to not only accurately capture local and global information when processing sequence tasks such as speech recognition, but also to efficiently learn the deep structural and semantic information in the data.

[0095] Reference Figure 3 S20. Also includes:

[0096] S205: The decoding module includes a Transformer decoding module and a decoding module for an improved Transformer decoder;

[0097] S206: In the Transformer decoding module, based on residual connections, the connections from input to output are sequentially: input sequence module, masking multi-head attention module, first standard addition module, multi-head attention module, second standard addition module, feedforward module, third standard addition module, linear transformation module, and Softmax function module.

[0098] In this embodiment, the decoding module based on the Transformer decoding module and the improved Transformer decoder first performs deep fusion between the token sequence after embedding and positional encoding and the output of the Transformer encoding module, thereby obtaining high-level features rich in contextual information. Using these high-level features as input, the Transformer decoding module and the improved Transformer decoder further perform N-1 iterations. Each iteration refines the features, allowing the model to gradually refine its understanding of the sequence through continuous information transmission and interaction. After N-1 iterations, the final output features are transformed into the probability distribution of the token at the next time step through linear transformation and the Softmax function. This step is crucial to the decoding process, enabling the model to predict the most likely next token given the context.

[0099] Reference Figure 3 S206. Includes:

[0100] S1: The Transformer decoding module includes a first self-attention module and a second self-attention module. The decoding module of the improved Transformer decoder includes a third self-attention module and a cross-attention module.

[0101] S2: In the first self-attention module, the second self-attention module, and the third self-attention module, the connection from input to output is, in order, the input sequence module, the self-attention language module, the linear transformation module, and the Softmax function module;

[0102] S3: In the self-attention language module, the connection from input to output is sequentially: masked multi-head attention module, fourth standard addition module, feedforward module, and fifth standard addition module;

[0103] S4: Given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence, after embedding and adding position encoding, into M identical self-attention language modules, fuse it with the output of the Conformer encoding module to obtain high-level features, and output the predicted sequence after linear transformation and Softmax function.

[0104] In this embodiment, when the input sequence has only 5 tokens, QK T QK represents the attention score obtained from the input token. T Each row represents the attention score obtained by the i-th token and all other tokens; in the Mask matrix, green values ​​of 1 indicate no occlusion, and black values ​​of 0 indicate occlusion; when the Mask matrix and QK T After performing bitwise multiplication, we obtain Mask QK. T Mask QK T Each row contains only the corresponding token and the information preceding the corresponding token, and then the data is weighted and summed using the Softmax function.

[0105] Reference Figure 4 S4. Includes:

[0106] S41: When the input sequence contains only 5 tokens, QK T The QK represents the attention score obtained from the input token. T Each line represents the attention score obtained by the i-th token and all other tokens except the i-th token;

[0107] S42: In the Mask matrix, green values ​​of 1 indicate that no occlusion is needed, and black values ​​of 0 indicate that occlusion is needed.

[0108] S43: When the Mask matrix and the QK T After performing bitwise multiplication, we obtain Mask QK. T The Mask QK T Each row contains only the corresponding token and the information preceding the corresponding token, and then the data is weighted and summed using the Softmax function.

[0109] In this embodiment, by introducing a Mask operation, the output of each time step in the sequence is independently calculated based on itself and the input information of the previous time steps in the output of each module. This autoregressive characteristic is particularly evident in the Transformer decoding module. The Transformer decoding module works by predicting the next token in the sequence based on known tokens. This sequentially dependent prediction method allows the model to better understand the coherence and contextual relationships of language. To achieve this goal, this application employs the following... Figure 4The structured training data shown is carefully designed to ensure that during training, the model learns how to predict future tokens based on current and previous contextual information. Training using the cross-entropy loss function first ensures that the model strictly adheres to the time series order during prediction, unaffected by future information. Second, as a common method for measuring prediction error in classification problems, the cross-entropy loss function effectively guides the model to learn a more accurate token distribution, thereby improving decoding accuracy. Finally, after this training, the model exhibits higher performance in sequence generation tasks such as speech recognition and machine translation, not only generating fluent sequences but also handling long-distance dependencies and complex language structures well, providing strong technical support for practical applications.

[0110] Reference Figure 5 S4. Also includes:

[0111] S44: The self-attention language module includes a self-attention mechanism and a neural network;

[0112] S45: When using the self-attention mechanism, calculate the weight of each element in the sequence relative to all elements in the sequence, and perform a weighted summation of each element in the sequence based on the weights between the elements;

[0113] SA: The weighted summation is divided into the following 4 steps:

[0114] SA1: Calculate the query vector, key vector, and value vector for each element in the sequence:

[0115] Given an input sequence vector X = [x1, x2, ..., x...] n Weight matrix W Q W K W V For each element x in the sequence i This requires performing linear transformations with the three weight matrices to obtain the final query vector, key vector, and value vector for each element.

[0116] q i =W Q x i

[0117] k i =W K x i

[0118] v i =W V x i

[0119] q i ,ki ,v i These represent the query vector, key vector, and value vector corresponding to the i-th element, respectively.

[0120] SA2: The attention score is obtained by performing a dot product of each query vector with each key vector.

[0121] For each element, the corresponding query vector q i Sum value vector k j First, take the dot product of the two to obtain the attention score, which represents the degree of attention element i pays to element j. Then, divide the attention score by a scaling factor to obtain the final attention score.

[0122]

[0123] SA3: The attention weights are obtained by using the Softmax function on the attention score of each element.

[0124] The attention scores s of element i and other elements are obtained. i =[s i1 ,s i2 ,…,s ij …,s in After that, the Softmax function is used to adjust the attention score s. i This is converted into attention weights, where each attention weight is a number between 0 and 1 that sums to 1, representing the degree of attention element i receives from each of the n elements.

[0125] w i =softmax(s i )=[w i1 ,w i2 ,…,w ij …,w in ]

[0126] SA4: The attention weights of elements are weighted and summed with the value vectors corresponding to each element to obtain the attention weight w for element i. i Next, each attention weight is multiplied by its corresponding value vector, and then the multiplied vectors are summed.

[0127]

[0128] z i Includes the required information for each element in the sequence;

[0129] For the entire sequence X, the corresponding self-attention calculation method is as follows:

[0130] Q = W Q X

[0131] K = W K X

[0132] V = W V X

[0133]

[0134] In this embodiment, the self-attention mechanism is unique in that it comprehensively considers the relationships between each element and all other elements in the sequence when processing sequential data. Through self-attention, the model can examine each element in the sequence from multiple perspectives, as if it had multiple "focuses." This capability makes the model more comprehensive and accurate in extracting sequence information; it can not only capture explicit information in the sequence but also discern important information hidden in the subtle relationships between elements. Therefore, the self-attention mechanism greatly improves the efficiency and quality of information extraction when processing sequential data.

[0135] This superior information extraction capability directly leads to a significant improvement in the model's accuracy when processing sequential data. Whether it's word meaning understanding and sentence structure analysis in natural language processing, or phoneme identification and intonation perception in speech recognition, the self-attention mechanism helps the model capture key information more accurately, thereby making more precise judgments during decoding and prediction.

[0136] Furthermore, the introduction of self-attention significantly enhances the model's ability to handle long-distance dependencies. In traditional sequence processing models, long-distance dependencies are often a significant challenge, but self-attention effectively solves this problem by directly establishing a connection between any two elements in the sequence. This allows the model to maintain higher performance and lower error rates when handling tasks such as long sentence understanding and long speech recognition.

[0137] Reference Figure 6 S4. Also includes:

[0138] S46: The third self-attention module is trained using a cold fusion method based on a public text dataset and a speech model;

[0139] S47: Given the input token sequence of the decoder, after passing through the third self-attention module, perform one information extraction and output a high-level feature that extracts important information from the input token sequence;

[0140] S48: Based on the cross-attention module, the token sequence after one information extraction is fused with the high-level features and processed.

[0141] In this embodiment, high-level features are cleverly used as input to the cross-attention query vector in the cross-attention module. This design significantly enhances the information extraction and integration capabilities of the decoding module of the improved Transformer decoder when processing sequence data. The core of the cross-attention module is the cross-attention mechanism, which can effectively interact and fuse features from different sources, thereby uncovering deeper semantic information.

[0142] Specifically, the improved Transformer decoder's decoding module first uses the token sequence output after one information extraction as input to the key and value vectors in the cross-attention module. This step ensures that the improved Transformer decoder's decoding module can fully utilize the extracted sequence information in subsequent processing, providing rich context for the query vector. In this way, the cross-attention module can more accurately locate and focus on key elements in the sequence, thereby improving recognition accuracy.

[0143] Subsequently, the improved Transformer decoder's decoding module deeply mines and refines the information in the sequence through N iterations of cross-attention mechanism. Each iteration is a meticulous processing of the sequence features, enabling the model to gradually optimize its understanding of each token in the sequence through continuous attention weighting. This recursive processing method not only enhances the model's grasp of the global structure of the sequence but also improves its ability to recognize local details.

[0144] Finally, after N complex calculations including cross-attention, the improved Transformer decoder's decoding module outputs the recognition result for the next token sequence. This result is based on a comprehensive consideration of all relevant information, including the sequence's historical information, the current token's contextual information, and external information fused in through the cross-attention mechanism. Therefore, this recognition result has high reliability and accuracy, providing strong support for sequence processing tasks such as speech recognition.

[0145] In Example 2, refer to Figure 6 S48. Includes:

[0146] S481: Use the high-level features as input to the cross-attention query vector in the cross-attention module;

[0147] S482: The cross-attention module includes a cross-attention mechanism;

[0148] S483: The output of the token sequence after one information extraction is used as the input of the key vector and value vector. After N operations such as cross-attention mechanism, the recognition result of the next token sequence is output.

[0149] In this embodiment, to further improve the efficiency of input token sequence information extraction, this application trains the speech recognition model using a cold fusion method of language models. Based on this, this application has carefully designed a method such as... Figure 6 The improved decoder shown cleverly combines self-attention and cross-attention mechanisms to achieve efficient utilization and deep understanding of sequence information.

[0150] The application of self-attention mechanism in the decoder enables the decoding module of the improved Transformer decoder to automatically learn the intrinsic relationships between each token in the sequence, thereby capturing long-distance dependencies and complex semantic structures within the sequence. This mechanism not only enhances the model's grasp of global information about the sequence but also increases its attention to local features, providing rich contextual information for subsequent decoding processes.

[0151] Meanwhile, the cross-attention mechanism further enhances the model's ability to interact with information across different feature dimensions. By cross-fusing the information of the input token sequence with the high-level features output by the encoder, the decoder can more accurately locate and identify key elements in the sequence, thereby generating a more accurate sequence that conforms to language logic during the decoding process.

[0152] In Example 3, referring to Tables 1-2, two types of data were used in the experiment. For speech recognition, Aishell-1 was used as the experimental dataset. This dataset is an open-source Mandarin Chinese speech dataset with a total duration of 178 hours. It was recorded by 400 speakers from different accent regions of China in a quiet indoor environment, with an audio sampling rate of 16kHz. The recorded texts cover five major areas: finance, sports, technology, entertainment, and current affairs news, and the transcription and annotation accuracy is over 95%. For the language model, in order to use the same similar text dataset, the experiment additionally used Aishell-2 text data, with a duration of 1000 hours. The recorded texts cover 12 areas, including wake words, voice control words, smart home, autonomous driving, and industrial production, and the database text accuracy is over 96%. The experiment divided the Aishell-1 dataset into training, validation, and test sets. The training set contains 120,098 data points totaling 150 hours, the validation set contains 14,326 data points totaling 18 hours, and the test set contains 7,176 data points totaling 10 hours. The text data from Aishell-2 was used to pre-train the language model. The data included text data from the validation and test sets of Aishell-1. After removing duplicates, there were a total of 762,354 data entries.

[0153] Experimental procedure:

[0154] First, the input audio and transcribed text tags for the model were processed. The Conformer encoding module used 80-dimensional Fbank features with a frame length of 25ms and a frame shift of 10ms as input, processed using Global Cepstral Mean Variance Normalization (Global-CMVN), and applied spectral augmentation techniques to enhance the model's robustness. Additionally, audio velocity perturbations of 0.9 and 1.1 were used to augment the training dataset. The Aishell-2 training dataset contains a total of 5156 transcribed text characters, with additional whitespace characters added. <blank>Unknown symbol <unk>and start and end symbols <sos eos>Since the model in this invention uses characters as the smallest recognition unit, the entire model dictionary contains a total of 5159 tokens.

[0155] Next, the model parameters were configured. In the Conformer encoding module, convolutional subsampling was used on the spectrally enhanced features to reduce their length to 1 / 4 of the original. The number of units in the linear layers was 2048, and a total of 12 Conformer modules were used. For each Conformer module, the output dimension was 256, the number of hidden layers in the feedforward module was 2048, the number of heads in the multi-head self-attention module was 4, the number of hidden layer units was 256, the depthwise-convolution kernel size in the convolutional module was 15, and dropout was set to 0.1. In the Conformer decoder, the token embedding vector dimension was 256, the number of decoding blocks was set to 8, the number of hidden layers in the multi-head attention module in the decoding block was set to 256, and the number of hidden layers in the feedforward module was 2048. In the improved decoder, except that the number of self-attention layers M and the number of cross-attention layers N were not set, the other parameters were the same as those in the Conformer decoder.

[0156] Finally, the hyperparameters of the model were configured. In the experiment of pre-training the language model, the batch size was set to 70, and an Adam optimizer with a learning rate of 0.002 was used for a total of 100 epochs. Then, the pre-trained language model was used to train the speech recognition model, with the batch size set to 16 and an Adam optimizer with a learning rate of 0.001, for a total of 240 epochs. After training, the 20 models with the smallest loss in the validation set were selected, and their parameters were averaged to obtain the final recognition model. This model was then used to decode the test audio to obtain the final recognition result.

[0157] Evaluation indicators:

[0158] The experiment used the Character Error Rate (CER), a commonly used metric in Chinese speech recognition, as the evaluation index. A lower CER indicates better recognition performance. The calculation formula is as follows:

[0159]

[0160] Where N represents the total number of characters in the test data, S (substitution) represents the number of characters replaced in the recognition result, D (deletion) represents the number of characters deleted in the recognition result, and I (insertion) represents the number of characters inserted in the recognition result.

[0161] To measure the performance of the self-attention language model, the experiment used perplexity (PPL) as the evaluation metric; the lower the perplexity, the better the language model's performance. The calculation formula is as follows:

[0162] H(P,Q)=-∑ x P(x)logQ(x)

[0163] PPL=2 H(P,Q)

[0164] Where H(P,Q) is the cross-entropy loss of the self-attention language model.

[0165] In this embodiment, the experimental results and analysis are as follows: First, the language model in the improved decoder was not pre-trained, and the recognition performance of the model was tested using different numbers of attention mechanism layers. The experimental results are shown in Table 1.

[0166] Table 1. Experimental results of different attention layers on Aishell-1

[0167]

[0168]

[0169] Then, to explore the role of the self-attention language model and its impact on the final speech recognition results, different training datasets were selected for pre-training of the language model. The experimental results are shown in Table 2.

[0170] Table 2 Experimental results for different datasets

[0171]

[0172] As shown in Table 2, for the self-attention language model, the test set PPL gradually decreases with the increase of the number of self-attention layers M and the text training dataset. This indicates that the model learns more information as the number of model parameters and favorable data increases. For the speech recognition model, when pre-trained using only the Aishell-1 text dataset, its CER is the same as or higher than that of the unpre-trained model. When pre-trained using the Aishell-2 text data, additional text-related domain information is introduced. When M=8 and N=2, its CER decreases by 1.5% compared to the unpre-trained model.

[0173] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0174] For specific limitations regarding a speech recognition method and system applied to classroom assessment, please refer to the limitations of the speech recognition method applied to classroom assessment mentioned above, which will not be repeated here. The various modules in the aforementioned speech recognition system for classroom assessment can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0175] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores student information and ranking information. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a speech recognition method for classroom assessment.

[0176] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a speech recognition method for classroom assessment.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.< / sos> < / unk> < / blank>

Claims

1. A speech recognition method for classroom assessment, characterized in that: Including the following steps: When an assessment instruction is received from the teacher's terminal, audio is received from the assessment instruction, and audio features are extracted. Teacher terminal refers to the PC set up in the classroom or the mobile terminal carried by the teacher for operation. Assessment instructions refer to the instructions issued by the teacher through the teacher terminal for collecting, processing and assessing audio. Audio features refer to a series of parameters taken from the captured audio to represent various attributes of the audio. Audio features include pitch features, spectrum features and energy features. Input audio features into the Conformer speech recognition model, which includes a Conformer encoding module and a decoding module; The Conformer encoding module is responsible for processing the input audio feature sequence to generate a portion of the audio feature sequence that includes contextual information. The decoding module is responsible for generating a portion of the output sequence based on the feature representation provided by the Conformer encoding module and predicting the next element in the sequence. In the Conformer encoding module, based on residual connections, the connections from input to output are sequentially: first feedforward module, multi-head self-attention module, convolution module, second feedforward module, and layer norm module. The multi-head self-attention module adopts a multi-head attention approach with residual connection, layer normalization, and positional encoding. The positional encoding multi-head attention adopts a sinusoidal encoding scheme for relative positions. The convolution module employs normalization, activation functions, pointwise convolution, and one-dimensional depthwise separable convolution. The first feedforward module and the second feedforward module include a linear transformation and a swish activation function; The decoding module includes a Transformer decoding module and a decoding module for an improved Transformer decoder; In the Transformer decoding module, based on residual connections, the connections from input to output are sequentially: input sequence module, masking multi-head attention module, first standard addition module, multi-head attention module, second standard addition module, feedforward module, third standard addition module, linear transformation module, and Softmax function module. The decoding module based on the Transformer decoding module and the improved Transformer decoder first performs deep fusion between the token sequence after embedding and positional encoding and the output of the Transformer encoding module to obtain high-level features rich in contextual information. By using the obtained high-level features as input, the decoding module of the Transformer decoding module and the improved Transformer decoder further perform N-1 iterations. After N-1 iterations, the final output features are transformed into the probability distribution of the token in the next time step through linear transformation and Softmax function. After the audio features are extracted again in the Conformer encoding module, the high-level features are obtained based on the Transformer decoding module in the decoding module and the decoding module of the improved Transformer decoder in the decoding module. High-level features refer to the deep processing of audio features by the Transformer decoding module and the improved Transformer decoder in the decoding module. The audio features are further abstracted and transformed to form feature representations that can better represent the high-level semantic and structural information of the audio signal. High-level features can then be used by the decoding module to generate the final recognition result; Based on the self-attention mechanism of the Transformer decoding module in the decoding module and the cross-attention mechanism of the decoding module of the improved Transformer decoder in the decoding module, high-level features are decoded and the corresponding recognition results are output. Decoding refers to converting high-level features into readable text.

2. The speech recognition method for classroom assessment according to claim 1, characterized in that: In the Transformer decoding module, based on residual connections, the connections from input to output are sequentially: input sequence module, masking multi-head attention module, first standard addition module, multi-head attention module, second standard addition module, feedforward module, third standard addition module, linear transformation module, and Softmax function module; including: The Transformer decoding module includes a first self-attention module and a second self-attention module. The decoding module of the improved Transformer decoder includes a third self-attention module and a cross-attention module. In the first self-attention module, the second self-attention module, and the third self-attention module, the connection from input to output is, in sequence, the input sequence module, the self-attention language module, the linear transformation module, and the Softmax function module; In the self-attention language module, the connections from input to output are sequentially: masked multi-head attention module, fourth standard addition module, feedforward module, and fifth standard addition module; Given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module feed each token in the sequence, after embedding and adding positional encoding, into M identical self-attention language modules. The high-level features are then fused with the output of the Conformer encoding module and the predicted sequence is output after linear transformation and the Softmax function.

3. The speech recognition method for classroom assessment according to claim 2, characterized in that: Given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module feed each token in the sequence, after embedding and adding positional encoding, into M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features. The output is the predicted sequence after linear transformation and the Softmax function; it also includes: The self-attention language module includes a self-attention mechanism and a neural network; When using the self-attention mechanism, the weight of each element in the sequence with respect to all elements in the sequence is calculated, and a weighted sum is performed on each element in the sequence based on the weights between the elements. SA: The weighted summation is divided into the following 4 steps: SA1: Calculate the query vector, key vector, and value vector for each element in the sequence: Given an input sequence vector Weight matrix For each element in the sequence This requires performing linear transformations with the three weight matrices to obtain the final query vector, key vector, and value vector for each element. ； ； ； They represent the first Each element corresponds to a query vector, a key vector, and a value vector; SA2: The attention score is obtained by performing a dot product of each query vector with each key vector. For the query vector corresponding to each element Sum value vector First, take the dot product of the two to obtain the attention score. This score represents the element. For elements The level of attention is calculated, and then the attention score is divided by a scaling factor to obtain the final attention score: ； SA3: The attention weights are obtained by using the Softmax function on the attention score of each element. After obtaining the element Attention score compared to other elements Then, the Softmax function is used to adjust the attention score. This is converted into attention weights, where the attention weights are numbers between 0 and 1 that sum to 1, representing elements. exist The degree of attention given to each element within a range of elements, i.e. ； SA4: The attention weights of the elements are weighted and summed with the value vector corresponding to each element. This results in the element... Attention weights Next, each attention weight is multiplied by its corresponding value vector, and then the multiplied vectors are summed. ； Includes the required information for each element in the sequence; For the entire sequence The corresponding self-attention calculation method is as follows: ； ； ； 。 4. The speech recognition method for classroom assessment according to claim 3, characterized in that: Given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module feed each token in the sequence, after embedding and adding positional encoding, into M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features. The output, after linear transformation and the Softmax function, is the predicted sequence, including: When the input sequence contains only 5 tokens The attention score obtained from the input token is represented by the following. Each line represents the attention score obtained by the i-th token and all other tokens except the i-th token; When the Mask matrix and the After performing bitwise multiplication, we obtain the Mask. The Mask Each row contains only the corresponding token and the information preceding the corresponding token, and then the data is weighted and summed using the Softmax function.

5. The speech recognition method for classroom assessment according to claim 4, characterized in that: Given an input sequence, the first self-attention module, the second self-attention module, and the third self-attention module feed each token in the sequence, after embedding and adding positional encoding, into M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features. The output is the predicted sequence after linear transformation and the Softmax function; it also includes: The third self-attention module is trained using a cold fusion method based on a public text dataset and a speech model; Given the input token sequence to the decoder, after passing through the third self-attention module, information extraction is performed once, and a high-level feature that extracts important information from the input token sequence is output. Based on the cross-attention module, the token sequence after one information extraction is fused with the high-level features and then processed.

6. The speech recognition method for classroom assessment according to claim 5, characterized in that: The cross-attention module-based process fuses the token sequence after one information extraction with the high-level features for further processing; including: The high-level features are used as input to the cross-attention query vector in the cross-attention module; The cross-attention module includes a cross-attention mechanism; The output of the token sequence after one information extraction is used as the input of the key vector and value vector. After N operations such as cross-attention mechanism, the recognition result of the next token sequence is output.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a speech recognition method for classroom assessment as described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a speech recognition method for classroom assessment as described in any one of claims 1-6.

Images