Speech recognition method and system applied to classroom evaluation

Abstract

The invention relates to a speech recognition method and system applied to classroom evaluation, and the method comprises the steps: receiving an audio from an evaluation instruction when the evaluation instruction sent by a teacher terminal is received, and extracting the audio features; the audio features are input to a Conform voice recognition model, wherein the Conform voice recognition model comprises a Conform coding module and a decoding module; after the audio features are subjected to feature extraction again in the Conformer encoding module, high-level features are obtained based on a Transformer decoding module in the decoding module and a decoding module of an improved Transformer decoder in the decoding module; and decoding the high-level features based on a self-attention mechanism of a Transform decoding module in the decoding module and a cross attention mechanism of a decoding module of an improved Transform decoder in the decoding module, and outputting a corresponding recognition result. The method and the device have the effect of improving the voice recognition model by using cold fusion more effectively.

Description

Technical Field

[0001] The present application relates to the technical field of classroom assessment, and in particular to a speech recognition method and system applied to classroom assessment. Background Art

[0002] In the current smart classroom environment, in order to improve the effectiveness and efficiency of language teaching, automatic speech recognition technology is used to evaluate students' pronunciation and language expression ability.

[0003] However, when dealing with language evaluation in classroom environments, existing speech recognition models often only focus on optimizing the acoustic model, while ignoring the impact of the language model on the acoustic effect. Due to the variability of the classroom environment, the diversity of students' pronunciation and the complexity of language expression, a single acoustic model is difficult to achieve ideal evaluation results.

[0004] In addition, the current methods for integrating acoustic models and language models in speech recognition models include shallow fusion, deep fusion, and cold fusion. The mainstream fusion method is shallow fusion. Although it improves the recognition accuracy to a certain extent, it relies on the traditional N-gram statistical model and fails to fully utilize the potential of deep learning technology. During the model training and fusion process, due to the lack of optimization strategies and network structures for specific classroom scenarios, the performance of speech recognition models in classroom evaluation language still needs to be improved, so it needs to be improved. Summary of the invention

[0005] In order to use cold fusion to improve the speech recognition model more effectively, the present application provides a speech recognition method and system for classroom assessment.

[0006] The first object of the invention of this application is achieved through the following technical solutions:

[0007] A speech recognition method for classroom assessment includes the following steps:

[0008] When receiving an evaluation instruction from the teacher terminal, receiving audio from the evaluation instruction and extracting audio features;

[0009] Input audio features to the Conformer speech recognition model, which includes a Conformer encoding module and a decoding module;

[0010] After the audio features are extracted again in the Conformer encoding module, high-level features are obtained based on the Transformer decoding module in the decoding module and the decoding module of the improved Transformer decoder in the decoding module;

[0011] Based on the self-attention mechanism of the Transformer decoding module in the decoding module and the cross-attention mechanism of the decoding module of the improved Transformer decoder in the decoding module, the high-level features are decoded and the corresponding recognition results are output.

[0012] In a preferred embodiment of the present application, the input audio features are input to a Conformer speech recognition model, and the Conformer speech recognition model includes a Conformer encoding module and a decoding module. Including:

[0013] In the Conformer encoding module, based on the residual connection, the connections from input to output are the first feedforward module, the multi-head self-attention module, the convolution module, the second feedforward module, and the layer norm module in sequence;

[0014] The multi-head self-attention module adopts a multi-head attention method of residual connection, layer normalization and position encoding, and the multi-head attention of position encoding adopts a relative position sinusoidal encoding scheme;

[0015] The convolution module adopts normalization, activation function, point-by-point convolution and one-dimensional depth separation convolution;

[0016] The first feedforward module and the second feedforward module include linear transformation and swish activation function.

[0017] In a preferred embodiment of the present application, the input audio features are input to a Conformer speech recognition model, and the Conformer speech recognition model includes a Conformer encoding module and a decoding module. It also includes:

[0018] The decoding module includes a Transformer decoding module and a decoding module of an improved Transformer decoder;

[0019] In the Transformer decoding module, based on the residual connection, the connections from input to output are input sequence module, masked multi-head attention module, first added standard module, multi-head attention module, second added standard module, feedforward module, third added standard module, linear transformation module, and Softmax function module.

[0020] In a preferred embodiment of the present application, in the Transformer decoding module, based on the residual connection, the connections from input to output are input sequence module, masked multi-head attention module, first adding standard module, multi-head attention module, second adding standard module, feedforward module, third adding standard module, linear transformation module, and Softmax function module. Including:

[0021] The Transformer decoding module includes a first self-attention module, a second self-attention module, and the decoding module of the improved Transformer decoder includes a third self-attention module and a cross attention module;

[0022] In the first self-attention module, the second self-attention module, and the third self-attention module, the connections from input to output are, in order, an input sequence module, a self-attention language module, a linear transformation module, and a Softmax function module;

[0023] In the self-attention language module, the connections from input to output are, in order, a masked multi-head attention module, a fourth addition standard module, a feedforward module, and a fifth addition standard module;

[0024] After a given input sequence, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position encoding to M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features. After linear transformation and Softmax function, the predicted sequence is output.

[0025] In a preferred embodiment of the present application, after the input sequence is given, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position coding to M identical self-attention language modules, fuse them with the output of the Conformer encoding module to obtain high-level features, and output the predicted sequence after linear transformation and Softmax function. Including:

[0026] When the input sequence has only 5 tokens, QK T Represents the attention score obtained by the input token, the QK T Each row represents the attention score obtained by the i-th token and all tokens except the i-th token;

[0027] In the Mask matrix, a green value of 1 indicates that it is not used for occlusion, and a black value of 0 indicates that it is used for occlusion;

[0028] When the Mask matrix and the QK T After bitwise multiplication, we get Mask QK T , the Mask QK T Each row only contains the corresponding token and the information before the corresponding token, and then the weighted sum is performed through the Softmax function.

[0029] In a preferred embodiment of the present application, after the given input sequence, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position coding to M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features, and output the predicted sequence after linear transformation and Softmax function. It also includes:

[0030] The self-attention language module includes a self-attention mechanism and a neural network;

[0031] When using the self-attention mechanism, the weight of each element in the sequence and all the sequence elements is calculated, and a weighted sum is performed on each element in the sequence according to the weights between the elements;

[0032] SA: The weighted summation is divided into the following 4 steps:

[0033] SA1: Calculate the query vector, key vector, and value vector for each element in the sequence:

[0034] Given an input sequence vector X = [x1, x2, ..., x n ], weight matrix W Q ,W K ,W V , for each element x in the sequence i , it is necessary to perform linear transformation with the three weight matrices to obtain the final query vector, key vector and value vector of each element, that is,

[0035] q i =W Q x i

[0036] k i =W K x i

[0037] v i =W V x i

[0038] q i ,k i ,v i Respectively represent the query vector, key vector and value vector corresponding to the i-th element;

[0039] SA2: Each query vector performs a dot product with each key vector to obtain the corresponding attention score:

[0040] For each element corresponding to the query vector q i Sum value vector k j, first do a dot product of the two to get the attention score, which represents the degree of attention of element i to element j, and then divide the attention score by a scaling factor to finally get the attention score:

[0041]

[0042] SA3: Use the Softmax function to get the attention weight for each element:

[0043] After obtaining the attention score s of element i and other elements i =[s i1 ,s i2 ,…,s ij …,s in ], the Softmax function is used to make the attention score s i Converted to attention weight, where the attention weight is a number between 0 and 1 and the sum is 1, indicating the degree of attention of element i to each element within the range of n elements, that is

[0044] w i =softmax(s i )=[w i1 ,w i2 ,…,w ij …,w in ]

[0045] SA4: Weighted sum of the attention weight of the element and the value vector corresponding to each element: After obtaining the attention weight w of element i i After that, each attention weight is multiplied by the corresponding value vector, and then the multiplied vectors are added together, that is,

[0046]

[0047] z i Include the required information for each element in the sequence;

[0048] For the entire sequence X, the corresponding self-attention calculation method is:

[0049] Q=W Q X

[0050] K=W K X

[0051] V=W V X

[0052]

[0053] In a preferred embodiment of the present application, after the given input sequence, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position coding to M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features, and output the predicted sequence after linear transformation and Softmax function. It also includes:

[0054] The third self-attention module is trained based on the public text dataset and the speech model cold fusion method;

[0055] Given the input token sequence of the decoder, after passing through the third self-attention module, an information extraction is performed and a high-level feature that extracts important information of the input token sequence is output;

[0056] Based on the cross attention module, the token sequence after information extraction is fused with the high-level features for processing.

[0057] In a preferred embodiment of the present application, the cross-attention module is used to fuse the token sequence after information extraction with the high-level features for processing. This includes:

[0058] Using the high-level features as input to a criss-cross attention query vector in the criss-cross attention module;

[0059] The cross-attention module includes a cross-attention mechanism;

[0060] The output of the token sequence after information extraction is used as the input of the key vector and value vector. After N times of cross-attention mechanism and other operations, the recognition result of the next token sequence is output.

[0061] The second invention objective of this application is achieved through the following technical solutions:

[0062] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned speech recognition method applied to classroom assessment are implemented.

[0063] The third invention objective of this application is achieved through the following technical solutions:

[0064] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned speech recognition method applied to classroom assessment.

[0065] In summary, the present application includes at least one of the following beneficial technical effects:

[0066] 1. Based on the Transformer decoding module and the improved Transformer decoder decoding module, the token sequence after embedding and position encoding is first deeply fused with the output of the Conformer encoding module to obtain high-level features rich in contextual information. By taking the obtained high-level features as input, the Transformer decoding module and the improved Transformer decoder decoding module further iterate N-1 times. Each iteration is a refinement of the features, allowing the model to gradually refine its understanding of the sequence in continuous information transmission and interaction. After N-1 iterations, the final output features are converted into the probability distribution of the token at the next time step through linear transformation and Softmax function processing. This step is the key to the decoding process, which enables the model to predict the next token that is most likely to appear given the context.

[0067] 2. High-level features are cleverly used as the input of the cross-attention query vector in the cross-attention module. This design significantly improves the information extraction and integration capabilities of the decoding module of the improved Transformer decoder when processing sequence data. The core of the cross-attention module is the cross-attention mechanism, which can effectively interact and fuse features from different sources to mine deeper semantic information.

[0068] Specifically, the decoding module of the improved Transformer decoder first outputs the token sequence after one information extraction as the input of the key vector and value vector in the cross-attention module. This step ensures that the decoding module of the improved Transformer decoder can make full use of the extracted sequence information in subsequent processing to provide rich context for the query vector. In this way, the cross-attention module can more accurately locate and focus on the key elements in the sequence, thereby improving the accuracy of recognition.

[0069] Subsequently, the decoding module of the improved Transformer decoder deeply mines and refines the information in the sequence through N iterative operations of the cross-attention mechanism. Each iteration is a detailed processing of the sequence features, allowing the model to gradually optimize its understanding of each token in the sequence in the process of continuous attention weighting. This recursive processing method not only enhances the model's grasp of the global structure of the sequence, but also improves its ability to recognize local details.

[0070] Finally, after N times of complex operations such as the cross-attention mechanism, the decoding module of the improved Transformer decoder outputs the recognition result of the next token sequence. This result is based on a comprehensive consideration of all relevant information, including the historical information of the sequence, the contextual information of the current token, and the external information integrated through the cross-attention mechanism. Therefore, this recognition result has a high degree of credibility and accuracy, providing strong support for sequence processing tasks such as speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flow chart of an implementation of a first embodiment of a speech recognition method for classroom assessment in the present application;

[0072] Figure 2 This is a flow chart for implementing S20 in Embodiment 1 of a speech recognition method for classroom assessment in the present application;

[0073] Figure 3 It is another implementation flowchart of S20 and an implementation flowchart of S206 in Embodiment 1 of a speech recognition method for classroom assessment of the present application;

[0074] Figure 4 This is a first implementation flow chart of S4 in Embodiment 1 of a speech recognition method for classroom assessment in the present application;

[0075] Figure 5 This is a second implementation flow chart of S4 in Embodiment 1 of a speech recognition method for classroom assessment of the present application;

[0076] Figure 6 It is a third implementation flowchart of S4 in Embodiment 1 of a speech recognition method for classroom assessment of the present application and an implementation flowchart of S48 in Embodiment 2;

[0077] Figure 7 This is a principle block diagram of a computer device of the present application. DETAILED DESCRIPTION

[0078] The following is combined with Figure 1-6 This application is described in further detail.

[0079] In the first embodiment, Figure 1 As shown, the present application discloses a speech recognition method applied to classroom assessment, which specifically includes the following steps:

[0080] S10: When receiving an evaluation instruction sent by the teacher terminal, receiving audio from the evaluation instruction and extracting audio features;

[0081] In this embodiment, the teacher terminal refers to a PC installed in the classroom for the teacher to operate or a mobile terminal carried by the teacher. The evaluation instruction refers to an instruction issued by the teacher through the teacher terminal for collecting audio, processing audio, and evaluating audio. The audio feature refers to a series of parameters extracted from the captured audio, representing various properties of the audio. The audio features include pitch features, spectrum features, and energy features.

[0082] Specifically, when a teacher evaluates a student's language, such as Mandarin, instructions for collecting, processing, and evaluating audio are issued through a PC logged in to the classroom or a mobile terminal APP carried by the teacher to extract the audio's pitch features, spectral features, and energy features.

[0083] S20: Input the audio features to the Conformer speech recognition model, which includes a Conformer encoding module and a decoding module;

[0084] In this embodiment, audio features are input to the Conformer speech recognition model. The Conformer encoding module refers to the part that generates an audio feature sequence including context information by processing the input audio feature sequence. The decoding module refers to the part that generates an output sequence according to the feature representation provided by the Conformer encoding module, and predicts the next element in the sequence.

[0085] S30: After the audio features are re-extracted in the Conformer encoding module, high-level features are obtained based on the Transformer decoding module in the decoding module and the decoding module of the improved Transformer decoder in the decoding module;

[0086] In this embodiment, high-level features refer to the audio features that have been further abstracted and transformed after deep processing by the Transformer decoding module in the decoding module and the improved Transformer decoder in the decoding module, forming a feature representation that can better represent the high-level semantics and structural information of the audio signal. These high-level features can then be used by the decoding module to generate the final recognition result, such as a text sequence.

[0087] S40: Based on the self-attention mechanism of the Transformer decoding module in the decoding module and the cross-attention mechanism of the decoding module of the improved Transformer decoder in the decoding module, the high-level features are decoded and the corresponding recognition results are output.

[0088] In this embodiment, decoding refers to converting high-level features into readable text.

[0089] Reference Figure 2 , S20. Includes:

[0090] S201: In the Conformer encoding module, based on the residual connection, the connections from input to output are, in sequence, a first feedforward module, a multi-head self-attention module, a convolution module, a second feedforward module, and a layer norm module.

[0091] S202: The multi-head self-attention module adopts a multi-head attention method of residual connection, layer normalization and position encoding, and the multi-head attention of the position encoding adopts a relative position sinusoidal encoding scheme;

[0092] S203: The convolution module adopts normalization, activation function, point-by-point convolution and one-dimensional depth separation convolution;

[0093] S204: The first feedforward module and the second feedforward module include a linear transformation and a swish activation function.

[0094] In this embodiment, the multi-head self-attention module significantly improves the ability of the multi-head self-attention module to process sequence data by integrating the multi-head attention mechanism of residual connection, layer normalization and position encoding. The sinusoidal coding scheme adopted by the position encoding can effectively generalize the length of the input sequence, ensuring that the multi-head attention module can maintain good performance when processing sequences of different lengths. The convolution module adopts normalization, activation function, point-by-point convolution and one-dimensional deep separation convolution technology in structure. The design of these two convolutional networks is to finely capture the local features in the input sequence, so that the Conformer encoding module can identify subtle changes and key patterns in the speech signal. The application of normalization enables the network to be trained more stably and efficiently. The composition of the first feedforward module and the second feedforward module mainly includes a linear layer and a swish activation function. This combination greatly enhances the model's ability to express features. The linear layer ensures the transmission and transformation of features, while the swish activation function can better retain the nonlinear information of features due to its smooth and non-monotonic characteristics. In general, the synergistic effect of the multi-head self-attention module, the convolution module, the first feedforward module, and the second feedforward module enables the entire network to not only accurately capture local and global information when processing sequential tasks such as speech recognition, but also efficiently learn the deep structure and semantic information in the data.

[0095] Reference Figure 3 , S20. Also includes:

[0096] S205: The decoding module includes a Transformer decoding module and a decoding module of an improved Transformer decoder;

[0097] S206: In the Transformer decoding module, based on the residual connection, the connections from input to output are, in sequence, an input sequence module, a masked multi-head attention module, a first added standard module, a multi-head attention module, a second added standard module, a feedforward module, a third added standard module, a linear transformation module, and a Softmax function module.

[0098] In this embodiment, based on the decoding module of the Transformer decoding module and the improved Transformer decoder, the token sequence after embedding and position encoding is first deeply fused with the output of the Conformer encoding module, thereby obtaining high-level features rich in contextual information. By taking the obtained high-level features as input, the Transformer decoding module and the improved Transformer decoder decoding module further perform N-1 iterations. Each iteration is a refinement of the features, so that the model can gradually refine its understanding of the sequence in continuous information transmission and interaction. After N-1 iterations, the final output features are converted into the probability distribution of the token at the next time step through linear transformation and Softmax function processing. This step is the key to the decoding process, which enables the model to predict the next token that is most likely to appear under a given context.

[0099] Reference Figure 3 , S206. Includes:

[0100] S1: The Transformer decoding module includes a first self-attention module and a second self-attention module. The decoding module of the improved Transformer decoder includes a third self-attention module and a cross attention module;

[0101] S2: In the first self-attention module, the second self-attention module, and the third self-attention module, the connections from input to output are input sequence module, self-attention language module, linear transformation module, and Softmax function module in sequence;

[0102] S3: In the self-attention language module, the connections from input to output are, in order, a masked multi-head attention module, a fourth addition standard module, a feedforward module, and a fifth addition standard module;

[0103] S4: After a given input sequence, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position encoding to M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features, and output the predicted sequence after linear transformation and Softmax function.

[0104] In this example, when the input sequence has only 5 tokens, QK T Indicates the attention score obtained by the input token, QK T Each row represents the attention score of the ith token and all tokens except the ith token. In the Mask matrix, a green value of 1 indicates that it is not used for occlusion, and a black value of 0 indicates that it is used for occlusion. When the Mask matrix and QK T After bitwise multiplication, we get Mask QK T , Mask QK T Each row only contains the corresponding token and the information before the corresponding token, and then the weighted sum is performed through the Softmax function.

[0105] Reference Figure 4 , S4. Includes:

[0106] S41: When the input sequence has only 5 tokens, QK T Represents the attention score obtained by the input token, the QK T Each row represents the attention score obtained by the i-th token and all tokens except the i-th token;

[0107] S42: In the Mask matrix, a green value of 1 indicates that it is not used for occlusion, and a black value of 0 indicates that it is used for occlusion;

[0108] S43: When the Mask matrix and the QK T After bitwise multiplication, we get Mask QK T , the Mask QK T Each row only contains the corresponding token and the information before the corresponding token, and then the weighted sum is performed through the Softmax function.

[0109] In this embodiment, by introducing the Mask operation, in the output of each module, the output of each time step in the sequence is independently calculated based on the input information of itself and the previous time step. Especially in the Transformer decoding module, this autoregressive characteristic is particularly evident. The working mechanism of the Transformer decoding module is to predict the next token in the sequence based on the known token. This sequentially dependent prediction method enables the model to better understand the coherence and contextual relationship of the language. To achieve this goal, this application adopts the following Figure 4The structured training data shown in . These data are carefully designed to ensure that during the training process, the model can learn how to predict future tokens based on current and previous contextual information. By using the cross-entropy loss function for training, first, it ensures that the model can strictly follow the order of the time series when predicting and will not be disturbed by future information. Secondly, as a common method to measure prediction errors in classification problems, the cross-entropy loss function can effectively guide the model to learn a more accurate token distribution, thereby improving the decoding accuracy. Finally, after such training, the model showed higher performance in sequence generation tasks such as speech recognition and machine translation. It can not only generate fluent sequences, but also better handle long-distance dependencies and complex language structures, providing strong technical support for practical applications.

[0110] Reference Figure 5 , S4. Also includes:

[0111] S44: The self-attention language module includes a self-attention mechanism and a neural network;

[0112] S45: When using the self-attention mechanism, calculate the weight of each element in the sequence and all sequence elements, and perform weighted summation on each element in the sequence according to the weights between the elements;

[0113] SA: The weighted summation is divided into the following 4 steps:

[0114] SA1: Calculate the query vector, key vector, and value vector for each element in the sequence:

[0115] Given an input sequence vector X = [x1, x2, ..., x n ], weight matrix W Q ,W K ,W V , for each element x in the sequence i , it is necessary to perform linear transformation with the three weight matrices to obtain the final query vector, key vector and value vector of each element, that is,

[0116] q i =W Q x i

[0117] k i =W K x i

[0118] v i =W V x i

[0119] q i ,ki ,v i Respectively represent the query vector, key vector and value vector corresponding to the i-th element;

[0120] SA2: Each query vector performs a dot product with each key vector to obtain the corresponding attention score:

[0121] For each element corresponding to the query vector q i Sum value vector k j , first do a dot product of the two to get the attention score, which represents the degree of attention of element i to element j, and then divide the attention score by a scaling factor to finally get the attention score:

[0122]

[0123] SA3: Use the Softmax function to get the attention weight for each element:

[0124] After obtaining the attention score s of element i and other elements i =[s i1 ,s i2 ,…,s ij …,s in ], the Softmax function is used to make the attention score s i Converted to attention weight, where the attention weight is a number between 0 and 1 and the sum is 1, indicating the degree of attention of element i to each element within the range of n elements, that is

[0125] w i =softmax(s i )=[w i1 ,w i2 ,…,w ij …,w in ]

[0126] SA4: Weighted sum of the attention weight of the element and the value vector corresponding to each element: After obtaining the attention weight w of element i i After that, each attention weight is multiplied by the corresponding value vector, and then the multiplied vectors are added together, that is,

[0127]

[0128] z i Include the required information for each element in the sequence;

[0129] For the entire sequence X, the corresponding self-attention calculation method is:

[0130] Q=W Q X

[0131] K=W K X

[0132] V=W V X

[0133]

[0134] In this embodiment, the self-attention mechanism is unique in that it can fully consider the relationship between each element in the sequence and all other elements when processing sequence data. Through the self-attention mechanism, the model can examine each element in the sequence from multiple angles at the same time, just like having multiple "focuses of attention". This ability makes the model more comprehensive and accurate in extracting sequence information. It can not only capture the explicit information in the sequence, but also gain insight into the important information implicit in the subtle relationships between elements. Therefore, the self-attention mechanism greatly improves the efficiency and quality of information extraction of the model when processing sequence data.

[0135] This excellent information extraction capability directly leads to a significant improvement in the accuracy of the model when processing sequence data. Whether it is word meaning understanding and sentence structure analysis in natural language processing, or phoneme recognition and intonation perception in speech recognition, the self-attention mechanism can help the model capture key information more accurately, thereby making more precise judgments in the decoding and prediction process.

[0136] In addition, the introduction of the self-attention mechanism has greatly enhanced the model's ability to handle long-distance dependencies. In traditional sequence processing models, long-distance dependencies are often a difficult challenge to overcome, but the self-attention mechanism effectively solves this problem by directly establishing the association between any two elements in the sequence. This enables the model to maintain higher performance and lower error rates when dealing with tasks such as long sentence understanding and long speech recognition.

[0137] Reference Figure 6 , S4. Also includes:

[0138] S46: Training the third self-attention module based on the public text dataset and the speech model cold fusion method;

[0139] S47: Given the input token sequence of the decoder, after passing through the third self-attention module, perform an information extraction and output a high-level feature that extracts important information of the input token sequence;

[0140] S48: Based on the cross attention module, the token sequence after information extraction is fused with the high-level features for processing.

[0141] In this embodiment, high-level features are cleverly used as the input of the cross-attention query vector in the cross-attention module. This design significantly improves the information extraction and integration capabilities of the decoding module of the improved Transformer decoder when processing sequence data. The core of the cross-attention module is the cross-attention mechanism, which can effectively interact and fuse features from different sources to mine deeper semantic information.

[0142] Specifically, the decoding module of the improved Transformer decoder first outputs the token sequence after one information extraction as the input of the key vector and value vector in the cross-attention module. This step ensures that the decoding module of the improved Transformer decoder can make full use of the extracted sequence information in subsequent processing to provide rich context for the query vector. In this way, the cross-attention module can more accurately locate and focus on the key elements in the sequence, thereby improving the accuracy of recognition.

[0143] Subsequently, the decoding module of the improved Transformer decoder deeply mines and refines the information in the sequence through N iterative operations of the cross-attention mechanism. Each iteration is a detailed processing of the sequence features, allowing the model to gradually optimize its understanding of each token in the sequence in the process of continuous attention weighting. This recursive processing method not only enhances the model's grasp of the global structure of the sequence, but also improves its ability to recognize local details.

[0144] Finally, after N times of complex operations such as the cross-attention mechanism, the decoding module of the improved Transformer decoder outputs the recognition result of the next token sequence. This result is based on a comprehensive consideration of all relevant information, including the historical information of the sequence, the contextual information of the current token, and the external information integrated through the cross-attention mechanism. Therefore, this recognition result has a high degree of credibility and accuracy, providing strong support for sequence processing tasks such as speech recognition.

[0145] In Example 2, referring to Figure 6 , S48. Includes:

[0146] S481: Using the high-level features as input of a cross-attention query vector in the cross-attention module;

[0147] S482: The cross attention module includes a cross attention mechanism;

[0148] S483: The output of the token sequence after information extraction is used as the input of the key vector and the value vector. After N times of cross-attention mechanism and other operations, the recognition result of the next token sequence is output.

[0149] In this embodiment, in order to further improve the efficiency of extracting input token sequence information, the present application trains the speech recognition model through the language model cold fusion method. On this basis, the present application carefully designs a method such as Figure 6 The improved decoder shown in Figure 1 cleverly combines the self-attention mechanism and the cross-attention mechanism to achieve efficient utilization and deep understanding of sequence information.

[0150] The application of the self-attention mechanism in the decoder enables the decoding module of the improved Transformer decoder to automatically learn the intrinsic connection between each token in the sequence, thereby capturing the long-distance dependencies and complex semantic structures within the sequence. This mechanism not only enhances the model's grasp of the global information of the sequence, but also increases its attention to local features, providing rich contextual information for the subsequent decoding process.

[0151] At the same time, the cross-attention mechanism further strengthens the model's ability to interact with information in different feature dimensions. By cross-fusing the information of the input token sequence with the high-level features output by the encoder, the decoder can more accurately locate and identify the key elements in the sequence, thereby generating a more accurate and language-logical sequence during the decoding process.

[0152] In Example 3, referring to Table 1-2, two types of data were used in the experiment. For speech recognition, the experiment used Aishell-1 as the experimental data set, which is an open source speech data set for Mandarin Chinese, with a total length of 178 hours. It was recorded by 400 speakers from different accent areas in China in a quiet indoor environment, with an audio sampling rate of 16kHz. The recorded texts involve five major fields: finance, sports, technology, entertainment, and current news, and the accuracy of the transcribed and annotated text is above 95%. For the language model, in order to use the same similar text data set, the experiment additionally used the text data of Aishell-2, which lasted for 1000 hours. The recorded texts involved 12 fields such as wake-up words, voice control words, smart homes, unmanned driving, and industrial production, and the database text accuracy was above 96%. The experiment divided the Aishell-1 data set into a training set, a validation set, and a test set, of which 120,098 pieces of data in the training set totaled 150 hours, 14,326 pieces of data in the validation set totaled 18 hours, and 7,176 pieces of data in the test set totaled 10 hours. The text data of Aishell-2 was used to pre-train the language model. The text data contained in the validation set and test set of Aishell-1 was removed from the data. After removal and deduplication, the total number of data was 762,354.

[0153] Experimental process:

[0154] First, the model's input audio and transcribed text labels are processed. The Conformer encoding module uses 80-dimensional Fbank features with a frame length of 25ms and a frame shift of 10ms as input, processes it using Global Cepstral Mean Variance Normalization (Global-CMVN), and uses spectral enhancement technology to enhance the robustness of the model. In addition, the experiment uses audio speed perturbations of 0.9 and 1.1 to expand the training data set. The transcribed text in the Aishell-2 training data set has a total of 5156 characters, and whitespace characters are added. <blank>、Unknown Symbol <unk>and start and end characters <sos eos>,Since the model in the present invention uses characters as the smallest recognition unit, the entire model dictionary has 5159 tokens in total.

[0155] Then, the model parameters are configured. In the Conformer encoding module, the features after spectrum enhancement are subsampled by convolution to make their length 1 / 4 of the original. The number of units in the linear layer is 2048, and a total of 12 Conformer modules are used. For each Conformer module, the output dimension is 256, the number of hidden layers in the feedforward module is 2048, the number of heads in the multi-head self-attention module is 4, the number of hidden layer units is 256, the depthwise convolution kernel size in the convolution module is 15, and the dropout is all set to 0.1. In the Conformer decoder, the dimension of the token embedding vector is 256, the number of decoding blocks is set to 8, the number of hidden layers of the multi-head attention in the decoding block is set to 256, and the number of hidden layers in the feedforward module is 2048. In the improved decoder, except that the number of self-attention layers M and the number of cross-attention layers N are not set, the rest of the parameters are the same as the Conformer decoder.

[0156] Finally, configure the model's hyperparameters. In the experiment of pre-training the language model, the batch size was set to 70, and the adam optimizer with a learning rate of 0.002 was used for a total of 100 epochs. Then the speech recognition model was trained using the pre-trained language model. At this time, the model batch size was set to 16, and the adam optimizer with a learning rate of 0.001 was used for a total of 240 epochs. After the training, the 20 models with the smallest loss in the validation set were selected and their parameters were averaged to obtain the final recognition model, and then this model was used to decode the test audio to obtain the final recognition result.

[0157] Evaluation indicators:

[0158] The experiment uses the Character Error Rate (CER), which is commonly used in Chinese speech recognition, as an evaluation indicator. The lower the character error rate, the better the recognition effect. The calculation formula is:

[0159]

[0160] Among them, N represents the total number of characters in the test data, S (substitution) represents the number of characters replaced in the recognition result, D (deletion) represents the number of characters deleted in the recognition result, and I (insertion) represents the number of characters inserted in the recognition result.

[0161] In order to measure the effect of the self-attention language model, the experiment uses perplexity (PPL) as an evaluation indicator. The lower the perplexity, the better the effect of the language model. The calculation formula is:

[0162] H(P,Q)=-∑ x P(x)logQ(x)

[0163] PPL=2 H(P,Q)

[0164] Among them, H(P,Q) is the cross entropy loss of the self-attention language model.

[0165] In this embodiment, experimental results and analysis: First, the language model in the improved decoder is not pre-trained, and the model recognition effect is tested using different attention mechanism layers. The experimental results are shown in Table 1.

[0166] Table 1 Experimental results of different attention layers on Aishell-1

[0167]

[0168]

[0169] Then, in order to explore the role of the self-attention language model and its impact on the final speech recognition results, the experiment selected different training data sets to pre-train the language model. The experimental results are shown in Table 2.

[0170] Table 2 Experimental results on different datasets

[0171]

[0172] As can be seen from Table 2, for the self-attention language model, as the number of self-attention layers M and the text training data set increase, the test set PPL gradually decreases, which means that as the model parameters and favorable data increase, the model learns more information. For the speech recognition model, when only the Aishell-1 text data set is used for pre-training, its CER is the same as or higher than that of the model without pre-training; when the Aishell-2 text data is used for pre-training, additional text-related field information is introduced. When M = 8, N = 2, its CER decreases by 1.5% compared with the model without pre-training.

[0173] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0174] For the specific limitations of a speech recognition method and system applied to classroom evaluation, please refer to the limitations of a speech recognition method applied to classroom evaluation in the above text, which will not be repeated here. Each module in the above-mentioned speech recognition system applied to classroom evaluation can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0175] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store student information and sorting information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech recognition method applied to classroom evaluation is implemented.

[0176] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a speech recognition method applied to classroom assessment is implemented.

[0177] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0178] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.< / sos> < / unk> < / blank>

Claims

1. A speech recognition method for classroom assessment, characterized in that: Includes steps: When receiving an evaluation instruction from the teacher terminal, receiving audio from the evaluation instruction and extracting audio features; Input audio features to the Conformer speech recognition model, which includes a Conformer encoding module and a decoding module; After the audio features are extracted again in the Conformer encoding module, high-level features are obtained based on the Transformer decoding module in the decoding module and the decoding module of the improved Transformer decoder in the decoding module; Based on the self-attention mechanism of the Transformer decoding module in the decoding module and the cross-attention mechanism of the decoding module of the improved Transformer decoder in the decoding module, the high-level features are decoded and the corresponding recognition results are output.

2. The speech recognition method for classroom assessment according to claim 1, characterized in that: The input audio features are fed into the Conformer speech recognition model, which includes a Conformer encoding module and a decoding module. In the Conformer encoding module, based on the residual connection, the connections from input to output are the first feedforward module, the multi-head self-attention module, the convolution module, the second feedforward module, and the layer norm module in sequence; The multi-head self-attention module adopts a multi-head attention method of residual connection, layer normalization and position encoding, and the multi-head attention of position encoding adopts a relative position sinusoidal encoding scheme; The convolution module adopts normalization, activation function, point-by-point convolution and one-dimensional depth separation convolution; The first feedforward module and the second feedforward module include linear transformation and swish activation function.

3. The speech recognition method for classroom assessment according to claim 1, characterized in that: The input audio features are fed into the Conformer speech recognition model, which includes a Conformer encoding module and a decoding module. It also includes: The decoding module includes a Transformer decoding module and a decoding module of an improved Transformer decoder; In the Transformer decoding module, based on the residual connection, the connections from input to output are input sequence module, masked multi-head attention module, first added standard module, multi-head attention module, second added standard module, feedforward module, third added standard module, linear transformation module, and Softmax function module.

4. The method for speech recognition applied to classroom assessment according to claim 3, characterized in that: In the Transformer decoding module, based on the residual connection, the connections from input to output are input sequence module, masked multi-head attention module, first adding standard module, multi-head attention module, second adding standard module, feedforward module, third adding standard module, linear transformation module, and Softmax function module. Including: The Transformer decoding module includes a first self-attention module, a second self-attention module, and the decoding module of the improved Transformer decoder includes a third self-attention module and a cross attention module; In the first self-attention module, the second self-attention module, and the third self-attention module, the connections from input to output are, in order, an input sequence module, a self-attention language module, a linear transformation module, and a Softmax function module; In the self-attention language module, the connections from input to output are, in order, a masked multi-head attention module, a fourth addition standard module, a feedforward module, and a fifth addition standard module; After a given input sequence, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position encoding to M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features. After linear transformation and Softmax function, the predicted sequence is output.

5. A speech recognition method for classroom assessment according to claim 4, characterized in that: After the input sequence is given, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position encoding to M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features. After linear transformation and Softmax function, the predicted sequence is output. Including: When the input sequence has only 5 tokens, QK T Represents the attention score obtained by the input token, the QK T Each row represents the attention score obtained by the i-th token and all tokens except the i-th token; In the Mask matrix, a green value of 1 indicates that it is not used for occlusion, and a black value of 0 indicates that it is used for occlusion; When the Mask matrix and the QK T After bitwise multiplication, we get Mask QK T , the Mask QK T Each row only contains the corresponding token and the information before the corresponding token, and then the weighted sum is performed through the Softmax function.

6. A speech recognition method for classroom assessment according to claim 4, characterized in that: After the input sequence is given, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position encoding to M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features. After linear transformation and Softmax function, the predicted sequence is output. It also includes: The self-attention language module includes a self-attention mechanism and a neural network; When using the self-attention mechanism, the weight of each element in the sequence and all the sequence elements is calculated, and a weighted sum is performed on each element in the sequence according to the weights between the elements; SA: The weighted summation is divided into the following 4 steps: SA1: Calculate the query vector, key vector, and value vector for each element in the sequence: Given an input sequence vector X = [x1, x2, ..., x n ], weight matrix W Q ,W K ,W V , for each element x in the sequence i , it is necessary to perform linear transformation with the three weight matrices to obtain the final query vector, key vector and value vector of each element, that is, q i =W Q x i k i =W K x i v i =W V x i q i ,k i ,v i Respectively represent the query vector, key vector and value vector corresponding to the i-th element; SA2: Each query vector performs a dot product with each key vector to obtain the corresponding attention score: For each element corresponding to the query vector q i Sum value vector k j , first do a dot product of the two to get the attention score, which represents the degree of attention of element i to element j, and then divide the attention score by a scaling factor to finally get the attention score: SA3: Use the Softmax function to get the attention weight for each element: After getting the attention score s of element i and other elements i =[s i1 ,s i2 ,…,s ij …,s in ], the Softmax function is used to make the attention score s i Converted to attention weight, where the attention weight is a number between 0 and 1 and the sum is 1, indicating the degree of attention of element i to each element within the range of n elements, that is w i =softmax(s i )=[w i1 ,w i2 ,…,w ij …,w in ] SA4: Weighted sum of the attention weight of the element and the value vector corresponding to each element: After obtaining the attention weight w of element i i After that, each attention weight is multiplied by the corresponding value vector, and then the multiplied vectors are added together, that is, z i Include the required information for each element in the sequence; For the entire sequence X, the corresponding self-attention calculation method is: Q=W Q X K=W K X V=W V X 7. A speech recognition method for classroom assessment according to claim 4, characterized in that: After the input sequence is given, the first self-attention module, the second self-attention module, and the third self-attention module send each token in the sequence after embedding and adding position encoding to M identical self-attention language modules, and fuse them with the output of the Conformer encoding module to obtain high-level features. After linear transformation and Softmax function, the predicted sequence is output. It also includes: The third self-attention module is trained based on the public text dataset and the speech model cold fusion method; Given the input token sequence of the decoder, after passing through the third self-attention module, an information extraction is performed and a high-level feature is output to extract important information of the input token sequence; Based on the cross attention module, the token sequence after information extraction is fused with the high-level features for processing.

8. A speech recognition method for classroom assessment according to claim 7, characterized in that: Based on the cross attention module, the token sequence after information extraction is fused with the high-level features for processing. It includes: Using the high-level features as input to a criss-cross attention query vector in the criss-cross attention module; The cross-attention module includes a cross-attention mechanism; The output of the token sequence after information extraction is used as the input of the key vector and value vector. After N times of cross-attention mechanism and other operations, the recognition result of the next token sequence is output.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a speech recognition method for classroom assessment as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of a speech recognition method for classroom assessment as described in any one of claims 1 to 8.

Images