Spoken english recognition method and system based on contrastive learning and hybrid attention

Abstract

The application provides a spoken English recognition method and system based on contrast learning and mixed attention, which comprises the following steps: acquiring audio of a spoken English test, adding noise of a random category to environmental recording to realize noise enhancement and further construct a positive sample; performing feature extraction on the data added with noise based on multi-scale and mixed attention; inputting the features after embedding and position coding into an encoder to perform context modeling; inputting the output of the encoder and the target features after embedding and position coding into a decoder to complete decoding; in the training process, calculating a contrast loss through input of the positive sample, calculating the loss of each sample at the same time, performing reverse transmission, and obtaining a recognition model; and inputting the audio of a candidate to be transcribed into the recognition model to obtain a recognition result.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, and in particular relates to a spoken English recognition method and system based on contrastive learning and hybrid attention. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Speech recognition is a technology that uses sound wave signals to transcribe speech into text that humans or computers can understand. This technology is widely used in fields such as natural language interaction, smart homes, autonomous driving, and financial services.

[0004] Early speech recognition relied on template matching, simply matching the extracted speech signal against existing pre-set templates. However, this method performed extremely poorly in noisy environments, and due to variations in speaker abilities and language proficiency, the pre-set templates struggled to accurately predict all situations. With the rise of machine learning, statistical learning modeling has become mainstream. This method, based on extensive training data, constructs probabilistic models, achieving higher recognition accuracy.

[0005] Deep learning has developed rapidly in recent years, achieving remarkable results in natural language processing, computer vision, and speech recognition. Deep learning models can better model the relationship between speech features and text, and are currently the most accurate method for recognition. The speech recognition model of this invention is also based on deep learning methods. In general, deep learning-based speech recognition models have significantly improved accuracy, robustness, and speed, but some shortcomings still exist, such as the difficulty in training excellent models with insufficient data, or the decrease in recognition accuracy in noisy environments.

[0006] Speech recognition and speech synthesis fields place high demands on both the quality and quantity of audio data. However, in practical applications, due to limitations in the environment or equipment, the recorded audio quality is often less than ideal, containing noise, echoes, distortion, or bass distortion. Low-quality audio data makes it difficult to train high-accuracy speech recognition models, causing significant inconvenience to post-transcriptional analysis. Previous methods for handling low-quality, noisy data have relied on preprocessing to reduce noise or its interference with the task, such as using filters and noise reduction algorithms. However, these methods are time-consuming and labor-intensive, and the noise reduction results are often unsatisfactory.

[0007] Compared to ordinary speech recognition, spoken English in the examination room is noisy and complex, due to the recording environment and equipment. This greatly increases the difficulty of speech recognition and reduces the accuracy. Therefore, ordinary speech recognition technology is difficult to apply directly to the recognition of spoken English. Summary of the Invention

[0008] To overcome the shortcomings of the existing technology, this invention provides a spoken English recognition method based on contrastive learning and hybrid attention. It analyzes the characteristics and environmental settings of spoken English in the examination room, with the main purpose of enhancing noise resistance. This helps to improve the accuracy and reliability of the spoken English scoring system and brings a better user experience to candidates and examiners.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] Firstly, a spoken English recognition method based on contrastive learning and hybrid attention is disclosed, including:

[0011] The audio of an oral English test is obtained, and noise enhancement is achieved by superimposing random categories of noise on the environmental recording to construct positive samples.

[0012] Feature extraction from noisy data based on multi-scale and hybrid attention;

[0013] The features are embedded and positionally encoded before being input into the encoder for context modeling.

[0014] The encoder output and the embedded and position-encoded target features are input into the decoder to complete the decoding.

[0015] During training, the contrastive loss is calculated using positive sample inputs, and the loss for each sample is calculated simultaneously. This process is then reversed to obtain the recognition model.

[0016] The candidate's audio to be transcribed is input into the recognition model to obtain the recognition result.

[0017] As a further technical solution, feature extraction is performed on the noisy data based on multi-scale and hybrid attention pairs, specifically:

[0018] Extracting audio features from data with added noise;

[0019] The extracted audio features are convolved with different window sizes to reduce dimensionality and then concatenated to obtain multi-scale feature representations.

[0020] A hybrid attention mechanism based on temporal and scale is used to calculate the attention score for each temporal pair of features at different scales;

[0021] The feature of each dimension is calculated by weighting the attention score, and the multi-scale fusion feature after the hybrid attention calculation is obtained.

[0022] As a further technical solution, features are concatenated along the scale dimension and the dimensionality is reduced by convolution to obtain a feature map of size (t,c,L), where L is the number of different scales, t is the number of time series at a single scale, and c is the feature dimension at a single time series position.

[0023] The process involves three branches to obtain q, k, and v vectors of dimension (t*L,1). The v vector is obtained through max pooling, while the q and k vectors are obtained through convolution dimensionality reduction. The q vector is transposed and then multiplied with the k vector to generate an attention matrix of shape (t*L,t*L). Finally, the attention matrix is ​​multiplied with the v vector to obtain a scaled attention score of size (t*L,1).

[0024] As a further technical solution, features in each dimension are calculated based on attention score weighting to obtain multi-scale fusion features after hybrid attention calculation, specifically:

[0025] After smoothing and extracting information from the q, k, and v vectors through two fully connected layers, the temporal and scale dimensions are split, and the attention score is normalized by softmax on the scale dimension to represent the importance of each scale in each temporal sequence.

[0026] The attention map is obtained by calculating the feature value of each time step in the multi-scale feature map using the normalized attention score in a weighted summation manner.

[0027] As a further technical solution, audio feature extraction is included before feature extraction of the noisy data based on multi-scale and hybrid attention pairs, specifically:

[0028] The features are windowed, with each window considered as a frame. Then, a fast Fourier transform is performed on each frame to obtain frequency features. Finally, the frames are stacked in the time dimension to obtain the spectrogram.

[0029] As a further technical solution, during training, the decoder input is the encoder output and the target sequence after feature embedding and position encoding. During inference, the probability distribution of each time step is obtained through the decoder. The features obtained at each time step are processed to obtain a classification vector, which corresponds to the probability of selecting each word at that time step. After obtaining the probability matrix of each time step, the optimal text sequence is searched through the decoding algorithm to obtain the recognition result.

[0030] As a further technical solution, the decoder uses a parallel decoder with forward and reverse directions to replace the LSTM in the original Conformer, and the reverse decoder is trained with the opposite target sequence.

[0031] Secondly, a spoken English recognition system based on contrastive learning and hybrid attention is disclosed, including:

[0032] The noise superposition module is configured to: acquire the audio of the spoken English test, superimpose random categories of noise based on the environmental recording to achieve noise enhancement and then construct positive samples;

[0033] The feature extraction module is configured to extract features from noisy data based on multi-scale and hybrid attention pairs.

[0034] The encoding and decoding module is configured to: input the features after embedding and positional encoding into the encoder for context modeling;

[0035] The encoder output and the embedded and position-encoded target features are input into the decoder to complete the decoding.

[0036] The training module is configured to: calculate the contrastive loss using positive sample input during training, calculate the loss for each sample simultaneously, backpropagate, and obtain the recognition model.

[0037] The recognition module is configured to input the candidate's audio to be transcribed into the recognition model and obtain the recognition result.

[0038] The above one or more technical solutions have the following beneficial effects:

[0039] To further improve the noise robustness of speech recognition models, this invention proposes a speech recognition training method based on contrastive learning. This method augments data by adding noise, which is then compared with the original clean speech as positive samples to improve the model's recognition accuracy in noisy environments. The core idea is that the model learns the mapping relationship between the original and noise-added frequencies, utilizing the difference between them to learn a more robust feature representation, thereby improving generalization ability in noisy environments.

[0040] To reduce the destructive impact of noisy features on speech recognition models and focus on more useful features, this invention introduces a multi-scale fusion feature extraction method into speech recognition tasks. To strengthen key features, reduce redundancy, and decrease computational cost, a hybrid attention mechanism is proposed to assign weights to features at different scales for each time series, automatically identifying and focusing on more important features in the task. This allows the model to have greater flexibility in processing global and local information.

[0041] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0042] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0043] Figure 1 This is a flowchart of an embodiment of the present invention, specifically an STFT flowchart.

[0044] Figure 2 This is a flowchart illustrating the multi-scale-temporal attention process according to an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of the codec structure according to an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the training framework according to an embodiment of the present invention;

[0047] Figure 5 This is a flowchart of the feature extraction process based on multi-scale hybrid attention proposed in this embodiment of the invention for audio.

[0048] Figure 6 This is a flowchart of the joint noise training process based on contrastive learning, as described in an embodiment of the present invention.

[0049] The attention mechanism is a part or one component of the encoder, and can be considered as... Figure 5 yes Figure 6 Part of the encoder in Environment 3. Detailed Implementation

[0050] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0051] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0052] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0053] Example 1

[0054] This embodiment discloses a spoken English recognition method based on contrastive learning and hybrid attention, including:

[0055] Step 1: Audio Feature Extraction

[0056] STFT, short for Short Time Fourier Transform, is a method for analyzing temporal features. It decomposes temporal information into two dimensions: time and frequency. Specifically, it involves windowing the features, treating each window as a frame, and then performing an FFT (Fast Fourier Transform) on each frame to obtain the frequency features. Finally, stacking these frames along the time dimension yields a spectrogram. Figure 1 As shown.

[0057] In STFT, key parameters include hop_length (frame shift), win_length (frame length), and n_fft (number of points in a single frame to perform FFT). For an audio signal of length n, it is first divided into windows with a step size of hop_length. The window size for each division is win_length, and the feature dimension obtained by a single window is (1, n_fft / 2). The formula for calculating the number of windows (frames) for a signal with dimension (1, n) is:

[0058] n_frames=1+ceil((n-win_length) / hop_Length)

[0059] The final audio feature vector has dimensions of (n_frames, n_fft / 2).

[0060] Step 2: Feature Extraction Based on Hybrid Attention

[0061] A multi-scale fusion feature extraction method aims to mitigate the impact of noisy features on language recognition models. It fully leverages the advantages of both global and local features, combining these two types of features to obtain more robust feature representations and reduce the impact of noise on model performance. Furthermore, a hybrid attention mechanism is introduced to strengthen key features and reduce redundancy at both the scale and temporal levels, improving model flexibility. The attention mechanism automatically assigns weights to identify and focus on more important features in the task, allowing the model to prioritize learning key features containing useful information and automatically adjust to reduce the contribution of noisy features to the results, thus minimizing the interference of noise on the model's predictive ability.

[0062] First, such as Figure 2 As shown, the obtained audio feature vectors are fed into convolutional layers with different window sizes to extract multi-scale features. To ensure consistent final feature map dimensions, 1x1 smooth convolutions are applied for preprocessing at all scales, using consistent phase lengths and appropriate edge padding. Then, these features are concatenated along the scale dimension, and the dimensionality is reduced through convolution to decrease computation, resulting in a feature map of size (t, c, L), where L represents the number of different scales (e.g., 3 in the example), t is the number of temporal sequences at a single scale, and c is the feature dimension at a single temporal position.

[0063] Next, three branches are used to obtain q, k, and v vectors of dimension (t*L, 1). The v vector is obtained through max pooling, while the q and k vectors are obtained through convolutional dimensionality reduction. The transpose of the q vector is then multiplied by the k vector to generate an attention matrix of shape (t*L, t*L). Finally, the attention matrix is ​​multiplied by the v vector to obtain a scaled attention score of size (t*L, 1).

[0064] Furthermore, after smoothing and information extraction of the above vectors through two fully connected layers (L->512 and 512->L), the temporal and scale dimensions are split, and the attention score is softmax normalized on the scale dimension to represent the importance of each scale in each temporal sequence. The normalized attention scores are used to calculate the feature value of each temporal sequence in the multi-scale feature map in a weighted summation manner, resulting in the attention map calculated by the attention mechanism.

[0065] The feature extraction method that combines scale and temporal attention can calculate the importance of each scale to each temporal position. It can capture both coarse-level global features and retain detailed local features, thereby improving the model's performance and generalization ability, while removing noise information and retaining more key details.

[0066] Global features can assist in modeling contextual information and establishing more comprehensive feature relationships, but they inevitably introduce noise. Local features, on the other hand, focus on details and include key features beyond the noise. To reduce the destructive impact of noisy features on speech recognition models and focus on more useful features, this invention introduces a multi-scale fusion feature extraction method into speech recognition tasks. To strengthen key features, reduce redundancy, and reduce computational cost, a hybrid attention mechanism is proposed to assign weights to features at different scales for each time series, automatically identifying and focusing on more important features in the task. This allows the model to have greater flexibility in processing global and local information.

[0067] Step 3: Speech Recognition Model:

[0068] The deep learning language recognition model used employs a classic encoder-decoder structure. The encoder adopts the same structure as Conformer, combining the advantages of CNN and Transformer. The decoder differs from the Long Short-Term Memory (LSTM) network in the Conformer paper, employing a "bidirectional" Transformer Encoder.

[0069] include:

[0070] Feature extraction module: This module converts audio into vector features. Speech features are obtained through the feature extraction module. Commonly used speech features include FBANK and MFCC. This patent uses the former because FBANK is closer to human auditory perception and retains more original information. When calculating FBANK features, STFT is performed on the speech signal to obtain the frequency spectrum and power spectrum. Then, a Mel filter bank is applied to the power spectrum to simulate the nonlinear frequency and energy response characteristics of the human ear, which is biologically closer to human perception.

[0071] Feature Encoding: The extracted acoustic features undergo feature embedding and positional encoding. The former maps discrete features to a continuous vector space, achieving dimensionality reduction, capturing semantic relationships, and providing continuous representation. The latter, positional encoding, is added to the input vector, providing the model with positional information of sequence elements, compensating for the Transformer's lack of sequential information representation.

[0072] The context encoder module employs a Conformer encoding structure, combining Transformer and CNN to achieve powerful and efficient sequence modeling. Each Conformer Encoder consists of a multi-head self-attention mechanism, a convolution module, and a feed-forward module. These modules are stacked according to the order of features, using residual connections and normalization to achieve a stable training process. In practical encoder design, multiple Conformer Encoders are often stacked to increase model depth and capture more and deeper sequence features. The embedded and positionally encoded features are then processed by the encoder to effectively model local and global relationships in the sequence, ultimately outputting the sequence-modeled features.

[0073] Decoder Module: The decoder structure consists of stacked Transformer decoders. Each Transformer decoder is similar to the Transformer encoder, except it incorporates a masked multi-head self-attention module. During training, the input is the encoder's output and the target sequence after feature embedding and positional encoding. During training, it is assumed that the target sequence is... <ss>-nice-to-meet-you- <eos>Here, SS (start symbol) is the start symbol, which serves as the first input to the decoder during training and inference; EOS (end of sentence) is the end symbol, added to the end of both the source and target sequences to indicate the end of decoding, allowing for appropriate truncation of the output. As shown in Table 1, masking is used during training to prevent overfitting due to seeing future information. The idea is to use existing information to predict unknown future information. <ss>Predict nice using <ss>The `nice` parameter is used to predict `to`, and so on. See Table 1 for the diagonal mask of the decoder's target sequence:

[0074] Table 1

[0075]

[0076] As a typical sequence-to-sequence task, audio relies heavily on contextual information. Therefore, to capture more contextual information and fully utilize bidirectional sequence features, a parallel decoder with both forward and reverse directions was chosen to replace the LSTM in the original Conformer. The forward decoder is unremarkable, while the reverse decoder is trained with the opposite target sequence as input.

[0077] The specific steps are as follows: First, reverse the original target sequence nice-to-meet-you to obtain the reversed sequence you-meet-to-nice, and then add the starting signal to both the forward and reverse sequences. <ss>and end signal <eos>The sequences are then fed into the left decoder (processing the forward sequence) and the right decoder (processing the reverse sequence), respectively. The outputs `logits_left` and `logits_right` are obtained from these two decoders and then integrated and voted on. Learnable parameter weights are used for voting; specifically, a learnable parameter `w` is set. `w` is mapped to the 0-1 interval after passing through the Sigmoid activation function. The mapped value is used as the parameter for the left decoder. The final output of the decoder is equal to `Sigmoid(w)*logits_left + (1-Sigmoid(w))*logits_right`.

[0078] During decoding, the bidirectional decoder combines the features generated by the encoder to understand the input sequence and output the probability distribution of the target sequence. Because inverse information is introduced, it acts as a "voting" mechanism, achieving a more accurate decoding output. (See appendix.) Figure 3 As shown.

[0079] Step 4: Loss calculation or decoding transcription:

[0080] During training, the CTC loss is calculated using the probability matrix and the target sequence. CTC loss is a loss function for sequence prediction tasks that automatically aligns temporal features with text labels, eliminating the need for separate alignment information and simplifying the training process. CTC loss can handle temporal features of varying lengths; models trained on short-timeframe data can still demonstrate good prediction performance on longer-timeframe data. Despite its many advantages, CTC loss still has some limitations, the most prominent being its poor modeling ability in noisy environments.

[0081] During inference, the probability distribution for each time step is obtained via the decoder. The features obtained at each time step are then processed by Softmax to obtain a classification vector, corresponding to the probability of selecting each word at that time step. After obtaining the probability matrix for each time step, the optimal text sequence is searched using a decoding algorithm (such as a greedy algorithm or Beam Search).

[0082] During training, the loss is calculated using the final output and labels, and then backpropagated. During inference, the final output is used for CTC decoding to obtain the inference result.

[0083] Step 5: Compare the losses:

[0084] In deep learning, contrastive loss is a loss function that measures similarity. Its ultimate goal is to maximize the similarity between sample pairs of the same category or with similar features, while minimizing the similarity between sample pairs of different categories or with significantly different features. A batch of N samples is obtained through random sampling. For each sample, new data is obtained through data augmentation or other methods and designated as the positive sample. The remaining 2N-1 samples are designated as negative samples. The loss function for a sample x1 and its positive sample x2 is defined as follows:

[0085]

[0086] The input to the function f(x) is sample data, typically the extracted features of the complete sample x. The sim function measures the similarity of variables, such as Euclidean distance, cosine similarity, or Pearson coefficient. t is a temperature parameter that controls the magnitude of the loss. As the formula shows, assuming f(x) is the feature extraction method, we hope to reduce the loss, which means we want the similarity between the feature spaces of positive samples to be as high as possible, while the similarity between negative samples to be as low as possible.

[0087] For this batch, the losses for all pairs will be calculated using the following formula:

[0088]

[0089] Step Six: Noise Enhancement Contrast Training:

[0090] In audio signal processing, noise enhancement is achieved by superimposing random types of noise to construct positive sample pairs, and feature extraction is performed using a feature extraction module with shared model parameters. Specifically, first, a batch of N samples is randomly sampled, and a certain type of noise is randomly superimposed on each sample. The data before and after noise addition are used as positive sample pairs, which are then combined with the other N-1 data to form negative sample pairs. Next, the feature extraction module extracts features from the 2N data, ensuring that the feature representation of different data is completely identical through shared model parameters.

[0091] In this implementation example, adding noise is an extremely important step. If the noise ratio is too high, the model will have difficulty learning basic recognition capabilities and will struggle to converge. Therefore, the noise length is controlled to be around 20% of the original clean audio length. Alternatively, the model can be gradually adapted to speech signals in noisy environments by gradually increasing the noise ratio and volume.

[0092] The selection of noise data is also crucial. Noise distribution varies across different scenarios, so it's necessary to choose noise data that matches the target application scenario to perform data augmentation and ultimately train the model. For specific scenarios, noise sampled from the real environment can be used, or noise generated through computer simulation can be employed. Furthermore, to achieve better training results, noise preprocessing can be performed appropriately, such as noise reduction or volume reduction, to improve noise quality and usability.

[0093] The specific operation involves adding noise to the audio during training. For a single audio track, it forms a positive sample pair with its noise-enhanced frequencies, and negative sample pairs with other audio tracks and their noise enhancements. Each audio track obtains its own feature vector through a feature extraction module, and the cosine similarity between them is calculated to construct a contrastive loss. The higher the similarity of positive sample pairs and the lower the similarity of negative sample pairs, the lower the loss. Through loss constraints, the goal is to increase the feature similarity of positive sample pairs while reducing the similarity of negative sample pairs. The contrastive loss is used as part of the total loss.

[0094] This implementation example collects audio from the CET-4 and CET-6 oral exams, including student readings, dialogues, and environmental recordings. A noise dataset is constructed based on the environmental recordings and used as noise audio for data augmentation, making it more suitable for oral recognition scenarios in the exam room.

[0095] In sequence tasks such as speech and natural language processing, multiple losses are often used to jointly optimize model performance. In this invention, the final loss consists of two parts: the CTC loss and the contrastive loss between positive samples. The former is a sequence learning loss function, often used for tasks with inconsistent input and output lengths, while the latter aims to measure and reduce the differences between positive samples and is widely used in tasks such as classification and clustering. The weights of the two are controlled by α and β, and the weight parameters can be adjusted based on experimental results.

[0096] This invention analyzes the characteristics and environment of oral exams and designs a complete training and reasoning process with the primary goal of enhancing noise resistance. This invention helps improve the accuracy and reliability of oral exam scoring systems, providing a better user experience for both test takers and examiners.

[0097] To reduce the impact of noise features on deep models, this invention understands, extracts, and fuses audio features at both fine-grained and global levels. It extracts more robust feature representations through multi-scale fusion. Furthermore, it proposes a learnable hybrid attention mechanism in both scale and temporal dimensions to compute the importance of different scales at each temporal level, resulting in a more reasonable fused feature representation.

[0098] To extract sequence information more effectively and accurately, this invention employs a positive and negative ensemble voting approach during decoding to enhance the model's contextual modeling capabilities, thereby further improving the model's performance.

[0099] This invention designs a joint noise enhancement training paradigm based on contrastive learning, which effectively improves the model's noise resistance. Real-world exam room noise is collected and superimposed on the training data to construct positive sample pairs for comparative training, enabling the model to better adapt to noisy environments.

[0100] Example 2

[0101] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0102] Example 3

[0103] The purpose of this embodiment is to provide a computer-readable storage medium.

[0104] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0105] Example 4

[0106] The purpose of this embodiment is to provide a spoken English recognition system based on contrastive learning and hybrid attention, including:

[0107] The noise superposition module is configured to: acquire the audio of the spoken English test, superimpose random categories of noise based on the environmental recording to achieve noise enhancement and then construct positive samples;

[0108] The feature extraction module is configured to extract features from noisy data based on multi-scale and hybrid attention pairs.

[0109] The encoding and decoding module is configured to: input the features after embedding and positional encoding into the encoder for context modeling;

[0110] The encoder output and the embedded and position-encoded target features are input into the decoder to complete the decoding.

[0111] The training module is configured to: calculate the contrastive loss using positive sample input during training, calculate the loss for each sample simultaneously, backpropagate, and obtain the recognition model.

[0112] The recognition module is configured to input the candidate's audio to be transcribed into the recognition model and obtain the recognition result.

[0113] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0114] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0115] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.< / eos> < / ss> < / ss> < / ss> < / eos> < / ss>

Claims

1. A method for spoken English recognition based on contrastive learning and hybrid attention, characterized in that, include: The audio of an oral English test is obtained, and noise enhancement is achieved by superimposing random categories of noise on the environmental recording to construct positive samples. Feature extraction is performed on noisy data based on multi-scale and hybrid attention methods, specifically as follows: Extracting audio features from data with added noise; The extracted audio features are convolved with different window sizes to reduce dimensionality and then concatenated to obtain multi-scale feature representations. A hybrid attention mechanism based on temporal and scale is used to calculate the attention score for each temporal pair of features at different scales; The feature of each dimension is calculated by weighting the attention score, and the multi-scale fusion feature after hybrid attention calculation is obtained. Features are concatenated along the scale dimension and the dimensionality is reduced by convolution to obtain a feature map of size (t, c, L), where L is the number of different scales, t is the number of time series at a single scale, and c is the feature dimension at a single time series position. The process involves three branches to obtain q, k, and v vectors of dimension (t*L, 1). The v vector is obtained through max pooling, while the q and k vectors are obtained through convolution dimensionality reduction. The q vector is transposed and then multiplied with the k vector to generate an attention matrix of shape (t*L, t*L). Finally, the attention matrix is ​​multiplied with the v vector to obtain a scaled attention score of size (t*L, 1). The features of each dimension are calculated based on attention score weighting to obtain the multi-scale fusion features after hybrid attention calculation, specifically: After smoothing and extracting information from the q, k, and v vectors through two fully connected layers, the temporal and scale dimensions are split, and the attention score is normalized by softmax on the scale dimension to represent the importance of each scale in each temporal sequence. The attention map is obtained by calculating the feature value of each time step in the multi-scale feature map using the normalized attention score in a weighted summation manner. The features are embedded and positionally encoded before being input into the encoder for context modeling. The encoder output and the embedded and position-encoded target features are input into the decoder to complete the decoding. During training, the contrastive loss is calculated using positive sample inputs, and the loss for each sample is calculated simultaneously. This process is then reversed to obtain the recognition model. The candidate's audio to be transcribed is input into the recognition model to obtain the recognition result.

2. The spoken English recognition method based on contrastive learning and hybrid attention according to claim 1, characterized in that, Before performing feature extraction on the noisy data based on multi-scale and hybrid attention, audio feature extraction is also included, specifically: The features are windowed, with each window considered as a frame. Then, a fast Fourier transform is performed on each frame to obtain frequency features. Finally, the frames are stacked in the time dimension to obtain the spectrogram.

3. The spoken English recognition method based on contrastive learning and hybrid attention as described in claim 1, characterized in that, During training, the decoder input is the encoder output and the target sequence after feature embedding and position encoding. During inference, the decoder obtains the probability distribution of each time step. The features obtained at each time step are processed to obtain a classification vector, which corresponds to the probability of selecting each word at that time step. After obtaining the probability matrix of each time step, the decoding algorithm searches for the best text sequence to obtain the recognition result.

4. The spoken English recognition method based on contrastive learning and hybrid attention as described in claim 1, characterized in that, The decoder uses a parallel decoder with both forward and reverse directions to replace the LSTM in the original Conformer. The reverse decoder is trained by inputting the opposite target sequence.

5. A spoken English recognition system based on contrastive learning and hybrid attention, employing the spoken English recognition method based on contrastive learning and hybrid attention as described in any one of claims 1-4, characterized in that, include: The noise superposition module is configured to: acquire the audio of the spoken English test, superimpose random categories of noise based on the environmental recording to achieve noise enhancement and then construct positive samples; The feature extraction module is configured to extract features from noisy data based on multi-scale and hybrid attention pairs. The encoding and decoding module is configured to: input the features after embedding and positional encoding into the encoder for context modeling; The encoder output and the embedded and position-encoded target features are input into the decoder to complete the decoding. The training module is configured to: calculate the contrastive loss using positive sample input during training, calculate the loss for each sample simultaneously, backpropagate, and obtain the recognition model. The recognition module is configured to input the candidate's audio to be transcribed into the recognition model and obtain the recognition result.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in any one of claims 1-4 above.

Images