A two-stage decoupled Chinese speech recognition model based on pinyin
By adopting the Transformer language model with two-stage decoupling design and homophone modeling in the Chinese speech recognition model, the existing end-to-end model has solved the problems of high demand for speech annotation data and high model complexity, and achieved efficient Chinese Voice recognition reduces the demand for computing resources and improves recognition accuracy.
Patent Information
- Application Number
- CN202210410441.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-19
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2042-04-19
AI Technical Summary
The existing end-to-end Chinese speech recognition model has high demands on voice labeling data, large model size, high computing and memory requirements, and high training difficulty.
Using a two-stage decoupled Chinese speech recognition model based on pinyin, through the decoupling design of acoustic model and language model, acoustic model with mixed downsampling and multipath cross-convolution structures is used, and a Transformer language model combined with homophone modeling is reduced. The need for phonetic annotation data and reduces the complexity of the model through pinyin as a modeling unit.
It greatly reduces the complexity of the model and training difficulty, saves computing resources and training time, and improves transcription accuracy, far exceeding the recognition accuracy of traditional models.
Smart Images
Figure CN114743544B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a Chinese speech recognition model, in particular to a pinyin-based two-stage decoupled Chinese speech recognition model. Background Art
[0002] Speech recognition technology is a technology that converts human speech into text sequences. As a convenient way of human-computer interaction, speech recognition technology has been widely used in various interactive intelligent devices, including smart speakers, car systems, and question-answering robots.
[0003] Traditional speech recognition architectures include acoustic models and language models. The acoustic model is responsible for converting speech input into acoustically represented output, while the language model is responsible for finding the most probable string sequence from candidate character sequences. The end-to-end model trains both the acoustic model and the language model at the same time. Although this simplifies the training process, the decoder replaces the language model in the traditional architecture, and only paired audio and text data can be used for training, which greatly increases the demand for speech annotation data. In addition, since the end-to-end model integrates the acoustic model and the language model, the model size is relatively large, which places high demands on computing and memory. Summary of the invention
[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a two-stage decoupled Chinese speech recognition model based on pinyin in view of the shortcomings of the prior art.
[0005] In order to solve the above technical problems, the present invention discloses a two-stage decoupled Chinese speech recognition model based on pinyin, comprising the following steps:
[0006] Step 1, obtain audio data from the Chinese speech data set and preprocess it to obtain a speech data training set, a validation set, and a test set; build a pinyin-based two-stage decoupled Chinese speech recognition model, including an acoustic model and a language model;
[0007] Step 2, performing dynamic data enhancement on the Mel spectrum features of the obtained speech data training set, including time masking and frequency masking;
[0008] Step 3, the Mel spectrum features after dynamic data enhancement are sent to the acoustic model, and the acoustic model is trained to obtain the connectionist temporal classification (CTC) loss and optimize the acoustic model parameters; the dynamic data enhancement and acoustic model training process described in steps 2 and 3 are repeated until the acoustic model converges;
[0009] Step 4, perform acoustic model performance evaluation;
[0010] Step 5, obtaining text data from the Chinese text dataset and performing preprocessing;
[0011] Step 6, establishing a pinyin dictionary, a Chinese character dictionary and a homophone dictionary based on the text data preprocessed in step 5, to obtain a text data training set including Chinese text;
[0012] Step 7, sending the pinyin sequence and homophone sequence corresponding to the Chinese text in the obtained text data training set to the language model, performing language model training, obtaining the cross entropy loss, and optimizing the language model parameters; repeating the language model training process described in step 7 until the language model converges;
[0013] Step 8: Perform a joint evaluation of the language model performance evaluation and the pinyin-based two-stage decoupled Chinese speech recognition model.
[0014] The data preprocessing in step 1 of the present invention includes:
[0015] All audio data are resampled at a uniform sampling rate; the audio data are pre-emphasized, framed and windowed to obtain overlapping framed signals; the framed signals are short-time Fourier transformed to obtain short-time amplitude spectra; Mel spectrum feature data are obtained through a Mel filter group; the obtained Mel spectrum feature data are divided into non-overlapping training sets, validation sets and test sets.
[0016] In the process of performing dynamic data enhancement on the Mel spectrum features of the obtained speech data training set in step 2 of the present invention, the masking ratios of time masking and frequency masking are random numbers.
[0017] In step 3 of the present invention, the acoustic model is composed of a hybrid downsampling module, a multi-path cross convolution module and a multi-layer feedforward neural network;
[0018] Among them, the hybrid downsampling module uses multi-path fusion to use maximum pooling and uniform pooling to reduce the loss of useful information in the downsampling process; the multi-path cross convolution module contains two identical groups, each of which is obtained by cascading multiple two-dimensional convolutions of different resolutions to extract low-level features under different receptive fields. The multi-path cross convolution module also uses a densely connected residual structure; the multi-layer feedforward neural network maps the input high-level features to the dimensional space of pinyin, and uses the softmax function to obtain the probability distribution.
[0019] The acoustic model performance evaluation in step 4 of the present invention includes: test set character recognition error rate and inference delay; wherein the character recognition error rate CER is calculated by dynamic string alignment, and the method is as follows:
[0020]
[0021] Among them, S represents the number of substitutions generated during the alignment process, D represents the number of deletions, I represents the number of insertions, and N represents the number of characters in the target sentence.
[0022] The Chinese text data set in step 5 of the present invention is an existing data set or a collected and collated Chinese text; the preprocessing includes:
[0023] Complex sentences are split into multiple simple sentences, numbers are converted into Chinese characters, traditional Chinese characters are converted into simplified Chinese characters, punctuation marks at the end of sentences are removed, and sentences containing non-Chinese characters are removed to obtain sentences containing only simplified Chinese characters; the obtained sentences are divided into non-overlapping training sets and validation sets, and sentences overlapping with the data set used for the final test are removed.
[0024] The dictionary described in step 6 of the present invention is stored in a python dictionary format; wherein the pinyin dictionary uses pinyin as the key and the ascending index as the value; the Chinese character dictionary uses Chinese characters as the key and the ascending index as the value; the homophone dictionary uses pinyin as the key and the sub-dictionary as the value, the key of the sub-dictionary is the Chinese character with the same pinyin, and the value is the ascending index.
[0025] The language model described in step 7 of the present invention is a Transformer model based on homophone modeling, which regards the transcription process from pinyin to Chinese characters as a one-to-one translation process and limits the translation result to the homophone space corresponding to the input pinyin.
[0026] In step 8 of the present invention, the language model performance evaluation includes the test set recognition accuracy and inference delay; the joint evaluation of the pinyin-based two-stage decoupled Chinese speech recognition model includes the size of the overall model parameters, the test set recognition accuracy and inference delay.
[0027] The speakers in the validation set and the test set in step 1 of the present invention do not appear in the training set.
[0028] The present invention uses hybrid downsampling, multi-path cross convolution and residual structure to construct an efficient full convolution acoustic model, performs recognition from the Mel spectrum features of audio to pinyin, and uses a Transformer language model based on homophone modeling to transcribe from pinyin to Chinese characters. Compared with the existing end-to-end Chinese speech recognition model, the present invention is characterized in that the decoupling of the acoustic model and the language model greatly reduces the demand for speech annotation data, and can fully utilize a large amount of existing Chinese text for self-supervised training to obtain an efficient language model; with pinyin as the modeling unit, an efficient full convolution acoustic model is established through hybrid downsampling, multi-path convolution and densely connected residual structure, which reduces the complexity and training difficulty of the model while ensuring the recognition accuracy, saving training time and computing resources; the relationship between pinyin and homophones is modeled, rather than the relationship between pinyin and all Chinese characters, and an efficient Transformer language model is established, while reducing more than half of the model parameters, achieving a transcription accuracy far exceeding other models.
[0029] Beneficial effects:
[0030] A new downsampling method that mixes maximum pooling and uniform pooling is proposed, which is much more effective than maximum pooling or uniform pooling in speech recognition tasks. The acoustic model constructed by the fully convolutional neural network enhances the extraction and fusion of shallow features in different dimensions through mixed downsampling and multi-path convolution, alleviates the gradient vanishing phenomenon through the residual structure, and strengthens the transmission of features between low and high layers. While ensuring the recognition accuracy, it greatly reduces the complexity and training difficulty of the model, saving training time and computing resources. An efficient language model is established with the homophone dictionary space as the output target space of the Transformer model, which achieves a transcription accuracy far exceeding that of other models while reducing the model parameters by more than half. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.
[0032] Figure 1 It is a schematic diagram of the overall architecture of the present invention.
[0033] Figure 2 It is a schematic diagram of the architecture of the acoustic model part of the present invention.
[0034] Figure 3 It is a schematic diagram of the architecture of the language model part of the present invention.
[0035] Figure 4 This is a schematic diagram of the data flow during training of Example 1.
[0036] Figure 5 Schematic diagram of data flow during training of Example 2.
[0037] Figure 6 It is a data flow diagram when Example 1 and Example 2 are combined for reasoning. DETAILED DESCRIPTION
[0038] like Figure 1 As shown, a two-stage decoupled Chinese speech recognition model based on pinyin includes the following steps:
[0039] Step 1, obtain audio data from the Chinese speech data set and preprocess it to obtain a speech data training set, a validation set, and a test set; build a pinyin-based two-stage decoupled Chinese speech recognition model, including an acoustic model and a language model;
[0040] The data preprocessing includes: resampling all audio data at a uniform sampling rate; pre-emphasizing, framing and windowing the audio data to obtain overlapping framed signals; performing short-time Fourier transform on the framed signals to obtain short-time amplitude spectra; obtaining Mel spectrum feature data through a Mel filter group; and dividing the obtained Mel spectrum feature data into non-overlapping training sets, validation sets and test sets. The speakers in the validation sets and test sets do not appear in the training set.
[0041] Step 2, dynamically enhance the Mel spectrum features of the obtained speech data training set, including time masking and frequency masking; in the process of dynamically enhancing the Mel spectrum features of the obtained speech data training set, the masking ratios of time masking and frequency masking are random numbers.
[0042] Step 3, the Mel spectrum features after dynamic data enhancement are sent to the acoustic model, and the acoustic model is trained to obtain the connectionist temporal classification (CTC) loss and optimize the acoustic model parameters; the dynamic data enhancement and acoustic model training process described in steps 2 and 3 are repeated until the acoustic model converges;
[0043] The acoustic model consists of a hybrid downsampling module, a multi-path cross convolution module and a multi-layer feedforward neural network;
[0044] Among them, the hybrid downsampling module uses multi-path fusion to use maximum pooling and uniform pooling to reduce the loss of useful information in the downsampling process; the multi-path cross convolution module contains two identical groups, each of which is obtained by cascading multiple two-dimensional convolutions of different resolutions to extract low-level features under different receptive fields. The multi-path cross convolution module also uses a densely connected residual structure to alleviate the gradient vanishing problem and strengthen the transmission of features between low and high layers; the multi-layer feedforward neural network maps the input high-level features to the dimensional space of pinyin, and uses the softmax function to obtain the probability distribution.
[0045] Step 4, perform acoustic model performance evaluation;
[0046] The acoustic model performance evaluation includes: test set character recognition error rate and inference delay; wherein the character recognition error rate CER is calculated by dynamic string alignment, and the method is as follows:
[0047]
[0048] Among them, S represents the number of substitutions generated during the alignment process, D represents the number of deletions, I represents the number of insertions, and N represents the number of characters in the target sentence.
[0049] Step 5, obtaining text data from the Chinese text dataset and performing preprocessing;
[0050] The Chinese text data set is an existing data set or a collected and collated Chinese text; the preprocessing includes:
[0051] Complex sentences are split into multiple simple sentences, numbers are converted into Chinese characters, traditional Chinese characters are converted into simplified Chinese characters, punctuation marks at the end of sentences are removed, and sentences containing non-Chinese characters are removed to obtain sentences containing only simplified Chinese characters; the obtained sentences are divided into non-overlapping training sets and validation sets, and sentences overlapping with the data set used for the final test are removed.
[0052] Step 6, establishing a pinyin dictionary, a Chinese character dictionary and a homophone dictionary based on the text data preprocessed in step 5, to obtain a text data training set including Chinese text;
[0053] The dictionary is stored in Python dictionary format; wherein the Pinyin dictionary uses Pinyin as key and ascending index as value; the Chinese character dictionary uses Chinese character as key and ascending index as value; the homophone dictionary uses Pinyin as key and sub-dictionary as value, the key of the sub-dictionary is the Chinese character with the same Pinyin and the value is the ascending index.
[0054] Step 7, sending the pinyin sequence and homophone sequence corresponding to the Chinese text in the obtained text data training set to the language model, performing language model training, obtaining the cross entropy loss, and optimizing the language model parameters; repeating the language model training process described in step 7 until the language model converges;
[0055] The language model is a Transformer model based on homophone modeling (references: Vaswani2017attention,Attention is all you need, Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan Nand Kaiser, Lukasz and Polosukhin, Illia, Advances in neural information processing systems, 30, 2017), which regards the transcription process from pinyin to Chinese characters as a one-to-one translation process, and limits the translation result to the homophone space corresponding to the input pinyin, thereby reducing the number of model parameters, greatly reducing the transcription error rate of the model, and simplifying the recognition process of the model.
[0056] Step 8: Perform a joint evaluation of the language model performance evaluation and the pinyin-based two-stage decoupled Chinese speech recognition model.
[0057] The language model performance evaluation includes the test set recognition accuracy and inference delay; the joint evaluation of the pinyin-based two-stage decoupled Chinese speech recognition model includes the overall model parameter size, test set recognition accuracy and inference delay.
[0058] Example 1
[0059] The present invention discloses a full convolution acoustic model based on a hybrid downsampling and multi-path cross convolution structure, such as Figure 2 As shown, this embodiment includes an encoder composed of a hybrid downsampling module and a multi-path cross convolution module and a decoder composed of a multi-layer feedforward neural network to perform conversion from the Mel spectrum of the speech signal to pinyin.
[0060] This embodiment provides a full convolution acoustic model based on a hybrid downsampling and multi-path cross convolution structure. The training process is as follows: Figure 4 The detailed process is as follows:
[0061] 1. Get audio data from the Chinese speech dataset
[0062] The data set used in this embodiment is the open source Chinese Mandarin speech data set "AISHELL-ASR0009-OS1" (Aishell-1) developed by Hill Shell Technology Co., Ltd.
[0063] 2. Data preprocessing
[0064] All audios are resampled at a uniform sampling rate of 16KHz; a first-order high-pass filter is used to pre-emphasize the audio; the audio is divided into frames with a frame shift of 10ms to obtain a frame signal with a frame length of 25ms; the frame signal is windowed using a Hamming window function; each frame signal is subjected to a short-time Fourier transform to obtain a short-time amplitude spectrum; and an 80-dimensional Mel spectrum feature is obtained through a Mel filter group.
[0065] 3. Divide the data set
[0066] The Aishell-1 dataset has already divided the data into three disjoint parts: training set, validation set, and test set, and the speakers in the validation set and test set do not appear in the training set. Therefore, this step can be omitted when training on the Aishell-1 dataset.
[0067] 4. Data Augmentation
[0068] The Mel spectrum features are randomly masked in the time domain and frequency domain, with a maximum masking ratio of 0.2. During the training process, a random number of 0 to 0.2 is generated each time data is obtained from the dataset, and the random number is used as the masking ratio for data enhancement.
[0069] 5. Calculate the CTC loss and optimize the acoustic model parameters
[0070] The obtained enhanced Mel features are sent to the acoustic model, the CTC loss is calculated, and then the parameters of the acoustic model are optimized using the AdamW optimizer.
[0071] The acoustic model is as follows Figure 2 As shown in the figure, it includes: hybrid downsampling module, multi-path cross convolution model and multi-layer feedforward neural network. The hybrid downsampling module first uses maximum pooling and uniform pooling on two parallel convolution paths to extract different feature patterns in the time domain and frequency domain, avoiding the empirical selection of pooling schemes:
[0072] f max (x)=relu(avg_pool(conv(x))),
[0073] f avg (x)=relu(avg_pool(conv(x))),
[0074] Among them, conv is a 2D convolution operation and relu is a linear rectification activation function:
[0075] relu(x)=max(x,0),
[0076] The output feature maps of the maximum pooling and average pooling are then concatenated together and fused through 1x1 convolution for feature extraction and dimensionality reduction:
[0077] y = relu(conv 1×1 (concat(f max ,f avg ))),
[0078] 1x1 convolution introduces learnable parameters that can adaptively activate and enhance important features in the output results of maximum pooling and average pooling. Hybrid downsampling preserves the advantages of maximum pooling and average pooling while making up for each other's shortcomings. Figure 2 As can be seen in the figure, the acoustic model uses a total of three hybrid downsampling modules, where each hybrid downsampling module downsamples the time domain by 2 times and the frequency domain by 2 times.
[0079] The multipath cross convolution module (reference: "Towards End-to-End Speech Recognition with Deep Multipath Convolutional Neural Networks", Towards end-to-end speech recognition with deep multipath convolutional neural networks, Zhang, Wei and Zhai, Minghao and Huang, Zilong and Liu, Chen and Li, Wei and Cao, Yi, International Conference on Intelligent Robotics and Applications, 332--341, 2019, Springer) contains two identical groups and a residual structure with dense connections between groups. In each group, three-way convolutions of different resolutions are first performed on the input feature map:
[0080] f i,j =relu(conv(x i )),i=1,2;j=1,2,3
[0081] Among them, i represents the group number, x i represents the input of the i-th group, j represents the path number, and f i,j Represents the output of the jth path in the i-th group.
[0082] Then use 1x1 convolution to fuse the features output by multiple parallel paths and compress the number of channels of the feature map:
[0083] y i =conv 1×1 (concat(f i,1 ,f i,2 ,f i,3 )),i=1,2
[0084] Among them, y i Represents the output of the i-th group. The multi-path cross convolution module introduces a densely connected residual structure on top of the two groups (reference: huang2017densely,Densely connected convolutionalnetworks,Huang,Gao and Liu,Zhuang and Van Der Maaten,Laurens and Weinberger,Kilian Q,Proceedings of the IEEE conference on computer vision and patternrecognition,4700--4708,2017):
[0085] x2=x1+y1,
[0086] y=x1+y1+y2,
[0087] Among them, x1 and y1 are the input and output of the first group, x2 and y2 are the input and output of the second group, and x1 and y are the outputs of the multi-path cross convolution module. The multi-path cross convolution module continuously performs path separation and aggregation, giving full play to the powerful feature extraction ability of multi-path, effectively extracting and fusing shallow features under different receptive fields. In addition, the introduction of densely connected residual structures ensures the transmission of features between low-level and high-level layers, alleviating the problem of gradient disappearance.
[0088] The multi-layer feedforward neural network uses two fully connected layers to map the feature dimension to the size of the pinyin space, and finally uses the softmax activation function to obtain the probability distribution of pinyin. Among them, the number of output neurons in the fully connected layer is 512-1191, forming a simplest bottleneck network.
[0089] 6. Model Performance Evaluation
[0090] The test set data is used to objectively evaluate the CTC loss of the acoustic model to evaluate the recognition accuracy, generalization ability, and inference speed of the model.
[0091] Example 2
[0092] The present invention discloses a Transformer language model based on homophone modeling, such as Figure 3 As shown, this embodiment includes an encoder composed of a self-attention module and a feedforward neural network stacked together, and a decoder composed of a mask attention module, an encoder-decoder attention module and a feedforward neural network stacked together to perform conversion from pinyin to Chinese characters.
[0093] This embodiment provides a Transformer language model based on homophone modeling. The training process is as follows: Figure 5 The detailed process is as follows:
[0094] 1. Get text data from the Chinese text dataset
[0095] The data set used in this embodiment is part of the Chinese text data in the multilingual corpus "Oscar", but any other Chinese text data set may also be used, or a Chinese text data set created by oneself may be used.
[0096] 2. Data preprocessing
[0097] The complex sentence is divided into multiple simple sentences; all Arabic numerals are converted into Chinese characters; all traditional Chinese characters are converted into simplified Chinese characters; all sentence-end punctuation marks are removed; all sentences containing non-Chinese characters are removed, and normalized plain text data containing only simplified Chinese characters is obtained; the obtained data is divided into non-overlapping training sets and validation sets, and sentences that appear in the Aishell-1 test set label text are deleted.
[0098] 3. Build a dictionary
[0099] Traverse the training set, validation set and test set to build a pinyin dictionary, a Chinese character dictionary and a homophone dictionary. All dictionaries are stored in Python dictionary format, where the pinyin dictionary uses pinyin as the key and the ascending index as the value; the Chinese character dictionary uses Chinese characters as the key and the ascending index as the value; the homophone dictionary uses pinyin as the key and the sub-dictionary as the value, the key of the sub-dictionary is the Chinese character with the same pinyin, and the value is the ascending index.
[0100] 4. Calculate the cross entropy loss and optimize the language model parameters
[0101] Each set of text data needs to be preprocessed before being input into the model: for each set of text data, first convert it into a pinyin sequence, and obtain the encoder input sequence according to the pinyin dictionary; then obtain the homophone sequence from the pinyin sequence and the text sequence according to the homophone dictionary, and add the start symbol " <sos>"Get the decoder input sequence and add a terminator at the end of the homophone sequence" <eos>"Get the decoder target sequence.
[0102] Model training: Input the encoder input sequence into the encoder of the language model to obtain the encoder output sequence; input the encoder output sequence and the decoder input sequence into the decoder of the language model to obtain the decoder output sequence; calculate the penalized cross entropy loss based on the decoder output sequence and the decoder target sequence, and use the Adam optimizer to optimize the language model parameters. The calculation formula for the penalized cross entropy loss is as follows:
[0103]
[0104] Where N represents the length of the input sequence, C i Indicates the length of the homophone sequence corresponding to the i-th pinyin in the input sequence, y ic represents the probability that the true category of the sample at position i is c, p ic represents the probability of classifying the sample at position i into category c, l max represents the decoder output dimension, that is, the maximum sequence length in the homophone dictionary, and K represents the penalty coefficient, which penalizes the part of the probability distribution of the decoder output outside the homophone sequence, and its value is 10.
[0105] 5. Model performance evaluation
[0106] Use the test set data to objectively evaluate the cross entropy loss of the Transformer language model to evaluate the recognition accuracy, generalization ability, and inference speed of the model.
[0107] The present invention provides a method and idea for a two-stage decoupled Chinese speech recognition model based on Pinyin. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.< / eos> < / sos>
Claims
1. A two-stage decoupled Chinese speech recognition model based on Pinyin, characterized in that: The steps include: Step 1, obtain audio data from the Chinese speech data set and preprocess it to obtain a speech data training set, a validation set, and a test set; build a pinyin-based two-stage decoupled Chinese speech recognition model, including an acoustic model and a language model; Step 2, performing dynamic data enhancement on the Mel spectrum features of the obtained speech data training set, including time masking and frequency masking; Step 3, the Mel spectrum features after dynamic data enhancement are sent to the acoustic model, and the acoustic model is trained to obtain the joint temporal classification loss and optimize the acoustic model parameters; the dynamic data enhancement and acoustic model training process described in steps 2 and 3 are repeated until the acoustic model converges; Step 4, perform acoustic model performance evaluation; Step 5, obtaining text data from the Chinese text dataset and performing preprocessing; Step 6, establishing a pinyin dictionary, a Chinese character dictionary and a homophone dictionary based on the text data preprocessed in step 5, to obtain a text data training set including Chinese text; Step 7, sending the pinyin sequence and homophone sequence corresponding to the Chinese text in the obtained text data training set to the language model, performing language model training, obtaining the cross entropy loss, and optimizing the language model parameters; repeating the language model training process described in step 7 until the language model converges; Step 8, performing a joint evaluation of the language model performance evaluation and the pinyin-based two-stage decoupled Chinese speech recognition model; The acoustic model performance evaluation in step 4 includes: test set character recognition error rate and inference delay; wherein the character recognition error rate CER is calculated by dynamic string alignment, and the method is as follows: Among them, S represents the number of substitutions generated during the alignment process, D represents the number of deletions, I represents the number of insertions, and N represents the number of characters in the target sentence.
2. The pinyin-based two-stage decoupled Chinese speech recognition model according to claim 1, characterized in that: Data preprocessing in step 1 includes: All audio data are resampled at a uniform sampling rate; the audio data are pre-emphasized, framed and windowed to obtain overlapping framed signals; the framed signals are short-time Fourier transformed to obtain short-time amplitude spectra; Mel spectrum feature data are obtained through a Mel filter group; the obtained Mel spectrum feature data are divided into non-overlapping training sets, validation sets and test sets.
3. A two-stage decoupled Chinese speech recognition model based on pinyin according to claim 2, characterized in that: In step 2, during the process of dynamic data enhancement of the Mel spectrum features of the obtained speech data training set, the masking ratios of time masking and frequency masking are random numbers.
4. A two-stage decoupled Chinese speech recognition model based on pinyin according to claim 3, characterized in that: In step 3, the acoustic model consists of a hybrid downsampling module, a multi-path cross convolution module and a multi-layer feedforward neural network; Among them, the hybrid downsampling module uses multi-path fusion to use maximum pooling and uniform pooling to reduce the loss of useful information in the downsampling process; the multi-path cross convolution module contains two identical groups, each of which is obtained by cascading multiple two-dimensional convolutions of different resolutions to extract low-level features under different receptive fields. The multi-path cross convolution module also uses a densely connected residual structure; the multi-layer feedforward neural network maps the input high-level features to the dimensional space of pinyin, and uses the softmax function to obtain the probability distribution.
5. A two-stage decoupled Chinese speech recognition model based on pinyin according to claim 4, characterized in that: The Chinese text dataset in step 5 is an existing dataset or a collected and collated Chinese text; the preprocessing includes: Complex sentences are split into multiple simple sentences, numbers are converted into Chinese characters, traditional Chinese characters are converted into simplified Chinese characters, punctuation marks at the end of sentences are removed, and sentences containing non-Chinese characters are removed to obtain sentences containing only simplified Chinese characters; the obtained sentences are divided into non-overlapping training sets and validation sets, and sentences overlapping with the data set used for the final test are removed.
6. A two-stage decoupled Chinese speech recognition model based on pinyin according to claim 5, characterized in that: The dictionary described in step 6 is stored in python dictionary format; among them, the pinyin dictionary uses pinyin as the key and the ascending index as the value; the Chinese character dictionary uses Chinese characters as the key and the ascending index as the value; the homophone dictionary uses pinyin as the key and the sub-dictionary as the value, the key of the sub-dictionary is the Chinese character with the same pinyin, and the value is the ascending index.
7. A two-stage decoupled Chinese speech recognition model based on pinyin according to claim 6, characterized in that: The language model in step 7 is a Transformer model based on homophone modeling, which treats the transcription process from pinyin to Chinese characters as a one-to-one translation process and restricts the translation result to the homophone space corresponding to the input pinyin.
8. A two-stage decoupled Chinese speech recognition model based on pinyin according to claim 7, characterized in that: In step 8, the language model performance evaluation includes the test set recognition accuracy and inference delay; the joint evaluation of the pinyin-based two-stage decoupled Chinese speech recognition model includes the overall model parameter size, test set recognition accuracy and inference delay.
9. The pinyin-based two-stage decoupled Chinese speech recognition model according to claim 8, characterized in that: The speakers in the validation set and the test set described in step 1 do not appear in the training set.