A dialect speech recognition method and system in a vehicle-mounted scenario

Abstract

The present invention discloses a dialect speech recognition method and system in an in-vehicle scenario, belonging to the field of speech recognition technology. In response to the challenges of noise, dialect diversity, and speaker overlap in the complex environment of the vehicle, a systematic solution from data acquisition, data preprocessing to model training and decoding is proposed. In the data acquisition stage, by arranging far-field microphones, wearing near-field microphones, and recording the noise in the vehicle, a variety of high-quality voice data is obtained, and text and speaker timestamps are annotated. The present invention significantly improves the performance of the speech recognition system in complex in-vehicle scenarios and is suitable for practical application scenarios such as in-vehicle navigation and voice assistants in the case of dialects.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and provides a method and system for dialect speech recognition in an in-vehicle scenario. The method aims to train an efficient speech recognition system that can adapt to complex in-vehicle scenarios and dialogue modes by utilizing speech data unique to the in-vehicle scenario. Background Art

[0002] With the continuous development of deep learning, speech recognition technology has ushered in revolutionary progress. The introduction of deep neural networks has significantly improved speech recognition performance. By using multi-layer neural networks to model speech features, significant breakthroughs have been achieved in accuracy and generalization.

[0003] With the continuous advancement of deep learning, convolutional neural networks and attention-based neural networks have emerged. Convolutional neural networks effectively extract the spatiotemporal features of speech signals, while attention-based neural networks better capture long-range temporal dependencies. These two architectures have demonstrated powerful performance in speech recognition, providing greater potential for improving system accuracy and robustness.

[0004] In recent years, the rise of pre-trained models has ushered in a new trend in speech recognition technology. By pre-training on large-scale data, models learn more general speech representations. This enables fine-tuning on smaller datasets, improving the performance of speech recognition systems on specific tasks. The introduction of pre-trained models such as Wav2vec, Hubert, and WavLM has brought deeper contextual understanding and awareness to speech recognition.

[0005] In the field of in-vehicle speech recognition, we face multiple challenges, including noise and dialects. Noise is a significant issue in in-vehicle scenarios. The diverse noise levels inside a vehicle, including engine noise, wind noise, and passenger conversations, place higher demands on traditional speech recognition systems. To address this challenge, researchers are leveraging the time-frequency domain feature extraction capabilities of deep learning models to improve speech recognition performance in complex and noisy environments. Furthermore, the diversity of dialects and accents also places complex demands on speech recognition systems, especially when vehicles travel across diverse regions. Traditional models may perform poorly in specific regions due to the limited diversity of dialects.

[0006] To address these issues, researchers have introduced the concept of self-supervised learning, which uses models to learn speech representation features for optimization and enhance system robustness. However, for specific in-vehicle speech recognition tasks, efficiently adapting these models to new scenarios remains challenging. Summary of the Invention

[0007] In response to the challenges in in-vehicle speech recognition, such as in-vehicle noise, dialect diversity, and speaker overlap, the present invention proposes a dialect speech recognition method and system for in-vehicle scenarios, aiming to improve the accuracy of speech recognition in in-vehicle scenarios to cope with environmental noise, dialect differences, and scenarios where multiple people speak at the same time, thereby achieving better speech recognition effects in different regions and complex environments.

[0008] The present invention proposes a dialect speech recognition method and system in a vehicle-mounted scenario, comprising the following steps: data collection, data preprocessing, model training, and model decoding.

[0009] For the data collection part, the following detailed steps are included:

[0010] During normal driving of the vehicle, the driver and passengers can talk freely in the vehicle;

[0011] Place a far-field microphone near each door (for example, four doors) to collect far-field real data;

[0012] Each speaker (at least 2 and at most 4) is required to wear a near-field microphone on their neck to capture near-field speech data.

[0013] When no one is talking in the car, these four far-field microphones are used to collect far-field noise data;

[0014] The collected far-field real data, near-field speech data, and far-field noise data are summarized and stored as WAV format files with a 16kHz sampling rate and 16-bit depth.

[0015] Perform text annotation and speaker timestamp annotation on the above speech content.

[0016] For the data preprocessing part, the following detailed steps are included:

[0017] The near-field speech data is segmented by timestamp, and the DNSMOS model is used to filter out data with low audio quality, and the Paraformer model is used to filter out data with low annotation quality.

[0018] Far-field simulated speech data was synthesized based on near-field speech data. The Pyroomacoustics tool was used to simulate in-car reverberation. Far-field noise was randomly selected and the signal-to-noise ratio of the noise superposition was controlled to simulate in-car noise. This simulation generated far-field simulated speech data with 96 times the data volume.

[0019] The far-field real speech data is segmented according to the marked speaker timestamps and whether there is speaker overlap, thereby obtaining far-field real speech data with speaker overlap and far-field real speech data without speaker overlap.

[0020] For the model training part, it is divided into 4 stages, including the following detailed steps:

[0021] In the first stage, near-field speech data is used to continue pre-training based on the open-source Chinese pre-trained HuBERT-LARGE model, allowing the model to learn the distribution characteristics of the target scene audio.

[0022] In the second phase, near-field speech data and corresponding text annotations are used. A 12-layer Conformer network is added to the HuBERT-LARGE model in the first phase as the encoder, and a 6-layer Transformer network is added as the decoder to form the entire speech recognition model. This data is used for fine-tuning, enabling the model to learn speech recognition capabilities for the target scenario.

[0023] In Phase 3, we used far-field simulated speech data to fine-tune the model weights obtained in Phase 2. This 96 times more simulated data allowed the model to learn the complex and diverse in-vehicle environment.

[0024] In stage 4, real far-field speech data is used. Based on the model weights obtained in stage 3, fine-tuning is performed for situations with overlapping and non-overlapping speakers. The audio with overlapping speakers is processed by the speech separation model to obtain two models: Model 4-1 is used to process audio after speech separation, and Model 4-2 is used to process audio without speech separation. This enables the model to handle speaker overlap.

[0025] For the model decoding part, the following detailed steps are included:

[0026] Different decoding strategies are used for different usage scenarios: If the target scenario provides speaker timestamps, the audio is divided into speaker overlapping segments and non-overlapping segments according to the timestamps, and speech separation is performed on the overlapping audio segments. Then, model 4-1 and model 4-2 are used to decode the speaker overlapping and non-overlapping parts respectively, and the results are summarized; if the target scenario does not provide speaker timestamps, speech separation is performed on all audio, and model 4-1 is used for decoding to obtain the results; if there is little speaker overlap in the target scenario, the nearest far-field microphone signal is directly selected and decoded using model 4-2.

[0027] Beneficial effects

[0028] The speech recognition training method for vehicle-mounted scenarios proposed in this invention has the following beneficial effects:

[0029] Adapting to complex environmental noise: By collecting real far-field noise data from in-vehicle scenarios and combining it with simulation to generate large-scale far-field speech data, the model can effectively adapt to complex environmental noises such as engine noise, wind noise, and passenger conversations, significantly improving the robustness of speech recognition in in-vehicle scenarios.

[0030] Improving dialect and accent recognition capabilities: During model training, by using diverse voice data and fine-tuning pre-trained models, the speech recognition system can better adapt to the differences in dialects and accents in vehicle scenarios, enhancing the model's versatility across regions.

[0031] Normal use in scenarios with overlapping speakers: By introducing speaker timestamp annotation and speech separation, decoding models are designed for overlapping and non-overlapping scenarios, improving the accuracy and adaptability of the speech recognition system in scenarios with multiple speakers speaking simultaneously.

[0032] Improved model training efficiency: A continuous pre-training strategy based on the HuBERT model, combined with staged fine-tuning of the Conformer and Transformer networks, fully leverages the distribution characteristics of in-vehicle scene data, significantly shortening model training time and improving performance.

[0033] Improve the flexibility of decoding strategies: Design multiple decoding strategies based on actual usage scenarios, and dynamically select the optimal solution for different vehicle environments and needs, further improving the practicality and adaptability of speech recognition.

[0034] Efficient and accurate data processing: DNSMOS and Paraformer models are used to screen audio and annotated data, ensuring high-quality data input to the model. This reduces the negative impact of low-quality data on training results and further enhances the system's recognition capabilities.

[0035] In summary, the present invention significantly improves the accuracy, robustness, and adaptability of speech recognition in vehicle-mounted scenarios, and provides technical support for the application of vehicle-mounted audio transcription and vehicle-mounted human-computer interaction systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to make the description of the embodiments of the present invention clearer, the following will introduce the legends needed to describe the examples of the present invention.

[0037] Figure 1 This is a detailed diagram of the segmentation and cleaning process when using near-field voice data in the present invention;

[0038] Figure 2 A detailed diagram of the far-field simulated speech data synthesis method of the present invention;

[0039] Figure 3 Schematic diagram of the HuBERT structure of the self-supervised pre-training model used in the present invention;

[0040] Figure 4 This is the overall training flow chart of the present invention;

[0041] Figure 5This is the overall network structure diagram of the present invention. DETAILED DESCRIPTION

[0042] The following will describe the embodiments of the present invention in detail in the order of data collection, data preprocessing, model training and model decoding with reference to the accompanying drawings.

[0043] The specific implementation method of the present invention is described in detail below. Figures 1 to 4 The technical implementation process of the entire invention is described below. The overall system construction process is as follows: Figure 1 As shown in the figure, in the data preprocessing stage, DNSMOS and Paraformer models are used to screen data quality, combined with in-vehicle reverberation and noise simulation to generate large-scale far-field data, and the far-field speech data is segmented. In the model training stage, a four-stage strategy is adopted: continued pre-training based on near-field speech data, model fine-tuning combining Conformer and Transformer architectures, efficient learning using far-field simulation data, and special optimization for real speech to construct models suitable for overlapping and non-overlapping speaker scenarios respectively. In the model decoding stage, different decoding strategies are selected according to the speaker timestamp or its characteristics provided by the target scenario to improve recognition accuracy.

[0044] For the data collection part, it is achieved through the following methods:

[0045] 2-4 people are allowed to have a conversation with each other during normal driving, with each conversation lasting 30-60 minutes.

[0046] Far-field microphones are installed near the four doors of the vehicle to collect far-field voice and noise data;

[0047] Each vehicle occupant is fitted with a near-field microphone on their collar to capture high-quality near-field voice data.

[0048] When no one is talking in the car, collect the background noise inside the car, including engine noise, wind noise and road noise;

[0049] All collected data is uniformly stored as WAV format files with a 16kHz sampling rate and 16-bit depth to ensure audio quality;

[0050] The obtained voice data is manually annotated to record the specific voice content;

[0051] The Silero VAD model is used to perform preliminary active speech timestamp annotation, followed by manual proofreading and differentiation of timestamps of different speakers to clarify the speaking time range of each speaker, facilitating subsequent processing of overlapping speech.

[0052] For the data preprocessing part, it is achieved in the following ways:

[0053] For the processing of near-field voice data, Figure 1 , containing the following: Segment the speech data into independent segments based on the timestamps of the speakers.

[0054] The DNSMOS model is used to evaluate speech signal quality, and poor-quality speech segments with a DNSMOS score below 2.5 are removed.

[0055] The Paraformer model was used to evaluate the quality of text annotations. Speech segments with a word accuracy of less than 25% between the annotation results and the Paraformer model transcription results were removed. The word accuracy was calculated as follows:

[0056]

[0057] The processed high-quality near-field speech data is stored in folders according to the long audio segments before segmentation, providing a clean and accurate data source for subsequent model training.

[0058] For the synthesis of far-field simulated speech data, Figure 2 , containing the following:

[0059] By randomly controlling the reverberation size, noise type, and signal-to-noise ratio within a given range, the requirements for far-field data simulation can be met;

[0060] We used the Pyroomacoustics tool to simulate reverberation in a car interior, controlling the room impulse response coefficient between 0.075 and 0.2. We randomly selected six times the number of samples, generating six times the audio data of a real car interior acoustic scene.

[0061] Randomly select far-field background noise data segments, and randomly select four different background noises from the 60 noise segments for superposition, thus obtaining audio data with four times the data volume superimposed on the real car interior noise scene;

[0062] The selected background noise data is superimposed on the speech signal at different signal-to-noise ratios, with the signal-to-noise ratio range controlled between -10 and 5. This is randomly selected four times, thus obtaining four times the amount of diverse in-car speech data.

[0063] Each segment of near-field speech data is simulated multiple times using different reverberation and noise superposition schemes, ultimately expanding to obtain 96 times the scale of far-field simulated speech data.

[0064] For the model training part, it is achieved in the following way:

[0065] The open source Chinese HuBERT-LARGE pre-training model is selected as the basis for subsequent training. The HuBERT model structure is as follows Figure 3 ;

[0066] The convolutional neural network encoder of the HuBERT-LARGE model uses a 7-layer 1D convolutional processing module to encode speech into frame-level abstract features. The number of channels in each convolutional processing module is 512, and the sizes of the 7 convolution kernels are {10, 3, 3, 3, 2, 2}, and the convolution step sizes are {5, 2, 2, 2, 2, 2}.

[0067] The Transformer network of the HuBERT-LARGE model has 24 layers of Transformer modules, each with an input dimension of 1024. The internal feedforward neural network dimension of each module is 4096, the number of attention heads is 16, the final output dimension is 768, and the model parameter count is 317M.

[0068] For the subsequent four-stage training process, the open source framework Espnet is selected for training. The detailed training process is as follows Figure 4 ;

[0069] In phase 1, the segmented and cleaned near-field speech data is fed into the HuBERT model without labels for continued pre-training. The model is optimized using the Adam optimizer and a warm-up learning rate strategy, with the number of training rounds set between 10 and 20.

[0070] In stage two, based on the HuBERT weights from stage one, a 12-layer Conformer encoder and a 6-layer Transformer decoder were connected to build a speech recognition model. The output of the HuBERT model was mapped to a 1024-dimensional dimension through a linear layer as the input of the Conformer encoder. The input dimension of each Conformer module was 1024, the internal feedforward neural network dimension in each module was 2048, the number of attention heads was 8, and the final output dimension was 512. The input dimension of each Transformer module in the decoder was 1024, the internal feedforward neural network dimension in each module was 2048, and the number of attention heads was 8, forming the entire model. Speed ​​perturbation and spectral enhancement strategies were used during training, and near-field speech data and its text annotations were used for fine-tuning.

[0071] In Phase 3, based on the model weights from Phase 2, the model was further optimized using 96-fold expanded far-field simulated speech data, enabling it to recognize speech in complex in-vehicle environments.

[0072] In stage 4, based on the model weights of stage 3, the GSS model is used to perform multi-channel speech separation on the audio of the overlapping speakers for far-field real speech data. Then, fine-tuning is performed separately according to whether there is speaker overlap, and a model for processing overlapping speech (Model 4-1) and a model for processing non-overlapping speech (Model 4-2) are obtained respectively.

[0073] For the model decoding part, it is implemented in the following way:

[0074] If the target scenario provides speaker timestamps, the audio is segmented by timestamps, speech separation is performed on the overlapping segments, and the overlapping and non-overlapping segments are decoded using Model 4-1 and Model 4-2 respectively, and the results are merged. If the target scenario does not provide timestamps and there is a lot of speaker overlap, speech separation is performed on all audio, and the final decoding result is obtained using Model 4-1.

[0075] If the target scene does not provide a timestamp and there is little speaker overlap, the nearest far-field microphone signal is directly selected and decoded using model 4-2.

[0076] The model was tested in a scenario where two to four speakers spoke freely in a normal vehicle environment, with some speakers speaking in dialects. The requirement was to transcribe the audio of each speaker. The model trained using this strategy compared its performance with that of a classic end-to-end Branchformer model trained from scratch and a HuBERT model with the same structure fine-tuned directly using speech-separated far-field speech.

[0077] Table 1 Performance comparison with other models

[0078] Verification set word error rate↓ Test set word error rate↓ Branchformer 32.9 26.2 HuBERT direct fine-tuning 26.3 21.7 HuBERT four-stage fine-tuning (Ours) 23.5 18.2

[0079] This training strategy was continuously improved during the iterative process, gradually alleviating various issues encountered in speech recognition in vehicle scenarios. The model performance obtained at different stages is as follows, with the first stage lacking speech recognition capabilities:

[0080] Table 2 Comparison of model performance at different stages in this training strategy

[0081] Verification set word error rate↓ Test set word error rate↓ HuBERT second stage fine-tuning 25.3 21.2 HuBERT third stage fine-tuning 25.8 20.7 HuBERT fourth stage fine-tuning 23.5 18.2

[0082] In summary, the speech recognition training method proposed in this paper for in-vehicle scenarios significantly improves the performance of speech recognition models in complex in-vehicle scenarios through a phased fine-tuning strategy and a flexible decoding scheme. Experimental results demonstrate that this method not only outperforms both classic end-to-end models and pre-trained models using direct fine-tuning in terms of overall word error rate, but also addresses practical challenges in in-vehicle scenarios, such as noise interference, dialect differences, and speaker overlap, by gradually optimizing different stages of the model.

[0083] The training method of the present invention has broad application prospects in the field of in-vehicle speech recognition, and can provide efficient and accurate speech recognition solutions for fields such as in-vehicle scene audio transcription and in-vehicle human-computer interaction.

Claims

1. A dialect speech recognition method in a vehicle-mounted scenario, characterized in that: The following steps are involved: Data collection: Microphones are set up at different locations in the car to collect near-field voice data, far-field noise data, and far-field voice data; Data preprocessing: The collected data is divided into segments, cleaned into clean data, and then used for vehicle scene simulation and speaker separation. Model training: Speech recognition training is performed based on the open-source HuBERT model. The training is divided into four stages, with each stage initializing parameters based on the previous stage. During the model training process, the processed audio data is used for four stages of training and optimization. Stages 1 and 2 address dialect issues, stage 3 addresses noise issues, and stage 4 addresses speaker overlap issues. In the first stage, we use the segmented and cleaned near-field speech data and continue pre-training based on the open-source HuBERT model weights. In stage 2, we use the segmented and cleaned near-field speech data and fine-tune the HuBERT model weights from stage 1 by adding the speech recognition downstream task module. In stage three, we use far-field simulated speech data to fine-tune the model weights obtained in stage two. In stage 4, we use far-field real speech data to fine-tune the model weights obtained in stage 3, targeting both cases where speakers overlap and those where speakers do not overlap. The audio from the overlapping speakers is processed by the speech separation model to produce two models: one for processing the audio after speech separation and one for processing the audio without speech separation. Model decoding: Based on the requirements of the in-vehicle scenario, the audio model trained in stage 4 after speech separation and the audio model trained without speech separation are used for decoding, and the results are integrated to obtain the final recognition result.

2. The method according to claim 1, wherein During the data collection process of the method, the driver and passengers in the car speak freely in the car; a far-field microphone is installed near each door to collect far-field real data; Each speaker wears a near-field microphone on his collar to capture near-field speech data; when no one is speaking in the car, the far-field microphone is used to collect far-field noise data.

3. The method according to claim 1, wherein The method performs in-depth processing on the collected data during data preprocessing, including: Segment and clean near-field speech data, and use open-source models to filter out high-quality audio and text data; Synthesize far-field simulated speech data and generate diverse data by randomly selecting room impulse response coefficients, noise data sequence numbers, and signal-to-noise ratios within a given range multiple times to augment the dataset. For far-field real speech data, different strategies are adopted for processing depending on whether there is speaker overlap to ensure that the voice of each speaker can be accurately captured.

4. The method according to claim 1, wherein The method uses different models for decoding according to different scenario requirements during the model decoding process, including: If the scenario provides speaker timestamps, the audio is segmented into overlapping and non-overlapping segments. Speech separation is performed on the overlapping segments. The audio model that processes the speech separation and the audio model that processes the speech separation are used, respectively, to decode the overlapping and non-overlapping segments, and the results are summarized. If the scenario does not provide a speaker timestamp, perform speech separation on all audio, and use the audio model after speech separation to decode and obtain the result; If there is little speaker overlap in the scene, directly select the nearest far-field microphone signal and use the audio model decoding without separation to obtain the result.

5. A dialect speech recognition system in an in-vehicle scenario, characterized in that: It consists of data acquisition module, data preprocessing module, model training module and model decoding module; The data acquisition module sets microphones at different locations in the vehicle to collect near-field voice data, far-field noise data and far-field voice data; The data preprocessing module divides the collected data into segments, cleans it into clean data, performs vehicle scene simulation and speaker separation; The model training module performs speech recognition training based on the open source HuBERT model and is divided into four stages, with each stage initializing parameters based on the previous stage; During the model training process, the processed audio data is used for four stages of training and optimization. Stages 1 and 2 address dialect issues, stage 3 addresses noise issues, and stage 4 addresses speaker overlap issues. In the first stage, we use the segmented and cleaned near-field speech data and continue pre-training based on the open-source HuBERT model weights. In stage 2, we use the segmented and cleaned near-field speech data and fine-tune the HuBERT model weights from stage 1 by adding the speech recognition downstream task module. In stage three, we use far-field simulated speech data to fine-tune the model weights obtained in stage two. In stage 4, we use far-field real speech data to fine-tune the model weights obtained in stage 3, targeting both cases where speakers overlap and those where speakers do not overlap. The audio from the overlapping speakers is processed by the speech separation model to produce two models: one for processing the audio after speech separation and one for processing the audio without speech separation. The model decoding module uses the audio model for processing speech separation and the audio model for processing non-separated audio obtained through stage 4 training to perform decoding according to the requirements of the in-vehicle scenario, and integrates the results to obtain the final recognition result.

Images