Dialect speech recognition method and system in vehicle-mounted scene

Abstract

The invention discloses a dialect speech recognition method and system in a vehicle-mounted scene, and belongs to the technical field of speech recognition. Aiming at challenges such as noise, dialect diversity and speaker overlapping in a complex environment in a vehicle, a systematic solution from data acquisition and data preprocessing to model training and decoding is provided. In the data acquisition stage, diversified high-quality voice data are acquired by arranging a far-field microphone, wearing a near-field microphone and recording noise in a vehicle, and text and speaker timestamp labeling is carried out. According to the method, the performance of the voice recognition system in a complex vehicle-mounted scene is remarkably improved, and the method is suitable for practical application scenes such as vehicle-mounted navigation and voice assistants under the dialect condition.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and provides a dialect speech recognition method and system in a vehicle-mounted scenario. The method aims to train an efficient speech recognition system that can adapt to complex vehicle-mounted scenarios and dialogue modes by utilizing speech data unique to the vehicle-mounted scenario. Background Art

[0002] With the continuous development of deep learning, speech recognition technology has ushered in revolutionary progress. The introduction of deep neural networks has significantly improved speech recognition performance. By using multi-layer neural networks to model speech features, major breakthroughs have been made in accuracy and generalization.

[0003] As deep learning continues to advance, convolutional neural networks and attention mechanism neural networks have emerged. Convolutional neural networks can effectively extract the spatiotemporal features of speech signals, while attention mechanism neural networks better capture long-range temporal dependencies. These two architectures have shown strong expressiveness in the field of speech recognition, providing more possibilities for the accuracy and robustness of the system.

[0004] In recent years, the rise of pre-trained models has led to a new trend in speech recognition technology. By pre-training on large-scale data, the model learns a more general speech representation. This makes it possible to fine-tune on small-scale data, improving the performance of speech recognition systems on specific tasks. The introduction of pre-trained models such as Wav2vec, Hubert, and WavLM has brought deeper contextual understanding and context perception to speech recognition.

[0005] In the field of in-vehicle speech recognition, we face challenges from multiple aspects such as noise and dialects. In the in-vehicle scenario, noise is an issue that cannot be ignored. There may be various noises from the engine, wind noise, and passengers' conversations in the car, which puts higher requirements on traditional speech recognition systems. To meet this challenge, researchers use the time-frequency domain feature extraction capabilities of deep learning models to improve the recognition performance of speech signals in complex and noisy environments. On the other hand, the diversity of dialects and accents also puts complex requirements on the speech recognition system, especially when the vehicle travels through different regions. This is very obvious. Traditional models may perform poorly in specific areas due to the limitations of dialect diversity.

[0006] To solve these problems, researchers have also introduced the concept of self-supervised learning, which optimizes the model by learning speech representation features to enhance the robustness of the system. However, for specific in-vehicle speech recognition tasks, it is still very difficult to efficiently adjust these models to adapt to new scenarios. Summary of the invention

[0007] In the face of challenges in in-vehicle speech recognition, such as in-vehicle noise, dialect diversity and speaker overlap, the present invention proposes a dialect speech recognition method and system in an in-vehicle scenario, aiming to improve the accuracy of speech recognition in an in-vehicle scenario to cope with environmental noise, dialect differences and scenarios where multiple people speak at the same time, thereby achieving better speech recognition effects in different regions and complex environments.

[0008] The present invention proposes a dialect speech recognition method and system in a vehicle-mounted scenario, comprising the following steps: data collection, data preprocessing, model training and model decoding.

[0009] For the data collection part, the following detailed steps are included:

[0010] During normal driving of the vehicle, the driver and passengers can freely have conversations in the vehicle;

[0011] A far-field microphone is set near each door (for example, four doors) to collect far-field real data;

[0012] Each speaker (at least 2 and at most 4) is equipped with a near-field microphone worn on the neckline to capture near-field speech data;

[0013] When no one is talking in the car, these four far-field microphones are used to collect far-field noise data;

[0014] The collected far-field real data, near-field speech data and far-field noise data are summarized and stored as WAV format files with a sampling rate of 16kHz and a depth of 16 bits;

[0015] The above speech content is annotated with text and speaker timestamp.

[0016] For the data preprocessing part, the following detailed steps are included:

[0017] The near-field speech data is segmented by timestamp, and the DNSMOS model is used to filter out data with low audio quality, and the Paraformer model is used to filter out data with low annotation quality;

[0018] The far-field simulated voice data is synthesized based on the near-field voice data. The Pyroomacoustics tool is used to simulate the reverberation in the car. The far-field noise is randomly selected and the signal-to-noise ratio of the noise superposition is controlled to simulate the noise in the car. The simulation obtains 96 times the data volume of the far-field simulated voice data.

[0019] The far-field real speech data is segmented according to the marked speaker timestamps and whether there is speaker overlap, thereby obtaining far-field real speech data with speaker overlap and far-field real speech data without speaker overlap.

[0020] For the model training part, it is divided into 4 stages, including the following detailed steps:

[0021] In the first stage, near-field speech data is used to continue the pre-training process based on the open source Chinese pre-trained HuBERT-LARGE model, so that the model can learn the distribution characteristics of the target scene audio;

[0022] In stage 2, near-field speech data and corresponding text annotations are used. A 12-layer Conformer network is connected as an encoder and a 6-layer Transformer network is connected as a decoder to the HuBERT-LARGE model in stage 1 to form the entire speech recognition model. The above data is used for fine-tuning to enable the model to learn the speech recognition capabilities of the target scenario.

[0023] In the third stage, far-field simulated speech data is used to further fine-tune the model weights obtained in the second stage. The model learns the complex and diverse vehicle environment through 96 times the simulated data.

[0024] In stage 4, far-field real speech data is used, and based on the model weights obtained in stage 3, fine-tuning is performed for the situations of speaker overlap and non-overlap. The audio of the speaker overlap is processed by the speech separation model to obtain two models: Model 4-1 is used to process the audio after speech separation, and Model 4-2 is used to process the audio without speech separation, so that the model has the ability to handle speaker overlap.

[0025] For the model decoding part, the following detailed steps are included:

[0026] Different decoding strategies are used for different usage scenarios: If the target scenario provides speaker timestamps, the audio is divided into speaker overlapping segments and non-overlapping segments according to the timestamps, and speech separation is performed on the overlapping audio segments. Then, model 4-1 and model 4-2 are used to decode the speaker overlapping and non-overlapping parts respectively, and the results are summarized; if the target scenario does not provide speaker timestamps, speech separation is performed on all audio, and model 4-1 is used to decode to obtain the results; if there is less speaker overlap in the target scenario, the nearest far-field microphone signal is directly selected and decoded using model 4-2.

[0027] Beneficial Effects

[0028] The speech recognition training method for vehicle-mounted scenarios proposed in the present invention has the following beneficial effects:

[0029] Adapt to complex environmental noise: By collecting real far-field noise data in vehicle-mounted scenarios and combining simulation to generate large-scale far-field speech data, the model can effectively adapt to complex environmental noise including engine noise, wind noise and passenger conversations, significantly improving the robustness of speech recognition in vehicle-mounted scenarios;

[0030] Improve dialect and accent recognition capabilities: During the model training phase, by using diverse voice data and fine-tuning the pre-trained model, the voice recognition system can better adapt to the differences in dialects and accents in vehicle scenarios, enhancing the versatility of the model in cross-regional scenarios.

[0031] Normal use in scenarios with overlapping speakers: By introducing speaker timestamp annotation and speech separation, decoding models are designed for overlapping and non-overlapping scenarios respectively, which improves the accuracy and adaptability of the speech recognition system in scenarios with multiple people speaking at the same time;

[0032] Improve model training efficiency: Adopt a continuous pre-training strategy based on the HuBERT model, combined with staged fine-tuning of the Conformer and Transformer networks, and make full use of the distribution characteristics of vehicle scene data, which greatly shortens the model training time and improves performance;

[0033] Improve the flexibility of decoding strategies: Design a variety of decoding strategies based on actual usage scenarios, and dynamically select the optimal solution for different vehicle environments and needs, further improving the practicality and adaptability of speech recognition.

[0034] Efficiency and accuracy of data processing: DNSMOS and Paraformer models are used to screen audio and annotated data to ensure high data quality for input into the model, reducing the negative impact of low-quality data on training results and further enhancing the system's recognition capabilities;

[0035] In summary, the present invention significantly improves the accuracy, robustness and adaptability of speech recognition in vehicle-mounted scenarios, and provides technical support for the application of vehicle-mounted scene audio transcription and vehicle-mounted human-computer interaction systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to make the description of the implementation mode of the present invention clearer, the following will introduce the legends needed to describe the examples of the present invention.

[0037] Figure 1 A detailed diagram of the segmentation and cleaning process when the present invention uses near-field voice data;

[0038] Figure 2 A detailed diagram of the far-field simulated speech data synthesis method of the present invention;

[0039] Figure 3 Schematic diagram of the structure of the self-supervised pre-training model HuBERT used in the present invention;

[0040] Figure 4 It is the overall training flow chart of the present invention;

[0041] Figure 5This is the overall network structure diagram of the present invention. DETAILED DESCRIPTION

[0042] The following will describe the implementation methods of the present invention in detail in the order of data collection, data preprocessing, model training and model decoding in conjunction with the accompanying drawings.

[0043] The specific implementation method of the present invention is described in detail below. Figures 1 to 4 The technical implementation process of the entire invention is described below. The overall system construction process is as follows: Figure 1 As shown in the figure, in the data preprocessing stage, DNSMOS and Paraformer models are used to screen data quality, and large-scale far-field data is generated in combination with in-vehicle reverberation and noise simulation, and the far-field speech data is segmented; a four-stage strategy is adopted in the model training stage: continued pre-training based on near-field speech data, model fine-tuning combined with Conformer and Transformer architectures, efficient learning using far-field simulation data, and special optimization for real speech, to build models suitable for speaker overlapping and non-overlapping scenarios respectively; in the model decoding stage, different decoding strategies are selected according to the speaker timestamp or its characteristics provided by the target scenario to improve the recognition accuracy.

[0044] For the data collection part, it is achieved through the following methods:

[0045] 2-4 people are allowed to have a conversation freely during normal driving, and each conversation lasts 30-60 minutes;

[0046] Far-field microphones are installed near the four doors in the car to collect far-field voice and noise data;

[0047] Each vehicle occupant is fitted with a near-field microphone on their collar to capture high-quality near-field voice data;

[0048] When no one is talking in the car, collect the background noise inside the car, including engine noise, wind noise and road noise;

[0049] All collected data is uniformly stored as WAV format files with a sampling rate of 16kHz and a depth of 16 bits to ensure audio quality;

[0050] The obtained voice data is manually annotated to record the specific voice content;

[0051] The Silero VAD model is used to perform preliminary active speech timestamp annotation, followed by manual proofreading and differentiation of timestamps of different speakers to clarify the speaking time range of each speaker, facilitating subsequent processing of overlapping speech.

[0052] For the data preprocessing part, it is achieved in the following ways:

[0053] For the processing of near-field speech data, Figure 1 , containing the following: According to the timestamps of the marked speakers, the speech data is divided into independent speech segments;

[0054] The DNSMOS model is used to evaluate the quality of speech signals, and speech segments with poor sound quality and a DNSMOS score below 2.5 are removed;

[0055] The Paraformer model is used to evaluate the quality of text annotations. Speech segments whose word accuracy between the annotation results and the Paraformer model transcription results is less than 25% are removed. The word accuracy calculation formula is as follows:

[0056]

[0057] The processed high-quality near-field speech data is sorted and stored in folders according to the long audio segments before segmentation, providing a clean and accurate data source for subsequent model training.

[0058] For the synthesis of far-field simulated speech data, Figure 2 , containing the following:

[0059] By randomly controlling the reverberation size, noise type, and signal-to-noise ratio within a given range, the requirements for far-field data simulation can be met;

[0060] Use the Pyroomacoustics tool to simulate the reverberation effect based on the in-car environment, control the room impulse response coefficient to 0.075-0.2, and randomly select 6 times to obtain audio data of the real in-car acoustic scene with 6 times the data volume;

[0061] Randomly select far-field background noise data segments, and randomly select 4 different background noises from 60 noise segments for superposition, so as to obtain 4 times the amount of audio data superimposed with the real in-car noise scene;

[0062] The selected background noise data is superimposed on the speech signal with different signal-to-noise ratios, and the signal-to-noise ratio is controlled to be in the range of -10 to 5. Four random selections are performed, thereby obtaining four times the amount of diversified in-car speech data;

[0063] Each segment of near-field speech data is simulated multiple times using different reverberation and noise superposition schemes, and ultimately expanded to obtain 96 times the scale of far-field simulated speech data.

[0064] For the model training part, it is implemented in the following way:

[0065] The open source Chinese HuBERT-LARGE pre-trained model is selected as the basis for subsequent training. The HuBERT model structure is as follows Figure 3 ;

[0066] The convolutional neural network encoder part of the HuBERT-LARGE model uses a 7-layer 1D convolution processing module to encode speech into frame-level abstract features. The number of channels in each convolution processing module is 512, the sizes of the 7-layer convolution kernels are {10, 3, 3, 3, 2, 2}, and the convolution step sizes are {5, 2, 2, 2, 2, 2}.

[0067] The Transformer network part of the HuBERT-LARGE model has 24 layers of Transformer modules, with an input dimension of 1024, an internal feedforward neural network dimension of 4096 in each module, 16 attention heads, a final output dimension of 768, and 317M model parameters.

[0068] For the following four-stage training process, we selected the open source framework Espnet for training. The detailed training process is as follows: Figure 4 ;

[0069] In the first stage, the segmented and cleaned near-field speech data is sent to the HuBERT model without labels for continued pre-training. The Adam optimizer and warm-up learning rate strategy are used for optimization, and the number of training data rounds is set between 10 and 20.

[0070] In stage 2, based on the HuBERT weights of stage 1, a 12-layer Conformer encoder and a 6-layer Transformer decoder are connected to build a speech recognition model. The output of the HuBERT model is mapped to a 1024-dimensional model through a linear layer as the input of the Conformer encoder. The input dimension of each Conformer module is 1024, the internal feedforward neural network dimension in each module is 2048, the number of attention heads is 8, and the final output dimension is 512. The input dimension of each Transformer module of the decoder is 1024, the internal feedforward neural network dimension in each module is 2048, and the number of attention heads is 8, which constitute the entire model. Speed ​​perturbation and spectral enhancement strategies are used during training, and near-field speech data and its text annotations are used for fine-tuning.

[0071] In the third stage, based on the model weights of the second stage, the model is further optimized using 96 times expanded far-field simulation voice data to enable it to have voice recognition capabilities in complex in-vehicle environments.

[0072] In stage 4, based on the model weights of stage 3, the GSS model is used to perform multi-channel speech separation on the audio of the overlapping speakers for far-field real speech data. Then, the model is fine-tuned according to whether there is speaker overlap, and a model for processing overlapping speech (Model 4-1) and a model for processing non-overlapping speech (Model 4-2) are obtained respectively.

[0073] For the model decoding part, it is implemented in the following way:

[0074] If the target scene provides speaker timestamps, the audio is segmented by timestamps, speech separation is performed on the overlapping segments of speakers, and the overlapping and non-overlapping audio segments are decoded using models 4-1 and 4-2 respectively, and the results are merged; if the target scene does not provide timestamps and there is a lot of speaker overlap, speech separation is performed on all audio, and the final result is obtained by decoding using model 4-1;

[0075] If the target scene does not provide a timestamp and there is little speaker overlap, the nearest far-field microphone signal is directly selected and decoded using model 4-2.

[0076] The test scenario of the model is: in a normal driving environment, 2-4 speakers in the car speak freely, and the speakers speak in dialects, and the audio of each speaker in the car is required to be transcribed. The performance of the model trained by this strategy is compared with the classic end-to-end model Branchformer trained from scratch and the HuBERT model with the same structure fine-tuned directly with far-field speech after speech separation as follows:

[0077] Table 1 Performance comparison with other models

[0078] Verification set word error rate↓ Test set word error rate↓ Branchformer 32.9 26.2 HuBERT Direct Fine-tuning 26.3 21.7 HuBERT four-stage fine-tuning (Ours) 23.5 18.2

[0079] This training strategy is constantly improved during the iteration process, gradually alleviating various problems encountered in speech recognition in vehicle scenarios. The model performance obtained at different stages is as follows, among which the first stage does not have speech recognition capabilities:

[0080] Table 2 Comparison of model performance at different stages in this training strategy

[0081] Verification set word error rate↓ Test set word error rate↓ HuBERT second stage fine-tuning 25.3 21.2 HuBERT third stage fine-tuning 25.8 20.7 HuBERT fourth stage fine-tuning 23.5 18.2

[0082] In summary, the speech recognition training method for in-vehicle scenarios proposed in this invention significantly improves the performance of the speech recognition model in complex in-vehicle scenarios through a phased fine-tuning strategy and a flexible decoding scheme. Experimental results show that this method not only outperforms the classic end-to-end model and the pre-trained model with direct fine-tuning in terms of overall word error rate, but also solves practical problems such as noise interference, dialect accent differences, and speaker overlap in in-vehicle scenarios by gradually optimizing different stages of the model.

[0083] The training method of the present invention has broad application prospects in the field of in-vehicle speech recognition, and can provide efficient and accurate speech recognition solutions for fields such as in-vehicle scene audio transcription and in-vehicle human-computer interaction.

Claims

1. A dialect speech recognition method in a vehicle-mounted scenario, characterized in that: The following steps are involved: Data collection: Microphones are set up at different locations in the car to collect near-field voice data, far-field noise data, and far-field voice data; Data preprocessing: Divide the collected data into segments, clean it into clean data, and perform vehicle scene simulation and speaker separation; Model training: Speech recognition training is based on the open source HuBERT model, which is divided into four stages. Each stage initializes parameters based on the previous stage. During the model training process, the processed audio data is used for four-stage training and optimization. Stages 1 and 2 target dialect issues, stage 3 targets noise issues, and stage 4 targets speaker overlap issues. In stage 1, the segmented and cleaned near-field speech data is used to continue pre-training based on the open source HuBERT model weights; In stage 2, the segmented and cleaned near-field speech data is used, and based on the HuBERT model weights in stage 1, the speech recognition downstream task module is added for fine-tuning. In stage 3, the far-field simulated speech data is used to further fine-tune the model weights obtained in stage 2. In stage 4, far-field real speech data is used to fine-tune the model weights obtained in stage 3, respectively for the cases of speaker overlap and non-overlap. The audio of the speaker overlap is processed by the speech separation model to obtain two models: a model for processing the audio after speech separation and a model for processing the audio without separation. Model decoding: According to the requirements of the in-vehicle scenario, the audio model after speech separation and the audio model without separation obtained in stage 4 training are used for decoding, and the results are integrated to obtain the final recognition result.

2. The method according to claim 1, characterized in that In the data collection process of the method, the driver and passengers in the car talk freely in the car; a far-field microphone is installed near any door to collect far-field real data; Each speaker wears a near-field microphone on his collar to capture near-field speech data; when no one is speaking in the car, the far-field microphone is used to collect far-field noise data.

3. The method according to claim 1, characterized in that The method performs in-depth processing on the collected data during data preprocessing, including: Segment and clean near-field voice data, and use open source models to filter out high-quality audio and text data; Synthesize far-field simulated speech data and generate diversified data by randomly selecting room impulse response coefficients, noise data numbers, and signal-to-noise ratios within a given range multiple times to augment the data set; For far-field real speech data, different strategies are adopted for processing according to whether there is speaker overlap to ensure that the speech of each speaker can be accurately captured.

4. The method according to claim 1, characterized in that The method uses different models for decoding according to different scene requirements during the model decoding process, including: If the scenario provides speaker timestamps, the audio is segmented into speaker overlapping segments and non-overlapping segments, speech separation is performed on the overlapping audio segments, and then the audio model that processes speech separation and the audio model that processes without separation are used to decode the speaker overlapping and non-overlapping parts respectively, and the results are summarized; If the scene does not provide the speaker timestamp, perform speech separation on all audios, and use the audio model after speech separation to decode and obtain the result; If there is little overlap in the speakers in the scene, directly select the nearest far-field microphone signal and use the audio model decoding without separation to obtain the result.

5. A dialect speech recognition system in a vehicle-mounted scenario, characterized in that: It consists of a data acquisition module, a data preprocessing module, a model training module and a model decoding module; The data acquisition module sets microphones at different locations in the vehicle to collect near-field voice data, far-field noise data and far-field voice data; The data preprocessing module divides the collected data into segments, cleans it into clean data, performs vehicle scene simulation and speaker separation; The model training module performs speech recognition training based on the open source HuBERT model, which is divided into four stages, and each stage performs parameter initialization based on the previous stage; During the model training process, the processed audio data is used for four-stage training and optimization. Stages 1 and 2 target dialect issues, stage 3 targets noise issues, and stage 4 targets speaker overlap issues. In stage 1, the segmented and cleaned near-field speech data is used to continue pre-training based on the open source HuBERT model weights; In stage 2, the segmented and cleaned near-field speech data is used, and based on the HuBERT model weights in stage 1, the speech recognition downstream task module is added for fine-tuning. In stage 3, the far-field simulated speech data is used to further fine-tune the model weights obtained in stage 2. In stage 4, far-field real speech data is used to fine-tune the model weights obtained in stage 3, respectively for the cases of speaker overlap and non-overlap. The audio of the speaker overlap is processed by the speech separation model to obtain two models: a model for processing the audio after speech separation and a model for processing the audio without separation. The model decoding module uses the audio model trained in stage 4 for processing speech separation and the audio model for processing unseparated audio to perform decoding according to the requirements of the in-vehicle scenario, and integrates the results to obtain the final recognition result.

Images