Automatic speech recognition method in complex environment

Abstract

The invention relates to an automatic speech recognition method in a complex environment, which comprises the following steps of: firstly, carrying out availability labeling classification on acquired speech data, and firstly excluding unavailable speech data; and then searching a comprehensive optimal VAD model to filter voice data, filtering out a non-voice part, and leaving effective voice data. Through the basic speech recognition model, preliminary automatic recognition and speaker labeling are carried out, dialects, environmental noise and background music are further recognized, and according to professional terms possibly appearing in different occasions, the actual application scene of the recognition model is generalized. Finally, secondary correction is carried out manually for model optimization, in the model optimization, through speaker voiceprint recognition, the voice of a speaker can be positioned more accurately, and environment noise and background music are regarded as individual speakers for identity recognition, so that the recognition precision and performance of the training model are remarkably improved. The method has more accurate speech recognition performance, and has better generalization at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and in particular to an automatic speech recognition method in a complex environment. BACKGROUND

[0002] Automatic Speech Recognition (ASR) technology has experienced more than half a century of development since the 1950s, and has gone through three stages of artificial acoustic feature extraction, statistical learning and deep learning. The recognition accuracy of standard speech (such as standard English or Mandarin) in a quiet environment has reached a very high level. However, in a complex environment (with environmental noise, background music, containing multiple voices, large volume difference, dialects, etc.), the accuracy of standard speech recognition model is greatly reduced. In many industries, especially in sales-related industries, there is a huge demand for technology, application and market for recording, transcribing and content analysis of the conversation between sales personnel and customers using intelligent badges and other recording devices. However, the existing general and customized technical solutions have great problems in invalid signal filtering, speaker recognition, dialect recognition, background noise separation, and term recognition, resulting in low recognition accuracy and insufficient practical effect.

[0003] The development of automatic speech recognition technology mainly goes through three stages. The early exploration stage (1950s-1990s) - this stage is mainly based on manual extraction of acoustic features and hidden Markov model for sequence modeling. The recognition accuracy of this stage is basically below 70%. Statistical model era (1990s-2010s) - the model of this stage is mainly based on N-gram language model, Gaussian mixture model, etc. Based on the progress of acoustic feature extraction, decoder improvement and adaptive technology, the recognition accuracy has been greatly improved. Deep learning era (after 2010s) - the development of deep learning technology promotes the progress of ASR technology. On the basis of previous sequence models, end-to-end models have become the mainstream models. The error rate is generally below 10% on the standardized high-quality speech data set. The recognition accuracy of standard speech has reached a high level. The recognition accuracy of standard Mandarin and English speech in a quiet environment may reach about 98%, but when entering a complex speech environment, the recognition accuracy will be greatly reduced.

[0004] There are still some deficiencies in existing voice recognition technical solutions. The first is to use a standard ASR model to recognize speech in a complex environment and remove obvious abnormal content from the transcribed text. Speech signals in a complex environment contain a lot of noise, including environmental noise, background music, non-speech signals, etc. Automatically recognizing these mixed speech signals indiscriminately will consume server computing power to process invalid speech on the one hand, and may introduce error data from non-speech parts on the other hand. The second is to add a Voice Activity Detection (VAD) technology to the standard ASR model to filter out parts that do not contain valid speech and reduce processing complexity. Using VAD can filter out invalid speech to some extent, but it cannot handle other complex environmental interference signals such as human voice signals in music and different volumes of input using multiple dialects by different speakers. The third is to use a deep learning-based speech large model to recognize complex environmental speech, based on the context reasoning ability of the built-in text large model, to generate coherent content text. For high-quality speech data in a quiet environment, using a deep learning speech model can achieve better transcription results. Some unclear pronunciation in the speech data can be probabilistically inferred based on the context reasoning ability of the embedded text model in the speech model, thereby generating coherent high-quality text. However, when the data contains multiple conversations and some of the volumes are low, the probabilistic generation mechanism of the text model may introduce hallucinated output. SUMMARY

[0005] To address the deficiencies of the prior art, an automatic speech recognition method in a complex environment first performs usability labeling and classification of speech data, matches a comprehensive optimal VAD model to filter speech data, inputs the filtered data into a speech recognition model for preliminary recognition and labeling training, a human annotator performs secondary manual labeling, and then inputs the data into the speech recognition model for optimization until the preset recognition accuracy is reached to stop training. Finally, it is applied to actual generation scenarios. The method comprises: Step 1: Based on the set application scenario, collect a large amount of dialogue data with different speech quality, upload the server, and perform usability labeling and classification of the dialogue data. The classification includes three categories: completely unusable, partially usable, and basically completely usable. Fine-grained data labeling is performed on the usable dialogue data in the latter two categories to obtain a labeled speech data set. Step 2: Obtain multiple mainstream VAD models and deploy them locally. Train them based on the labeled speech dataset from Step 1. Compare the performance of each mainstream VAD model and determine the optimal VAD model. Filter the non-speech parts in the second and third types of speech data using the optimal VAD model, retaining only the valid speech parts. Furthermore, select an appropriate threshold based on the detection accuracy index to improve the discrimination accuracy between the first, second, and third types. Step 3: Investigate basic speech recognition models, select basic speech recognition models to perform preliminary automatic speech recognition and speaker annotation on the data samples that have been filtered through speech data in Step 2, and generate speech recognition text and annotation data; Step 4: The human annotator performs secondary annotation and correction on the speech recognition text and annotation data output in Step 3, compares it with the original audio file, identifies the main errors, and obtains the corrected annotation data; Step 5: Optimize the speech recognition model selected in Step 3 based on the corrected labeled data in Step 4; Step 6: Repeat the automatic speech recognition and speaker labeling in Step 3 using the optimized speech recognition model, and perform the next round of manual verification and labeling until the preset recognition accuracy is achieved, or the recognition rate is not significantly improved. Step 7: Apply the optimized speech recognition model to the actual production environment for automatic speech recognition, and output the production environment speech and a transcribed text library containing identity recognition information; Step 8: The text information with identity recognition information extracted in Step 7 will be used in subsequent data processing, including identification of prohibited language and analysis of standard language. Step 9: The speech data and transcribed text information extracted from the actual production scenario are sampled into the case library with a set probability. The manual annotators regularly review the cases in the case library, mark obvious recognition errors, and then proceed to Step 5 to perform periodic speech recognition model optimization to dynamically update the model parameters.

[0006] According to a preferred embodiment, the manual secondary annotation method in step 4 includes: Step 41: Speaker identity verification. Verify whether the speaker's identity is correctly identified, especially in interruption scenarios during a conversation, where speaker confusion can easily occur. Step 42: Dialect recognition. Check if the dialect recognition accuracy in the dialogue is too low, and determine whether it is due to incorrect recognition of dialect type or weak dialect recognition ability. Step 43: Environmental and background noise verification to determine if there has been any incorrect transcription of environmental noise and background music; Step 44: Technical terminology check to determine if the technical terminology identification is accurate; Step 45: Check other key information. Check for obvious transcription errors in other key information. For non-key content, allow errors within the preset error rate range. Step 46: Based on the annotation process of steps 41 to 45, correct the key content to obtain the corrected annotation data, which will be used as the corrected training data.

[0007] According to a preferred embodiment, step 5, optimizing the speech recognition model, includes: Step 51: For cases with a high speaker identification error rate, add a speaker voiceprint recognition module to cluster the content in the audio signal based on the speaker's voice features in order to accurately distinguish the speech boundaries of different subjects. Step 52: For ambient noise and background music, after signal separation of the labeled data, they are treated as individual speakers for identification in order to extract interference audio information from the audio. Step 53: For cases where dialect recognition is incorrect or the accuracy is low, the corrected training data is used for model fine-tuning to improve the recognition accuracy of the corresponding dialect. Step 54: Introduce a keyword list related to the scene, and annotate its standard form and variants in the training data accordingly to ensure recognition accuracy.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Before training speech recognition, the method of the present invention first performs usability labeling and classification on the speech, excludes unusable speech data, improves the quality of speech training data, and saves server computing power.

[0009] 2. Existing technologies cannot handle different speakers and dialects well. The speech recognition model of the present invention can not only identify the speaker's identity, but also better recognize dialects, environmental noise and background music, and generalize the model's practical application scenarios according to professional terms that may appear in different occasions.

[0010] 3. The method of the present invention performs secondary correction and annotation by manual means for model optimization training. In the model optimization, the speaker's voiceprint recognition can more accurately locate the speaker's voice, and treat environmental noise and background music as separate speakers for identification, which significantly improves the recognition performance of the trained model. Attached Figure Description

[0011] Figure 1 This is a flowchart of the automatic speech recognition method of the present invention; Figure 2 This is a detailed processing flowchart of the automatic speech recognition method of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0013] The following is a detailed explanation with reference to the accompanying drawings.

[0014] The VAD technology of this invention refers to Voice Activity Detection.

[0015] Existing automatic speech recognition methods achieve high accuracy in quiet backgrounds and standard speech scenarios. However, in real-world applications, various interference factors such as background noise, music, multiple speakers, volume differences, dialectal mixing, and model illusions lead to poor transcribed text quality and limited practical applicability. This invention, using a pharmacy scenario as an example, provides a speech recognition technology solution that effectively addresses various interference factors. It can effectively eliminate invalid recordings and segments lacking valid speech, and iteratively optimize the speech recognition model to improve recognition accuracy and subsequent application effectiveness.

[0016] Step 1: Based on the defined application scenario, collect massive amounts of dialogue data with different speech quality and upload them to the server. Classify the dialogue data by availability labeling, including three categories: completely unusable, partially usable, and almost completely usable. Perform fine-grained data labeling on the latter two categories of usable dialogue data to obtain an labeled speech dataset.

[0017] The application scenarios of this invention include, but are not limited to, pharmacies, hospital outpatient clinics, emergency rooms, emergency response sites, supermarkets, large conferences, subways, high-speed rail stations, and large shopping malls. This explanation uses a pharmacy scenario as an example; similar methods can be applied in other scenarios with appropriate adjustments.

[0018] Step 11: Select mainstream recording equipment with representative hardware configurations, including pure recording headphones, smart office headphones, general-purpose work badges, professional-grade medical work badges, etc.

[0019] Step 12: In multiple pharmacy scenarios, store staff are required to record their interactions with customers using audio recording devices and upload the recordings to a designated online space.

[0020] Step 13: Based on the overall quality level of the recording files, label each audio segment as one of the following three categories: completely unusable, partially usable, and mostly fully usable.

[0021] Step 14: For the latter two types of samples, further fine-grained data annotation is performed, including the number and identity of the speakers, the level of background noise, the presence of music and vocals, the language tags used by the speakers, the volume and sound quality of the speakers, and annotations for parts without speech.

[0022] Step 2: Obtain multiple mainstream VAD models and deploy them locally. Train them on the labeled speech dataset from Step 1. Compare the performance of each mainstream VAD model to determine the optimal VAD model. Use the optimal VAD model to filter out the non-speech parts in the second and third class speech data, retaining only the valid speech parts. Furthermore, select appropriate thresholds based on detection accuracy metrics to improve the differentiation accuracy between the first, second, and third classes.

[0023] The first type of file is discarded entirely after identification. Step 2 is used to identify the portions of the available speech data in the second and third types that contain valid speech.

[0024] Step 21: Obtain mainstream VAD models, including TEN-VAD, Silero VAD, WebRTC VAD, etc., and deploy them locally.

[0025] Step 22: Based on the labeled training data collected in Step 1, compare the performance of each VAD model and evaluate it from objective evaluation indicators such as detection accuracy, latency, and computational cost.

[0026] Step 23: For the detection accuracy index, under the premise of selecting a reasonable threshold, we mainly consider the discrimination accuracy between the first type (completely unusable speech) and the second and third types (partially or completely usable) to ensure that low-quality speech signals are filtered with high accuracy at a low cost, thereby reducing the cost of subsequent training and model application.

[0027] Step 24: Apply the comprehensive optimal VAD model to filter out the non-speech parts in the latter two types of data, and retain only the valid speech parts.

[0028] The specific technical approach of VAD includes: input data is sampled at 16kHz by default, and the input data is divided using frame intervals of 10ms or 16ms. Based on this data, the VAD model is used to calculate the probability value (between 0 and 1) of each audio block containing valid speech, and then combined with a Hidden Markov Model (HMM) to identify each consecutive audio block. Based on a pre-set probability threshold (default is 0.5, multiple thresholds can be tried, and a value with good overall continuity within a certain range can be selected as the final parameter), the proportion of valid speech in the entire audio file is determined. Files with an overall proportion below a certain limit (e.g., 20%) are considered invalid files; otherwise, they are considered valid files, but only the valid speech portion is retained after applying VAD.

[0029] Step 3: Research basic speech recognition models. Select a basic speech recognition model to perform automatic speech recognition and speaker annotation on the data samples filtered in Step 2, generating speech recognition text and labeled data. Recognition information includes, but is not limited to, speaker identity, dialect recognition, environmental noise and background music, and technical terms. Mainstream commercial and open-source speech recognition models can be used. For example, Tencent, Alibaba, and iFlytek all have commercial models, and there are even more open-source models available. Most common speech models include ASR (Automatic Speech Recognition) functionality.

[0030] Taking a pharmacy scenario as an example, the result of transcribing a pharmacy audio file using a basic speech recognition model is shown in parentheses at the beginning, which indicates the start and end times of the speech signal.

[0031] The recording includes: [0:1.440,0:4.590,0] We only have boxes, oh, only boxes.

[0032] [0:6.490,0:8.990,1] No.

[0033] [0:16.470,0:32.920,2] So, are your pants shapely? [0:38.610,0:41.210,0] It hurts a lot, so I need to add an anti-inflammatory drug. It doesn't hurt that much.

[0034] [0:41.260,0:42.510,1] spores.

[0035] [0:44.060,0:45.660,0] Then just add them together and eat them.

[0036] [0:49.360,0:50.285,0] Is it just a sore throat? [0:50.285,0:51.360,0] Are there any other symptoms? [0:51.560,0:54.310,1] My throat is inflamed, is this okay? [0:55.210,0:56.460,0] Take it with cephalosporin.

[0037] [2:14.680,2:15.780,0] Scan successful.

[0038] The automatic speech recognition annotation of this text includes the following aspects.

[0039] Speaker identity: This model failed to correctly identify the speaker, mainly because the customer's voice was too low and the recognition failed. It basically only recognized the voice of the store clerk.

[0040] Dialect recognition: Judging from the original audio, the store clerks spoke Sichuan dialect, because the original audio was recorded in Chengdu, where most store clerks speak the local dialect.

[0041] Ambient noise and background music: No obvious music signal was detected in this audio segment, but the last sentence "scanning successful" was not spoken by the store clerk, but by the POS system, which needs to be distinguished.

[0042] Technical terms: Among the medical-related technical terms that can be identified are "anti-inflammatory drugs", "cephalosporins", and "throat inflammation". The fifth sentence should also refer to "cephalosporins", but it is not fully recognized due to the weak voice signal.

[0043] Other key information: The phrase "Then what shape are your pants?" in the third sentence should be a description of the name of the medicine based on the context, but it was not correctly identified. The correct text annotation needs to be provided by comparing it with the original audio.

[0044] Step 4: Human annotators perform secondary annotation and correction on the speech recognition text and annotation data output from Step 3. They compare the data with the original audio file, identify the main errors, and obtain the corrected annotation data. The annotation process mainly includes the following steps.

[0045] Step 41: Speaker identity verification. Verify whether the speaker's identity is correctly identified, especially in interruption scenarios during a conversation, where speaker confusion can easily occur.

[0046] Step 42: Dialect Recognition. Check whether the accuracy of dialect recognition in the dialogue is too low. Is it because of incorrect recognition of dialect type (such as recognizing Sichuan dialect as Chongqing dialect or Mandarin) or because the ability to recognize dialects is weak?

[0047] Step 43: Environmental and background noise verification to determine if there is any incorrect transcription of environmental noise and background music (especially human voices in the music).

[0048] Step 44: Technical terminology check to determine the accuracy of technical terminology recognition. Taking a pharmacy scenario as an example, the accuracy of the recognition of drug names, symptom descriptions, usage and dosage, applicable and contraindication descriptions, prices, etc., is marked accordingly.

[0049] Step 45: Check other key information. Check for obvious transcription errors in other key information. For non-key content, such as phone numbers and membership numbers, a certain error rate is allowed.

[0050] Step 46: Based on the annotation process of steps 41 to 45, correct the key content to obtain the corrected annotation data, which will be used as the corrected training data.

[0051] Step 5: Based on the calibration and annotation data in Step 4, and referring to the mainstream speech model technology roadmap, optimize the speech recognition model selected in Step 3.

[0052] Step 51: For cases with a high speaker identification error rate, add a speaker voiceprint recognition module to cluster the content in the audio signal based on the speaker's voice features to ensure that the speech boundaries of different subjects can be distinguished more accurately.

[0053] Step 52: For ambient noise and background music, after the signal separation of the labeled data, treat them as individual speakers for identification, and extract as much interference audio information as possible from the audio.

[0054] Step 53: For cases where dialect recognition errors occur or accuracy is low, use the corrected training data for model fine-tuning to improve the recognition accuracy of the corresponding dialect. During fine-tuning, be sure to mix in some basic data generated by the original model to avoid catastrophic forgetting.

[0055] Step 54: Introduce a keyword list relevant to the scenario. For example, in a pharmacy scenario, the keyword list can include drug names, symptom descriptions, etc., and their standard forms and variations (especially non-standard expressions in dialects) are labeled in the training data to ensure recognition accuracy.

[0056] Following mainstream speech model architectures, the speech recognition model used in this invention mixes speech and text signals as input, serving as both input and output representations. The original input audio is processed by an audio encoder, outputting 25Hz downsampled audio information. This output, along with historical data (containing interleaved audio and text tokens), is then input into a large language model for decoding. The output signal also contains interleaved text and audio data. During the fine-tuning phase, the original data includes real-world application scenario recordings and synthesized data. The latter includes speech data rewritten from text data (to create more conversational and dialogue-style text) and generated (using various speech rates, linguistic styles, and emotion tags) to cover a wider range of application areas and target scenarios. In optimizing and evaluating model performance, in addition to the language data and annotation results collected in this application, resources from recently published open-source projects in academia are also used, such as the LLaSO-Align dataset and the LLaSO-Eval evaluation framework.

[0057] Step 6: Repeat the automatic speech recognition and speaker annotation in Step 3 using the optimized speech recognition model, and perform the next round of manual verification and annotation. Currently, only manual verification is possible (unlike in the field of text models, there is no authoritative speech model that can guarantee near 100% accuracy), until a preset acceptable recognition accuracy is reached, or the recognition rate does not significantly improve.

[0058] Step 7: Apply the optimized speech recognition model to the actual production environment for automatic speech recognition, and output the production environment speech and transcribed text library.

[0059] Step 71: Based on a specific scenario, collect representative voice recordings of the main subjects in a quiet environment (such as pharmacy clerks or doctors in clinics) and identify their voiceprint features.

[0060] Step 72: Select appropriate recording equipment to collect voice signals from the production environment.

[0061] Step 73: Use the optimal VAD model to remove unqualified audio files and extract the portion of the usable audio files that contains valid speech signals. Unqualified audio refers to the first category of completely unusable speech data, while usable audio refers to the second and third categories of usable speech data.

[0062] Step 74: Based on the voiceprint features of the recording subject representative in Step 72 and the generated environmental speech signal collected in Step 73, use the optimized speech recognition model to perform automatic speech recognition, ensuring accurate identification of the main speaker (pharmacies clerks, outpatient doctors), while allowing a higher error rate threshold for secondary subjects (such as customers and other staff).

[0063] Step 8: The text information with identity recognition information extracted in Step 7 will be used in subsequent data processing procedures, such as identification of prohibited language and analysis of standard language.

[0064] Step 9: The voice data and transcribed text information extracted in the actual production scenario are sampled into the case library with a certain probability. The cases in the case library are reviewed regularly, and obvious recognition errors are marked. Then, the process proceeds to Step 5 for periodic model optimization to dynamically update the model parameters.

[0065] It should be noted that the specific embodiments described above are exemplary, and those skilled in the art can devise various solutions inspired by the disclosure of this invention. These solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents.

Claims

1. An automatic speech recognition method for complex environments, characterized in that, First, the usability of the speech data is labeled and classified. Then, the optimal VAD (Voice Availability Detection) model is matched for speech data filtering. The filtered data is input into the speech recognition model for initial recognition and labeling training. Human annotators then perform secondary manual labeling, and the data is input back into the speech recognition model for optimization until a preset recognition accuracy is reached, at which point training stops. Finally, the data is applied to real-world generation scenarios. The method includes: Step 1: Based on the set application scenario, collect massive amounts of dialogue data with different speech quality and upload them to the server. Classify the dialogue data by usability labeling, including three categories: completely unusable, partially usable, and basically completely usable. Perform fine-grained data labeling on the latter two categories of usable dialogue data to obtain labeled speech dataset. Step 2: Obtain multiple mainstream VAD models and deploy them locally. Train them based on the labeled speech dataset from Step 1. Compare the performance of each mainstream VAD model and determine the optimal VAD model. Filter the non-speech parts in the second and third types of speech data using the optimal VAD model, retaining only the valid speech parts. Furthermore, select an appropriate threshold based on the detection accuracy index to improve the discrimination accuracy between the first, second, and third types. Step 3: Investigate basic speech recognition models, select basic speech recognition models to perform preliminary automatic speech recognition and speaker annotation on the data samples that have been filtered through speech data in Step 2, and generate speech recognition text and annotation data; Step 4: The human annotator performs secondary annotation and correction on the speech recognition text and annotation data output in Step 3, compares it with the original audio file, identifies the main errors, and obtains the corrected annotation data; Step 5: Optimize the speech recognition model selected in Step 3 based on the corrected labeled data in Step 4; Step 6: Repeat the automatic speech recognition and speaker labeling in Step 3 using the optimized speech recognition model, and perform the next round of manual verification and labeling until the preset recognition accuracy is achieved, or the recognition rate is not significantly improved. Step 7: Apply the optimized speech recognition model to the actual production environment for automatic speech recognition, and output the production environment speech and a transcribed text library containing identity recognition information; Step 8: The text information with identity recognition information extracted in Step 7 will be used in subsequent data processing, including identification of prohibited language and analysis of standard language. Step 9: The speech data and transcribed text information extracted from the actual production scenario are sampled into the case library with a set probability. The manual annotators regularly review the cases in the case library, mark obvious recognition errors, and then proceed to Step 5 to perform periodic speech recognition model optimization to dynamically update the model parameters.

2. The automatic speech recognition method as described in claim 1, characterized in that, The manual secondary annotation method in step 4 includes: Step 41: Speaker identity verification. Verify whether the speaker's identity is correctly identified, especially in interruption scenarios during a conversation, where speaker confusion can easily occur. Step 42: Dialect recognition. Check if the dialect recognition accuracy in the dialogue is too low, and determine whether it is due to incorrect recognition of dialect type or weak dialect recognition ability. Step 43: Environmental and background noise verification to determine if there has been any incorrect transcription of environmental noise and background music; Step 44: Technical terminology check to determine if the technical terminology identification is accurate; Step 45: Check other key information. Check for obvious transcription errors in other key information. For non-key content, allow errors within the preset error rate range. Step 46: Based on the annotation process of steps 41 to 45, correct the key content to obtain the corrected annotation data, which will be used as the corrected training data.

3. The automatic speech recognition method as described in claim 1, characterized in that, Step 5, optimizing the speech recognition model, includes: Step 51: For cases with a high speaker identification error rate, add a speaker voiceprint recognition module to cluster the content in the audio signal based on the speaker's voice features in order to accurately distinguish the speech boundaries of different subjects. Step 52: For ambient noise and background music, after signal separation of the labeled data, they are treated as individual speakers for identification in order to extract interference audio information from the audio. Step 53: For cases where dialect recognition is incorrect or the accuracy is low, the corrected training data is used for model fine-tuning to improve the recognition accuracy of the corresponding dialect. Step 54: Introduce a keyword list related to the scene, and annotate its standard form and variants in the training data accordingly to ensure recognition accuracy.

Images