A dialect speech recognition method and system based on pre-training and fine-tuning
By employing pre-training and fine-tuning methods, the accuracy and reliability issues of dialect speech recognition were addressed, improving the accuracy of speech recognition and the generalization ability of the model, and providing a reliable data foundation.
Patent Information
- Application Number
- CN202410481956.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-22
- Publication Date
- 2025-11-04
- Estimated Expiration
- 2044-04-22
AI Technical Summary
Existing technologies cannot effectively recognize and understand the speech of different dialects, and cannot obtain reliable and accurate dialect speech datasets, resulting in the inability to accurately identify ambiguous content in speech recognition.
A pre-training and fine-tuning-based approach is adopted to obtain feature information and language features of the speech data to be recognized, filter and process dialect speech data, train and obtain a speech recognition model, and use speech matching index to correct ambiguity information to improve recognition accuracy.
It improves the accuracy and reliability of dialect speech recognition, enhances the generalization ability of speech recognition models, and provides a more reliable data foundation.
Smart Images
Figure CN118298814B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of speech recognition, in particular to a dialect speech recognition method and system based on pre-training and fine-tuning. BACKGROUND
[0002] Speech recognition is a cross-disciplinary subject. In the past two decades, speech recognition technology has made significant progress and has begun to move from the laboratory to the market. It is predicted that in the next 10 years, speech recognition technology will enter various fields such as industry, home appliances, communication, automotive electronics, medical care, home services, and consumer electronics. Dialect speech recognition technology is an important field of speech recognition. In recent years, with the development of artificial intelligence and deep learning technology, significant progress has been made. Dialect speech recognition technology refers to the technology that can recognize and understand different dialects. Early dialect speech recognition technology mainly relied on speech recognition algorithms and a large amount of dialect speech data. However, due to the diversity and complexity of dialects, this method did not achieve satisfactory results.
[0003] Currently, there are still problems in dialect speech recognition, such as being unable to identify speech data according to speech content, being unable to obtain reliable, accurate, and standard data sets according to dialect speech data, being unable to accurately recognize dialect speech, and being unable to recognize speech recognition ambiguous content. SUMMARY
[0004] To solve the above technical problems, a dialect speech recognition method and system based on pre-training and fine-tuning are provided. The technical solution solves the problems of identifying speech data according to speech content, being unable to obtain reliable, accurate, and standard data sets according to dialect speech data, being unable to accurately recognize dialect speech, and being unable to recognize speech recognition ambiguous content.
[0005] To achieve the above purposes, the technical solution adopted by the present application is as follows:
[0006] A dialect speech recognition method based on pre-training and fine-tuning, comprising:
[0007] Obtaining the to-be-recognized speech data, the to-be-recognized speech data comprising to-be-recognized speech data format information and to-be-recognized speech setting information;
[0008] Obtaining the to-be-recognized speech identification information according to the to-be-recognized speech data, the to-be-recognized speech identification information comprising no-speech identification information, speech transcription identification information, and speech translation identification information;
[0009] Obtaining the dialect speech data, the dialect speech data comprising dialect speech audio data and dialect speech transcription information;
[0010] According to the dialect speech data, based on data preprocessing and data fine-tuning, obtain the speech recognition data set;
[0011] According to the speech recognition data set, obtain the dialect speech recognition model;
[0012] According to the to-be-recognized speech identification information and the to-be-recognized speech data, based on the dialect speech recognition model, obtain the speech recognition information.
[0013] Preferably, the to-be-recognized speech identification information is obtained according to the to-be-recognized speech data, specifically including:
[0014] According to the to-be-recognized speech data, obtain to-be-recognized speech feature information, the to-be-recognized speech feature information includes speech segment information and speech blank segment information;
[0015] According to the speech blank segment information, based on the non-speech identification, obtain the non-speech identification information;
[0016] According to the speech segment information, obtain speech acoustic feature information, the speech acoustic feature information includes speech phoneme information, speech tone information and speech speed information;
[0017] According to the speech acoustic feature information, based on language acoustic feature matching, obtain speech language information;
[0018] According to the speech language information, judge whether the speech needs to be translated, if yes, according to the speech language information, obtain the speech translation identification information, if not, according to the speech language information, obtain the speech transcription identification information;
[0019] According to the speech translation identification information, add text tokens to the speech segment of the speech translation identification, wherein the text tokens include all speech translation identification speech segments;
[0020] According to the speech transcription identification information, give time stamps to the speech segment of the speech transcription identification, obtain speech timestamp information, the speech timestamp information includes text transcription start time and text transcription end time;
[0021] According to the speech timestamp information, add text tokens to the speech segment of the speech transcription identification, obtain text token information;
[0022] According to the non-speech identification information, the speech transcription identification information, the speech translation identification information and the text token information, obtain the to-be-recognized speech identification information.
[0023] Preferably, the speech recognition data set is obtained according to the dialect speech data, based on data preprocessing and data fine-tuning, specifically including:
[0024] According to the dialectal voice data, obtain dialectal voice audio data and dialectal voice transcription information corresponding to the dialectal voice audio;
[0025] According to the dialectal voice transcription information, obtain dialectal voice transcription feature information, the dialectal voice transcription feature information including transcription punctuation information, transcription formatting blank information and transcription style information;
[0026] According to the dialectal voice transcription feature information, screen the dialectal voice transcription information to obtain dialectal voice processing data;
[0027] According to the dialectal voice processing data, obtain dialectal voice audio environment information, dialectal voice setting information and language category information;
[0028] According to the dialectal voice audio environment information, dialectal voice setting information and language category information, obtain an audio data quality index;
[0029] According to the audio data quality index, screen the dialectal voice processing data to obtain a voice recognition data set.
[0030] Preferably, according to the audio data quality index, screening the dialectal voice processing data to obtain a voice recognition data set, specifically comprising:
[0031] According to the dialectal voice audio environment information, dialectal voice setting information and language category information, obtain an audio data quality index;
[0032] Based on the dialectal voice recognition requirement, obtain an audio data quality index threshold;
[0033] According to the audio data quality index and the audio data quality index threshold, screen the dialectal voice processing data to obtain dialectal voice secondary processing data;
[0034] According to the dialectal voice secondary processing data, obtain dialectal voice secondary processing transcription information;
[0035] According to the language category information, obtain dialectal voice spoken language information;
[0036] According to the dialectal voice secondary processing transcription information and the dialectal voice spoken language information, determine whether the dialectal voice transcription text and the voice spoken language are matched;
[0037] If the dialectal voice transcription text and the voice spoken language are matched, the dialectal voice secondary processing data is available, and according to the dialectal voice secondary processing data, a voice recognition data set is obtained;
[0038] The calculation formula of the audio data quality index is:
[0039]
[0040] In the formula, Q is an audio data quality index, is an audio signal-to-noise ratio of the dialect speech, x i is the intelligibility of the i th word, ω is a frequency coefficient of the speech language, and T is the dialect speech time.
[0041] Preferably, the dialect speech recognition model is obtained according to the speech recognition data set, and specifically includes the following steps.
[0042] The speech recognition data set is segmented based on a speech recognition time threshold according to the speech recognition data set, and speech recognition data set segment information is obtained.
[0043] The speech recognition initial model is trained according to the speech recognition data set segment information, and speech recognition initial training data is obtained.
[0044] The speech recognition training error information is obtained according to the speech recognition initial training data.
[0045] The speech recognition data set abnormal segment information is obtained based on a training error rate threshold according to the speech recognition training error information.
[0046] The speech recognition data set abnormal segment information is removed to obtain data set standard segment information.
[0047] The existing speech recognition model is trained according to the data set standard segment information, and the dialect speech recognition model is obtained.
[0048] Preferably, the speech recognition information is obtained based on the dialect speech recognition model according to the to-be-recognized speech identification information and the to-be-recognized speech data, and specifically includes the following steps.
[0049] The to-be-recognized speech data is subjected to speech recognition according to the to-be-recognized speech identification information and the dialect speech recognition model, and speech recognition initial information is obtained.
[0050] The speech recognition fuzzy information is obtained according to the speech recognition initial information and the to-be-recognized speech data.
[0051] The speech recognition possible information is obtained according to the speech recognition fuzzy information.
[0052] The to-be-recognized speech environment information is obtained according to the to-be-recognized speech data.
[0053] The speech matching index is obtained according to the speech recognition possible information and the to-be-recognized speech environment information.
[0054] The speech recognition fuzzy correction information is obtained by screening the speech recognition possible information according to the speech matching index.
[0055] According to the speech recognition fuzzy correction information and the speech recognition initial information, speech recognition information is obtained;
[0056] The formula for calculating the speech matching index is:
[0057]
[0058] In the formula, G is the speech matching index, π is the environment influence coefficient of the speech to be recognized, τ is the matching coefficient of the speech recognition possible information and the speech recognition fuzzy information, a j is the position of the jth speech recognition fuzzy in the speech, b φ(j) is the position of the speech recognition possible information after aligning the sentence where the speech recognition possible information is located with the sentence recognized by the speech recognition, and d(a j , b φ(j) ) represents the position distance difference between the speech recognition possible information and the speech recognition fuzzy information.
[0059] Further, a dialect speech recognition system based on pre-training and fine-tuning is proposed, which is used to implement the speech recognition method as described above, and includes:
[0060] The main control module is used to obtain speech feature information to be recognized according to the speech data to be recognized, obtain non-speech identification information based on non-speech identification according to the speech blank segment information, determine whether the speech needs to be translated according to the speech language information, determine whether the dialect speech transcription text and the speech spoken language match according to the dialect speech secondary processing transcription information and the dialect speech spoken language information, obtain a dialect speech recognition model according to the speech recognition data set, and obtain speech recognition information based on the dialect speech recognition model according to the speech identification information to be recognized and the speech data to be recognized.
[0061] The information acquisition module is used to obtain the speech data to be recognized, the speech data format information to be recognized, the speech setting information to be recognized, the dialect speech data, the dialect speech audio data, and the dialect speech transcription information, and transmit them to the data processing module.
[0062] The data processing module is used to obtain dialect speech transcription feature information according to the dialect speech transcription information, filter the dialect speech transcription information according to the dialect speech transcription feature information, obtain dialect speech processing data, obtain an audio data quality index according to the dialect speech audio environment information, the dialect speech setting information, and the language type information, filter the dialect speech processing data according to the audio data quality index and the audio data quality index threshold, and obtain dialect speech secondary processing data.
[0063] The display module interacts with the main control module and is used to display the speech recognition information.
[0064] Optionally, the main control module specifically comprises:
[0065] The control unit is configured to determine whether the dialect speech transcription text matches the dialect speech spoken language according to the dialect speech secondary processing transcription information and the dialect speech spoken language information, obtain a dialect speech recognition model according to a speech recognition data set, and obtain speech recognition information based on the dialect speech recognition model according to the to-be-recognized speech identification information and the to-be-recognized speech data.
[0066] The information receiving unit is configured to interact with the information obtaining module and the data processing module, receive information, and transmit the information to the speech identification unit.
[0067] The speech identification unit is configured to obtain to-be-recognized speech feature information according to the to-be-recognized speech data, obtain speech identification information based on speech blank segment information according to the speech blank segment information, and determine whether the speech needs to be translated according to the speech language information.
[0068] Optionally, the information obtaining module specifically comprises:
[0069] The first obtaining unit is configured to obtain to-be-recognized speech data, to-be-recognized speech data format information, and to-be-recognized speech setting information.
[0070] The second obtaining unit is configured to obtain dialect speech data, dialect speech audio data, and dialect speech transcription information, and transmit the information to the data processing module.
[0071] Optionally, the data processing module specifically comprises:
[0072] The first processing unit is configured to obtain dialect speech transcription feature information according to the dialect speech transcription information, filter the dialect speech transcription information according to the dialect speech transcription feature information, and obtain dialect speech processing data.
[0073] The second processing unit is configured to obtain an audio data quality index according to dialect speech audio environment information, dialect speech setting information, and language type information, filter the dialect speech processing data according to the audio data quality index and an audio data quality index threshold, and obtain dialect speech secondary processing data.
[0074] Compared with the prior art, the present application has the following beneficial effects:
[0075] The application provides a dialect speech recognition method and system based on pre-training and fine-tuning, which improves the accuracy of speech recognition through the identification information of the to-be-recognized speech, improves the accuracy and reliability of the data through the audio data quality index, provides a data basis for the speech recognition model, trains the existing speech recognition model through the standard segment information of the data set, obtains a dialect speech recognition model, screens the possible information of speech recognition according to the speech matching index, obtains speech recognition fuzzy correction information, and improves the generalization ability of the model and the accuracy of speech recognition. BRIEF DESCRIPTION OF DRAWINGS
[0076] Figure 1 A dialect speech recognition method based on pre-training and fine-tuning is provided in the application.
[0077] Figure 2 A to-be-recognized speech identification information acquisition flowchart is provided in the application.
[0078] Figure 3 A to-be-recognized speech identification schematic diagram is provided in the application.
[0079] Figure 4 A speech recognition data set acquisition flowchart is provided in the application.
[0080] Figure 5 A dialect speech recognition model acquisition flowchart is provided in the application.
[0081] Figure 6 A speech recognition information acquisition flowchart is provided in the application.
[0082] Figure 7 A dialect speech recognition system structure block diagram based on pre-training and fine-tuning is provided in the application. DETAILED DESCRIPTION
[0083] The following description is used to disclose the application so that those skilled in the art can implement the application. The preferred embodiments in the following description are only examples, and those skilled in the art can think of other obvious modifications.
[0084] REFERENCE Figures 1-6 As shown in the accompanying drawings, a dialect speech recognition method based on pre-training and fine-tuning in the application comprises:
[0085] S100: acquiring to-be-recognized speech data, wherein the to-be-recognized speech data comprises to-be-recognized speech data format information and to-be-recognized speech setting information;
[0086] S200: acquiring to-be-recognized speech identification information according to the to-be-recognized speech data, wherein the to-be-recognized speech identification information comprises no-speech identification information, speech transcription identification information and speech translation identification information;
[0087] Specifically, according to the voice data to be recognized, voice feature information to be recognized is obtained, specifically including:
[0088] According to the voice data to be recognized, voice feature information to be recognized is obtained, including voice segment information and voice blank segment information;
[0089] According to the voice blank segment information, non-speech identification information is obtained based on non-speech identification;
[0090] According to the voice segment information, voice acoustic feature information is obtained, including voice phoneme information, voice tone information and voice speed information;
[0091] According to the voice acoustic feature information, voice language information is obtained based on language acoustic feature matching;
[0092] According to the voice language information, it is judged whether the voice needs to be translated, if yes, voice translation identification information is obtained according to the voice language information, if not, voice transcription identification information is obtained according to the voice language information;
[0093] According to the voice translation identification information, text tokens are added to the voice segments of the voice translation identification, wherein the text tokens include all voice segments of the voice translation identification;
[0094] According to the voice transcription identification information, time stamps are given to the voice segments of the voice transcription identification, voice timestamp information is obtained, including text transcription start time and text transcription end time;
[0095] According to the voice timestamp information, text tokens are added to the voice segments of the voice transcription identification, text token information is obtained;
[0096] According to the non-speech identification information, the voice transcription identification information, the voice translation identification information and the text token information, voice identification information to be recognized is obtained.
[0097] In this scheme, through the voice data to be recognized, voice feature information to be recognized is obtained, the voice data is divided into voice segments and voice blank segments through voice features, the voice blank segments are non-speech identified, according to the voice language information, it is judged whether the voice needs to be translated, voice transcription identification or voice translation identification is performed, wherein if the voice needs to be translated, text tokens are directly given to the voice data for direct voice translation, if the voice does not need to be translated, the voice data is divided into multiple voice data segments through time stamps, text tokens are given to each voice data segment for voice transcription, which improves the accuracy of voice recognition.
[0098] It can be understood that in the embodiment, each timestamp contains a start time and an end time, and each timestamp interval is 20 ms.
[0099] S300: Obtain dialectal voice data, the dialectal voice data including dialectal voice audio data and dialectal voice transcription information;
[0100] S400: According to the dialectal voice data, obtain a voice recognition data set based on data preprocessing and data fine-tuning;
[0101] Specifically, according to the dialectal voice data, a voice recognition data set is obtained based on data preprocessing and data fine-tuning, specifically including:
[0102] According to the dialectal voice data, obtain dialectal voice audio data and dialectal voice transcription information corresponding to the dialectal voice audio;
[0103] According to the dialectal voice transcription information, obtain dialectal voice transcription feature information, the dialectal voice transcription feature information including transcription punctuation information, transcription formatting blank information and transcription style information;
[0104] According to the dialectal voice transcription feature information, the dialectal voice transcription information is screened to obtain dialectal voice processing data;
[0105] According to the dialectal voice processing data, obtain dialectal voice audio environment information, dialectal voice setting information and language category information;
[0106] According to the dialectal voice audio environment information, the dialectal voice setting information, the language category information, obtain an audio data quality index;
[0107] According to the audio data quality index, the dialectal voice processing data is screened to obtain a voice recognition data set.
[0108] Specifically, based on the dialectal voice recognition requirement, an audio data quality index threshold is obtained;
[0109] According to the audio data quality index and the audio data quality index threshold, the dialectal voice processing data is screened to obtain dialectal voice secondary processing data;
[0110] According to the dialectal voice secondary processing data, obtain dialectal voice secondary processing transcription information;
[0111] According to the language category information, obtain dialectal voice spoken language information;
[0112] According to the dialectal voice secondary processing transcription information and the dialectal voice spoken language information, it is judged whether the dialectal voice transcription text and the voice spoken language match;
[0113] If the dialect speech transcription text matches the speech spoken language, the dialect speech secondary processing data can be used to obtain the speech recognition data set according to the dialect speech secondary processing data;
[0114] The calculation formula of the audio data quality index is:
[0115]
[0116] In the formula, Q is the audio data quality index, is the audio signal-to-noise ratio of the dialect speech, x i is the intelligibility of the i th word, ω is the frequency coefficient of the speech language, and T is the dialect speech time.
[0117] In the scheme, dialect speech audio data and dialect speech transcription information corresponding to the dialect speech audio are obtained according to the dialect speech data, dialect speech transcription feature information is obtained according to the dialect speech transcription information, the dialect speech transcription information is screened according to the dialect speech transcription feature information, and dialect speech processing data is obtained.
[0118] It can be understood that the dialect text data covers a wide distribution of audio, recording settings and languages from many different environments. Although the diversity of audio quality can help to train the robustness of a model, the diversity of text quality is not the same advantage. Many existing ASR systems output speech recognition transcripts that only output a limited subset of written languages, which delete or normalize aspects that are difficult to predict from audio signals alone, such as complex punctuation (exclamation marks, commas and question marks), formatting white spaces such as paragraphs, or style aspects such as capitalization. A fully capitalized or fully lowercase text is unlikely to be generated by a language recognition system alone, and training with a mixed generated data set will significantly impair the performance of the translation system.
[0119] The scheme simultaneously obtains an audio data quality index according to dialect speech audio environment information, dialect speech setting information and language type information, screens the dialect speech processing data according to the audio data quality index and an audio data quality index threshold value, obtains dialect speech secondary processing data, and judges whether the dialect speech transcription text matches the speech spoken language according to the dialect speech secondary processing transcription information and the dialect speech spoken language information, thereby improving the accuracy and reliability of the data and providing a data basis for the speech recognition model.
[0120] S500: Obtain a dialect speech recognition model according to the speech recognition data set;
[0121] Specifically, the dialect speech recognition model is obtained according to the speech recognition data set, and specifically includes:
[0122] According to the speech recognition data set, the speech recognition data set is segmented based on a speech recognition time threshold to obtain speech recognition data set segment information;
[0123] According to the speech recognition data set segment information, the speech recognition initial model is trained to obtain speech recognition initial training data;
[0124] According to the speech recognition initial training data, speech recognition training error information is obtained;
[0125] According to the speech recognition training error information, based on a training error rate threshold, speech recognition data set abnormal segment information is obtained;
[0126] The speech recognition data set abnormal segment information is removed to obtain data set standard segment information;
[0127] According to the data set standard segment information, the existing speech recognition model is trained to obtain a dialect speech recognition model.
[0128] In the scheme, the speech recognition data set is segmented by a speech recognition time threshold to obtain speech recognition data set segment information, the speech recognition initial model is trained according to the speech recognition data set segment information to obtain speech recognition initial training data, the speech recognition training error information is obtained, based on a training error rate threshold, speech recognition data set abnormal segment information is obtained, and the data sources are checked by combining high error rate and data source size to effectively identify and delete low-quality data sources. According to the data set standard segment information, the existing speech recognition model is trained to obtain a dialect speech recognition model, which improves the accuracy of speech recognition.
[0129] S600: According to the to-be-recognized speech identification information and the to-be-recognized speech data, based on the dialect speech recognition model, speech recognition information is obtained.
[0130] Specifically, according to the to-be-recognized speech identification information and the to-be-recognized speech data, based on the dialect speech recognition model, speech recognition information is obtained, specifically including:
[0131] According to the to-be-recognized speech identification information and the dialect speech recognition model, the to-be-recognized speech data is subjected to speech recognition to obtain speech recognition initial information;
[0132] According to the speech recognition initial information and the to-be-recognized speech data, speech recognition ambiguous information is obtained;
[0133] According to the speech recognition ambiguous information, speech recognition possible information is obtained;
[0134] According to the to-be-recognized speech data, to-be-recognized speech environment information is obtained;
[0135] According to the speech recognition possible information and the to-be-recognized speech environment information, a speech matching index is obtained;
[0136] According to the speech matching index, the speech recognition possible information is screened to obtain speech recognition fuzzy correction information;
[0137] According to the speech recognition fuzzy correction information and the speech recognition initial information, speech recognition information is obtained;
[0138] The calculation formula of the speech matching index is:
[0139]
[0140] In the formula, G is the speech matching index, π is a to-be-recognized speech environment influence coefficient, τ is a matching coefficient of the speech recognition possible information and the speech recognition fuzzy information, a j is a position of the speech recognition fuzzy in the speech at the jth position, b φ(j) is a position of the speech recognition possible information after aligning a sentence in which the speech recognition possible information is located with a sentence recognized by the speech, and d(a j , b φ(j) ) represents a position distance difference value of the speech recognition possible information and the speech recognition fuzzy information.
[0141] In the scheme, the to-be-recognized speech data is subjected to speech recognition through the to-be-recognized speech identification information and the dialect speech recognition model to obtain speech recognition initial information, the speech recognition fuzzy information is obtained according to the speech recognition initial information and the to-be-recognized speech data, the speech recognition possible information is obtained according to the speech recognition fuzzy information, the speech matching index is obtained according to the speech recognition possible information and the to-be-recognized speech environment information, and the speech recognition possible information is screened according to the speech matching index to obtain speech recognition fuzzy correction information.
[0142] It can be understood that, in an actual speech recognition process, because of the difference in the speech environment, speech recognition may cause certain recognition fuzzy segments, and by comparing the position of the recognition fuzzy segment in the speech with the position of the possible speech content of the segment in the standard speech data in the speech recognition database, the generalization ability of the model and the accuracy of speech recognition are improved.
[0143] Referring to Figure 7 Further, based on the above-mentioned dialect speech recognition method based on pre-training and fine-tuning, a dialect speech recognition system based on pre-training and fine-tuning is provided, which comprises:
[0144] The main control module is used for obtaining voice feature information to be recognized according to voice data to be recognized, obtaining no-speech identification information based on no-speech identification according to voice blank segment information, judging whether the voice needs to be translated according to voice language information, judging whether the dialect voice transcription text and the voice spoken language are matched according to dialect voice secondary processing transcription information and dialect voice spoken language information, obtaining a dialect voice recognition model according to a voice recognition data set, and obtaining voice recognition information based on the dialect voice recognition model according to voice identification information to be recognized and voice data to be recognized.
[0145] The information acquisition module is used for acquiring voice data to be recognized, voice data to be recognized format information, voice setting information to be recognized, dialect voice data, dialect voice audio data and dialect voice transcription information, and transmitting to the data processing module.
[0146] The data processing module is used for obtaining dialect voice transcription feature information according to dialect voice transcription information, screening the dialect voice transcription information according to the dialect voice transcription feature information, obtaining dialect voice processing data, obtaining an audio data quality index according to dialect voice audio environment information, dialect voice setting information and language type information, screening the dialect voice processing data according to the audio data quality index and an audio data quality index threshold, and obtaining dialect voice secondary processing data.
[0147] The display module interacts with the main control module and is used for displaying voice recognition information.
[0148] The main control module specifically comprises:
[0149] The control unit is used for judging whether the dialect voice transcription text and the voice spoken language are matched according to dialect voice secondary processing transcription information and dialect voice spoken language information, obtaining a dialect voice recognition model according to a voice recognition data set, and obtaining voice recognition information based on the dialect voice recognition model according to voice identification information to be recognized and voice data to be recognized.
[0150] The information receiving unit interacts with the information acquisition module and the data processing module, and is used for receiving information and transmitting to the voice identification unit.
[0151] The voice identification unit is used for obtaining voice feature information to be recognized according to voice data to be recognized, obtaining no-speech identification information based on no-speech identification according to voice blank segment information, and judging whether the voice needs to be translated according to voice language information.
[0152] The information acquisition module specifically comprises:
[0153] The first acquisition unit is used for acquiring to-be-recognized voice data, to-be-recognized voice data format information and to-be-recognized voice setting information.
[0154] The second acquisition unit is used for acquiring dialect voice data, dialect voice audio data and dialect voice transcription information and transmitting to the data processing module.
[0155] The data processing module specifically comprises:
[0156] The first processing unit is used for acquiring dialect voice transcription feature information according to the dialect voice transcription information, screening the dialect voice transcription information according to the dialect voice transcription feature information and acquiring dialect voice processing data.
[0157] The second processing unit is used for acquiring an audio data quality index according to the dialect voice audio environment information, the dialect voice setting information and the language type information, screening the dialect voice processing data according to the audio data quality index and an audio data quality index threshold and acquiring dialect voice secondary processing data.
[0158] In summary, the application has the following advantages: to-be-recognized voice feature information is acquired from to-be-recognized voice data, speechless identification information is acquired based on speechless identification according to speech blank segment information, to-be-recognized voice identification information is acquired according to the speechless identification information, speech transcription identification information, speech translation identification information and text token information, the accuracy of voice recognition is improved, voice recognition data sets are acquired based on data preprocessing and data fine-tuning according to dialect voice data, the accuracy and reliability of data are improved through an audio data quality index, data basis is provided for a voice recognition model, a dialect voice recognition model is acquired by training an existing voice recognition model through data set standard segment information, voice recognition fuzzy correction information is acquired by screening voice recognition possible information according to a voice matching index, the generalization ability of the model and the accuracy of voice recognition are improved.
[0159] The basic principle, main features and advantages of the application are shown and described above. It should be understood by those skilled in the art that the application is not limited by the above examples, and the above examples and descriptions in the specification are only the principles of the application. Without departing from the spirit and scope of the application, various changes and improvements can be made to the application, and these changes and improvements all fall within the scope of the claimed application. The scope of protection claimed by the application is defined by the appended claims and their equivalents.
Claims
1. A dialect speech recognition method based on pre-training and fine-tuning, characterized in that, The method comprises the following steps: acquiring to-be-recognized voice data, wherein the to-be-recognized voice data comprises to-be-recognized voice data format information and to-be-recognized voice setting information; acquiring to-be-recognized voice identification information according to the to-be-recognized voice data, wherein the to-be-recognized voice identification information comprises non-voice identification information, voice transcription identification information and voice translation identification information; acquiring dialect voice data, wherein the dialect voice data comprises dialect voice audio data and dialect voice transcription information; acquiring a voice recognition data set based on data preprocessing and data fine-tuning according to the dialect voice data; specifically comprising: acquiring dialect voice audio data and dialect voice transcription information corresponding to the dialect voice audio according to the dialect voice data; acquiring dialect voice transcription feature information according to the dialect voice transcription information, wherein the dialect voice transcription feature information comprises transcription punctuation information, transcription formatted blank information and transcription style information; screening the dialect voice transcription information according to the dialect voice transcription feature information to acquire dialect voice processing data; the calculation formula of the audio data quality index is: In the formula, Q is an audio data quality index, is the audio signal-to-noise ratio of the dialect speech, x i is the intelligibility of the i-th word, ω is a frequency coefficient of use of the speech language, and T is the dialect speech time. screening the dialect voice processing data according to the audio data quality index to acquire the voice recognition data set; acquiring a dialect voice recognition model according to the voice recognition data set; acquiring voice recognition information based on the dialect voice recognition model according to the to-be-recognized voice identification information and the to-be-recognized voice data.
2. The dialect speech recognition method based on pre-training and fine-tuning according to claim 1, characterized in that, The method further comprises the following steps: acquiring to-be-recognized voice feature information according to the to-be-recognized voice data, wherein the to-be-recognized voice feature information comprises voice segment information and voice blank segment information; acquiring non-voice identification information based on non-voice identification according to the voice blank segment information; acquiring voice acoustic feature information according to the voice segment information, wherein the voice acoustic feature information comprises voice phoneme information, voice tone information and voice speed information; acquiring voice language information based on language acoustic feature matching according to the voice acoustic feature information; judging whether the voice needs to be translated according to the voice language information, if yes, acquiring voice translation identification information according to the voice language information, if not, acquiring voice transcription identification information according to the voice language information; adding text tokens to voice segments with voice translation identification according to the voice translation identification information, wherein the text tokens comprise all voice segments with voice translation identification; assigning time stamps to voice segments with voice transcription identification according to the voice transcription identification information to acquire voice timestamp information, wherein the voice timestamp information comprises text transcription start time and text transcription end time; adding text tokens to voice segments with voice transcription identification according to the voice timestamp information to acquire text token information; acquiring to-be-recognized voice identification information according to the non-voice identification information, the voice transcription identification information, the voice translation identification information and the text token information.
3. The dialectal speech recognition method based on pre-training and fine-tuning according to claim 2, characterized in that, The method further comprises the following steps: acquiring the audio data quality index according to dialect voice audio environment information, dialect voice setting information and language type information; Based on the dialect voice recognition demand, the audio data quality index threshold is obtained; According to the audio data quality index and the audio data quality index threshold, the dialect voice processing data is screened to obtain the dialect voice secondary processing data; According to the dialect voice secondary processing data, the dialect voice secondary processing transcription information is obtained; According to the language category information, the dialect voice oral information is obtained; According to the dialect voice secondary processing transcription information and the dialect voice oral information, it is judged whether the dialect voice transcription text and the voice oral are matched; Wherein, if the dialect voice transcription text and the voice oral are matched, the dialect voice secondary processing data is available, and the voice recognition data set is obtained according to the dialect voice secondary processing data.
4. The dialect speech recognition method based on pre-training and fine-tuning according to claim 1, characterized in that, The voice recognition data set is obtained according to the voice recognition data set, and the dialect voice recognition model is obtained, which specifically includes: According to the voice recognition data set, the voice recognition data set is segmented based on the voice recognition time threshold to obtain the voice recognition data set segment information; According to the voice recognition data set segment information, the voice recognition initial model is trained to obtain the voice recognition initial training data; According to the voice recognition initial training data, the voice recognition training error information is obtained; According to the voice recognition training error information, based on the training error rate threshold, the voice recognition data set abnormal segment information is obtained; The voice recognition data set abnormal segment information is removed to obtain the data set standard segment information; According to the data set standard segment information, the existing voice recognition model is trained to obtain the dialect voice recognition model.
5. A pre-training and fine-tuning based dialectal speech recognition system for implementing the speech recognition method according to any one of claims 1-4, characterized in that, It includes: The main control module is used for obtaining the voice feature information to be recognized according to the voice data to be recognized, obtaining the non-speech identification information based on the non-speech identification according to the voice blank segment information, judging whether the voice needs to be translated according to the voice language information, judging whether the dialect voice transcription text and the voice oral are matched according to the dialect voice secondary processing transcription information and the dialect voice oral information, obtaining the dialect voice recognition model according to the voice recognition data set, and obtaining the voice recognition information based on the dialect voice recognition model according to the voice identification information and the voice data to be recognized; The information acquisition module is used for obtaining the voice data to be recognized, the voice data format information to be recognized, the voice setting information to be recognized, the dialect voice data, the dialect voice audio data and the dialect voice transcription information, and transmitting to the data processing module; The data processing module is used for obtaining the dialect voice transcription feature information according to the dialect voice transcription information, screening the dialect voice transcription information according to the dialect voice transcription feature information, obtaining the dialect voice processing data, obtaining the audio data quality index according to the dialect voice audio environment information, the dialect voice setting information and the language category information, screening the dialect voice processing data according to the audio data quality index and the audio data quality index threshold, and obtaining the dialect voice secondary processing data; The display module interacts with the main control module and is used for displaying the voice recognition information.
6. The pre-training and fine-tuning based dialectal speech recognition system according to claim 5, wherein, The main control module specifically includes: The control unit is used for processing the transcription information and the dialectal speech spoken information according to the dialectal speech secondary processing, judging whether the dialectal speech transcription text matches the speech spoken information, obtaining a dialectal speech recognition model according to a speech recognition data set, obtaining speech recognition information based on the dialectal speech recognition model according to the to-be-recognized speech identification information and the to-be-recognized speech data; The information receiving unit is used for receiving information and transmitting the information to the speech identification unit. The speech identification unit is used for obtaining to-be-recognized speech feature information according to the to-be-recognized speech data, obtaining no-speech identification information based on no-speech identification according to the speech blank segment information, and judging whether the speech needs to be translated according to the speech language information.
7. The pre-training and fine-tuning based dialectal speech recognition system according to claim 5, wherein, The information acquisition module specifically comprises: The first obtaining unit is used for obtaining to-be-recognized speech data, to-be-recognized speech data format information and to-be-recognized speech setting information; The second obtaining unit is used for obtaining dialectal speech data, dialectal speech audio data and dialectal speech transcription information, and transmitting the information to the data processing module.
8. The pre-training and fine-tuning based dialectal speech recognition system of claim 5, wherein, The data processing module specifically comprises: The first processing unit is used for obtaining dialectal speech transcription feature information according to the dialectal speech transcription information, screening the dialectal speech transcription information according to the dialectal speech transcription feature information, and obtaining dialectal speech processing data; The second processing unit is used for obtaining an audio data quality index according to the dialectal speech audio environment information, the dialectal speech setting information and the language type information, screening the dialectal speech processing data according to the audio data quality index and an audio data quality index threshold, and obtaining dialectal speech secondary processing data.
Citation Information
Patent Citations
Multi-dialect identification method, apparatus and device and readable storage medium
CN110517664A
Speech recognition method and device, equipment and storage medium
CN114387950A