Non-autoregressive end-to-end dialect identification method for power supply service telephone system

Abstract

The invention provides a non-autoregressive end-to-end dialect recognition method for a power supply service telephone system. The method comprises the following steps of dual-track telephone recording corpus preprocessing, automatic pre-labeling and context construction, manual labeling and dialect word library construction, dialect sub-area division and classification modeling, and dialect recognition model training and optimization. Based on the ffmpeg audio processing tool, the standardization and automation level of voice data processing in a telephone system scene is improved, and the manual annotation efficiency is greatly improved. And meanwhile, aiming at the problems that dialects are various in type, significant in difference and the like, a unified dialect labeling rule system is constructed, a model training process is optimized in combination with a transfer learning strategy, and the robustness and generalization ability of the model in a complex telephone voice environment are enhanced while high accuracy is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of corpus processing and dialect voice training methods for power supply service telephone recordings, and in particular to a non-autoregressive end-to-end dialect recognition method for a power supply service telephone system. Background Art

[0002] Currently, the power grid's "cloud" telephone system has been fully implemented in all cities and counties. However, due to the heavy workload of power grids, the high pressure on grassroots service, the uneven service levels of grid workers, and the limited coverage of manual supervision, customer service quality and satisfaction still need to be improved. In particular, within the customer service system, the volume of call voice data is enormous, and significant data barriers exist between related multi-source heterogeneous data such as work order information, user profiles, and power outage data, making effective integration and value mining difficult.

[0003] Against this backdrop, building dialect recognition models covering diverse cities and counties, and transitioning customer service systems from traditional processes relying on manual listening by grid workers to automated text generation based on dialect speech recognition models, has become a key step in improving service efficiency and leveraging data value. Automatically converting voice data into structured text effectively breaks down the barriers between audio and text data, enhancing the automation and intelligence of subsequent business processing.

[0004] Existing open-source datasets for Chinese speech recognition primarily derive from textual material in everyday contexts. Most of these datasets are collected as monophonic audio in quiet environments using specialized equipment. These datasets lack the speech characteristics typical of real-world power service scenarios, particularly those associated with complex situations like noisy background noise and mixed channels commonly encountered in telephone conversations. Consequently, existing speech recognition models suffer from low accuracy when processing dual-channel phone recordings, prone to speaker confusion and confusion in recognized content, making them difficult to meet real-world business needs.

[0005] Furthermore, currently available open-source dialect recognition models have limited coverage, primarily focusing on specific regions. For county-level cities, where dialects vary widely and significantly, the model's recognition capabilities are severely insufficient, resulting in low speech recognition accuracy. More critically, traditional speech annotation methods often rely on manual phoneme-level annotation, which is not only time-consuming and labor-intensive, but also difficult to accurately annotate pronunciation relationships in dialect corpora that lack standardized pronunciation systems, further limiting the efficiency and accuracy of dialect speech recognition models.

[0006] Therefore, it is urgent to redesign an integrated processing system and method covering data segmentation, corpus annotation, model training and other links to meet the needs of dialect speech recognition in telephone recording scenarios, so as to improve the applicability, accuracy and generalization ability of the model in actual power customer service scenarios. Summary of the Invention

[0007] The present invention aims to provide a non-autoregressive end-to-end dialect recognition method for power supply service telephone systems, so as to solve the problems in existing speech recognition technology, such as insufficient processing capability of two-channel audio data for telephone recordings, confusion in text recognition of multi-person calls, low efficiency of manual annotation, and poor adaptability of dialect models. In particular, in power supply service telephone systems, due to the significant differences in initials, finals, tones, vocabulary and grammar among the dialects of various county-level cities, traditional general speech recognition models are difficult to achieve unified and efficient recognition of all dialects.

[0008] A non-autoregressive end-to-end dialect recognition method for a power supply service telephone system comprises the following steps: Step 1: Preprocessing of Dual-channel Telephone Recordings Based on the dual-channel recording data in the power supply service telephone system, audio processing tools are used to divide the audio into two mono channels, left and right, corresponding to the customer and customer service roles respectively, such as Figure 3 The results on the right are in WAV format and have a sampling rate of 16kHz. A voice activity detection model is used to remove non-speech segments from the audio and construct valid audio segments for the left and right channels suitable for annotation. Step 2: Automatic pre-annotation and context building The valid audio clips of the left and right channels constructed in step 1 are merged into a complete conversation form according to the original call time order, and a corpus dataset with contextual semantics is constructed. The corpus dataset is automatically transcribed based on the Mandarin dialect recognition base model to generate a dialect pre-annotated dataset of "audio ID-initial text", such as Figure 3 The results on the right are shown; Step 3: Manual annotation and dialect vocabulary construction Based on the dialect pre-annotated dataset in step 2, manual annotation is performed according to the established dialect annotation standards to construct a prefecture-level dialect dictionary, such as Figure 4 As shown in the Jingzhou dialect vocabulary, a standardized manually annotated dataset is formed; the dialect annotation standards include annotation standards for dialect pronunciation differences, naming entity and number annotation standards, and contextual review standards for semantic ambiguity; Step 4: Sub-dialect area division and classification modeling Combining the experience of linguistic experts with acoustic feature cluster analysis, based on the standardized manually annotated dataset obtained in step 3, dialects with similar acoustic features are grouped into the same sub-dialect area, and a dialect area classification dataset is constructed. A multi-classification acoustic model for sub-dialect area identification is trained based on the dialect area classification dataset to automatically identify the dialect area to which the input audio belongs. Step 5: Dialect Recognition Model Training and Optimization Based on the standardized manually annotated speech dataset obtained in step 3, a character-level training corpus is constructed; the Mandarin speech recognition pre-training model is used as the basic model, the low-level structure of its encoder is frozen to retain the general speech representation ability, and the high-level structure is fine-tuned to adapt to the dialect characteristics of each sub-region; for each sub-dialect region, a dialect recognition model is independently trained on its corresponding corpus; during recognition, the region to which the input audio belongs is first determined through the sub-dialect region recognition multi-classification acoustic model in step 4, and then the dialect recognition model of the corresponding sub-region is called for transcription; at the same time, a mechanism combining automatic annotation and manual review is introduced to continuously optimize the training corpus and model parameters to improve the overall recognition performance.

[0009] Furthermore, the non-voice segment in step 1 includes a ring tone, a silent segment and background noise.

[0010] Furthermore, the dialect pronunciation differences in step 3 include differences in initial consonants, finals, and tones.

[0011] Furthermore, the step 1 of pre-processing the dual-channel telephone recording corpus includes the following steps: 11) Constructing the original audio dataset D1: Collecting the full set of telephone recordings aggregated from the power grid's "cloud" platform, we organized the data into different levels, categorized by language region, prefecture, and county. We ensured that the total number of recordings was no less than 1 million, with no fewer than 150,000 recordings from each prefecture and city. This included recordings of calls from men, women, and children of all ages across all counties, ensuring language diversity, demographic diversity, and representativeness. 12) Construct left and right channel mono audio track dataset D2: Based on the original audio dataset D1, use audio processing tools to split each two-channel telephone recording into two mono tracks, one for each speaker, i.e., the customer and the other for the customer service representative. Figure 3 The results on the right are shown; the generated left and right channel audio are in a unified WAV format with a sampling rate of 16kHz to ensure standard consistency for subsequent speech processing and model input; 13) Constructing valid short audio segments D3: For the left and right channel single-track dataset D2, a voice activity detection model is introduced to automatically extract the start and end timestamps of valid speech in each long audio segment. Based on the VAD detection results, the audio is sliced ​​to generate several short speech segments. Subsequently, the invalid segments are further filtered out by combining the time annotation information of non-speech events, retaining only segments containing clear and complete speech information, thereby constructing valid short audio segments D3 for subsequent pre-labeling and manual annotation.

[0012] Furthermore, the automatic pre-annotation and context construction in step 2 includes the following steps: 21) Constructing a corpus dataset D4 with contextual semantics: Based on the valid audio clips of the left and right channels obtained in step 1, sort and merge the short audio clips of the left and right channels according to the time sequence of the original call, and reconstruct the complete conversation sequence of the two parties, such as Figure 3 As shown in the results on the right, a corpus dataset D4 with contextual semantics is formed; 22) Constructing the dialect pre-annotated dataset D5: Based on the contextual semantics corpus dataset D4, we used the Mandarin dialect recognition base model to transcribe each valid short audio clip, generating the corresponding Chinese text. Finally, we generated a text.txt file containing the "audio ID-pre-annotated text" pair. That is, each phone call contained a text.txt file, thus constructing the dialect pre-annotated dataset D5.

[0013] Furthermore, the step 3 of manual annotation and dialect vocabulary construction includes the following steps: 31) Word segmentation to assist manual annotation: Based on the dialect pre-annotated dataset D5, preliminary automatic word segmentation is performed on the pre-annotated audio transcriptions in the text.txt file, with structured segmentation performed based on the vocabulary within the sentence. 32) Dialect lexicon construction: Based on the lexical differences between dialects and Mandarin identified during manual annotation, uncommon characters with similar pronunciations to standard Mandarin are used as replacements for annotations. A mapping relationship between Mandarin and dialect word pairs is established. Each city independently constructs a dialect lexicon based on local phonetic characteristics, ultimately compiling the resulting dialect lexicon. 33) Construction of the manually annotated dataset D6: Local personnel were assigned to annotate the text.txt file corresponding to each phone call according to annotation standards that account for dialect pronunciation differences. The pre-annotated text corresponding to the audio file ID was modified to generate "audio ID-manually annotated text" pairs. These manually annotated "audio ID-manually annotated text" pairs were systematically post-processed to form the standardized manually annotated dataset D6.

[0014] Furthermore, the systematic post-processing includes: first, reading and applying the constructed dialect vocabulary to perform consistency verification, redundancy removal and normalization processing on the "Mandarin-dialect" word pairs; second, organizing reviewers with relevant experience to conduct sampling quality assessment on the labeled data, requiring that the manual labeling pass rate reach more than 80% before it can be used as subsequent model training data, and data that does not meet the standard must be re-labeled; finally, a standardized manual labeling dataset D6 is formed.

[0015] Furthermore, the annotation standards for dialect pronunciation differences are as follows: when the audio transcription text contains expressions that differ from Mandarin in terms of tone, initial consonants or finals, rare characters with extremely low frequency of use are used to replace them, so as to preserve the phonetic characteristics to the greatest extent and ensure the consistency of annotation; Named entity and number annotation standards: In scenarios involving standard named entities such as place names, personal names, time, and numbers, standard Mandarin Chinese characters must be used for annotation. The use of dialect words, colloquial symbols, or numerical symbols as substitutes is prohibited. Contextual review standards for semantically ambiguous audio clips: When faced with potentially ambiguous audio clips, annotators should combine the previously constructed dataset D4 with contextual semantic information and fully refer to the spoken context to avoid incorrect annotations due to being out of context.

[0016] Furthermore, step 4, sub-dialect area division and classification modeling, includes the following steps: 41) Constructing the dialect area classification dataset D7: In the original audio dataset D1, the audio clips were preliminarily sorted according to the "language area - city - county" hierarchical structure. The preliminary sorting includes: first, based on the classification experience of experts in the field of linguistics, the Mandarin dialects were systematically divided, and the audio samples of each county-level city were preliminarily classified into the dialect area language family; then, based on the high-dimensional semantic features extracted by the Paraformer model framework, t-SNE technology was used for visualization analysis, such as Figure 5 Cluster analysis, as shown, helps verify and optimize the accuracy of attribution of audio samples in each county-level city to sub-dialect regions, achieving a more data-driven region division. Finally, a corresponding region label is automatically generated for each audio sample, and a subdialect.txt file is constructed as the dialect region division dataset D7. Each line records a "audio ID-dialect region" correspondence, which serves as the basis for subsequent modeling and management. 42) Constructing a sub-dialect classification algorithm: Based on the dialect area classification dataset D7, a multi-classification acoustic model for sub-dialect area identification is trained using the Paraformer model architecture to achieve automatic identification of sub-dialect areas. The multi-classification acoustic model takes audio samples as input and outputs the area label to which they belong, thereby supporting the automatic classification of dialect areas in newly recorded data and providing an important foundation for subsequent dialect recognition systems.

[0017] Furthermore, the dialect recognition model training and optimization in step 5 includes the following steps: 51) Character-level modeling of annotated text: For manually annotated text data, a character-level modeling strategy is used, using "character" as the smallest morpheme unit. This involves inserting spaces between each Chinese character, so that each character corresponds to a separate token. 52) Constructing the dialect recognition model training dataset D8: Based on the standardized manually annotated dataset D6, each "audio ID-manually annotated text" pair was organized into a standardized JSON formatted data item. Each JSON data item contained the following fields: unique audio file identifier, audio file path, total audio duration, corresponding annotated text content, and the total number of Chinese characters in the annotated text. Based on the statistical results of the total audio duration, dataset D6 was divided into a training set and a validation set in an 8:2 ratio, thus forming the complete dialect recognition model training dataset D8 for subsequent modeling. 53) Training the sub-dialect recognition model: Using the Paraformer model as the foundation, fine-tuning training is performed on the speech data of different sub-dialect regions in the dialect recognition model training dataset D8 to adapt to the dialect characteristics of each region. The fine-tuning training includes: in terms of training strategy, adopting a "partial parameter freezing" method, that is, freezing the underlying network parameters in the Paraformer encoder to retain its basic modeling ability for Mandarin audio; only training the high-level structure in the encoder to enhance the model's perception of dialect speech characteristics.

[0018] The present invention proposes a non-autoregressive end-to-end dialect recognition method for a power supply service telephone system, which has the following beneficial effects: 1. Efficient data processing mechanism: This invention designs a set of processes and rule systems for automatic segmentation, pre-labeling, and manual proofreading of two-channel audio suitable for telephone voice scenarios. This effectively improves the standardization and automation level of voice data processing, greatly increases the efficiency of manual labeling, and reduces labor and time costs.

[0019] 2. Unified data structure and format specifications: By building a standardized voice dataset format (such as a JSON structure containing audio ID, path, duration, character-level text, and its length), a highly consistent and reusable basic data structure is provided for subsequent speech recognition training, model evaluation, and dialect classification tasks, facilitating model migration, deployment, and iterative optimization.

[0020] 3. Universal dialect annotation rule system: This systematized annotation rule system is highly transferable for dialect speech recognition and can be applied to dialect speech annotation work in any other region. It provides methodological support for building larger-scale, multi-language, and multi-dialect speech datasets, further promoting the regional adaptation and application of dialect recognition technology.

[0021] 4. Optimized Transfer Learning Strategy: This paper introduces a refined transfer training mechanism in the construction of the dialect classification model and the regional dialect recognition model. By freezing the underlying structure of the base model and fine-tuning the high-level semantic layer, the model's learning and generalization capabilities for dialect speech features are enhanced, significantly improving the recognition accuracy and robustness of the speech recognition system in complex telephone speech scenarios.

[0022] 5. System architecture suitable for actual deployment: The proposed non-autoregressive end-to-end recognition architecture achieves faster decoding speed while ensuring recognition performance. It is more in line with the real-time and concurrent processing requirements of telephone service systems and has good engineering feasibility and deployment adaptability.

[0023] In summary, this invention is not only innovative in data labeling and processing procedures, model training strategies, and system architecture, but also significantly improves the efficiency, accuracy, and practicality of dialect recognition tasks for telephone systems, providing technical support for the intelligent upgrade of multilingual service systems such as electricity and government affairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of a non-autoregressive end-to-end dialect recognition method for a power supply service telephone system according to the present invention; Figure 2 A schematic diagram of detailed implementation steps of a non-autoregressive end-to-end dialect recognition system and method for a power supply service telephone system according to the present invention; Figure 3 This is a data segmentation comparison example diagram of the present invention; Figure 4 This is an example diagram of the dialect vocabulary of the present invention; Figure 5 This is an example diagram of the dialect area division of the present invention. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0026] See also Figure 1 and Figure 2 The embodiment of the present invention provides a non-autoregressive end-to-end dialect recognition method for a power supply service telephone system, comprising the following steps: Step 1: Preprocessing of two-channel telephone recording data: Based on the two-channel recording data in the power supply service telephone system, use audio processing tools (such as ffmpeg) to divide the audio into two mono channels, corresponding to the customer and customer service roles respectively, such as Figure 3 The results are shown on the right, and are converted to WAV format with a 16kHz sampling rate. Subsequently, a voice activity detection (VAD) model was applied to remove ringing tones, silent segments, and background noise, retaining only valid speech segments to construct a raw corpus suitable for annotation.

[0027] Step 2: Automatic pre-annotation and context construction: Combine the left and right channel audio clips into a complete dialogue form, construct a corpus dataset with contextual semantics, and automatically transcribe it based on the Mandarin recognition model, such as Figure 3 As shown in the result on the right, a pre-labeled dataset of "audio ID-initial text" is formed.

[0028] Step 3: Manual annotation and dialect word library construction: Based on the pre-annotation, manual correction is performed according to the established dialect annotation standards, and a prefecture-level dialect dictionary is constructed, such as Figure 4 The Jingzhou dialect vocabulary is shown in this article. The annotation strategy includes rules for handling initials, finals, and tone differences, standardizing the use of numbers in specific place names, and implementing a contextual verification strategy for semantically ambiguous text, forming a standardized manually annotated dataset.

[0029] Step 4: Sub-dialect division and classification modeling: Combine the experience of linguistic experts with acoustic feature clustering analysis (such as visualization results based on Paraformer and t-SNE), such as Figure 5 As shown in cluster analysis, dialect data is divided into several sub-regions. Dialects with similar acoustic characteristics are grouped together for centralized training, while different dialect sub-regions are modeled separately. Based on this, a multi-classification model is trained to automatically identify the sub-region to which the input audio belongs, guiding the subsequent selection of the corresponding sub-dialect recognition model.

[0030] Step 5: Dialect Recognition Model Training and Optimization: A standard training set was constructed based on manually annotated data, employing a character-level modeling strategy to improve decoding efficiency and recognition robustness. Based on this foundation, using a pre-trained Mandarin Chinese model, the encoder's low-level structure was frozen to preserve general speech capabilities, while only the high-level structure was fine-tuned to adapt to dialect characteristics. The dialect recognition model was independently trained on the corpus from each sub-region. Subsequently, a new round of automatic annotation and manual review was implemented to continuously optimize the training set and support iterative improvements in model performance.

[0031] The step 1 includes: Construction of the original audio dataset D1: The data was collected from the full set of telephone recordings aggregated on the power grid's "cloud" platform and organized hierarchically by "language region, prefecture, and county." Data sampling ensured that the total number of recordings was no less than 1 million, with no fewer than 150,000 recordings per prefecture and city. This included recordings of calls from men, women, and children of all ages across all counties, ensuring language diversity, demographic diversity, and representativeness.

[0032] Construction of left and right channel mono audio track dataset D2: Based on dataset D1, use audio processing tools (such as ffmpeg) to split each two-channel telephone recording into two mono tracks, one for each speaker, i.e., customer service staff and the other for each user. Figure 3 The results are shown on the right. The generated left and right channel audio are in the unified WAV format, with a sampling rate of 16kHz to ensure standard consistency for subsequent speech processing and model input.

[0033] Constructing valid short audio segments D3: A voice activity detection (VAD) model was introduced for each of the left and right channel single-track datasets D2 to automatically extract the start and end timestamps of valid speech within each long audio segment. Based on the VAD detection results, the audio was sliced ​​to generate several short speech segments. Subsequently, the invalid segments were further filtered out by combining the time annotation information of non-speech events such as phone ringing and background noise, retaining only segments containing clear and complete speech information. This resulted in constructing a valid short audio dataset D3 for subsequent pre-annotation and manual annotation.

[0034] The step 2 includes: Construction of contextual semantic information dataset D4: Based on the valid short audio clips of the left and right channels obtained in step 1, the short audio clips of the left and right channels are sorted and merged according to the time sequence of the original call, and the complete conversation sequence of the two parties is reconstructed to form a dataset D4 with contextual semantic continuity, such as Figure 3 The results on the right are shown. This dataset preserves the natural interactions during calls, which helps subsequent models learn semantic clues and contextual reasoning capabilities.

[0035] Constructing the dialect pre-annotated dataset D5: Based on the contextual semantics dataset D4, we used the Mandarin dialect recognition base model to transcribe each valid short audio clip and generate the corresponding Chinese text. Finally, we generated a text.txt file containing the "audio ID - initial text" pair, constructing the dialect pre-annotated dataset D5.

[0036] The step 3 includes: Word segmentation processing for assisted manual annotation: Based on the pre-annotated dialect dataset D5, perform preliminary automatic word segmentation processing on the audio-transcribed text in the text.txt file, and perform structured splitting based on the words in the sentence. This processing helps the annotators quickly locate the positions of the words that need to be modified, significantly reduces the problems of over-annotation and under-annotation in the manual annotation process, and improves the annotation efficiency and accuracy.

[0037] Construction of the dialect vocabulary: For the vocabulary differences between dialects and Mandarin identified during the manual annotation process, especially the regional-specific expressions in dialects, use rare Chinese characters similar in pronunciation to standard Mandarin for alternative annotation, and establish a mapping relationship of "Mandarin vocabulary - dialect vocabulary". As Figure 4 shown in the Jingzhou dialect vocabulary, each city independently constructs a dialect dictionary based on the local phonetic features, and finally summarizes to form a dialect vocabulary.

[0038] Annotation specification for dialect pronunciation differences: When there are expressions in the audio-transcribed text that differ from Mandarin in terms of tone, initial consonant, or final vowel at the phonetic level, use rare Chinese characters with extremely low usage frequencies for substitution to retain the phonetic features to the greatest extent and ensure the consistency of annotation. For example, annotate "street lamp" as "lóu dēng", "can" as "kuò yǐ", etc., and record this substitution rule in the dialect dictionary of the corresponding city.

[0039] Annotation specification for named entities and numbers: In scenarios involving standard named entities such as place names, personal names, time, and numbers, it is strictly required to use standard Mandarin Chinese characters for annotation, and it is prohibited to use dialect words, colloquial symbols, or numerical symbols for substitution. For example, "Jingzhou" cannot be annotated as "Jīngzhōu", and "forty-seven percent" cannot be abbreviated as "47%".

[0040] Context review for semantic ambiguity situations: When faced with audio segments that may have semantic ambiguity, the annotators should combine the previously constructed dataset D4 with context semantic information, and fully refer to the common language context to avoid incorrect annotation caused by being out of the semantic environment. For example, correctly understand "The weather has electricity and you don't have space" as "The elevator has electricity and there is no power outage", so as to ensure the rationality of the annotation logic and the consistency of the context.

[0041] Construction of the manually annotated dataset D6: Perform systematic post-processing on the "audio ID - text" pair data that has completed manual annotation. First, read and apply the constructed dialect vocabulary, and perform consistency verification, redundancy removal, and normalization processing on the "Mandarin - dialect" word pairs. Second, organize reviewers with relevant experience to conduct sampling quality assessment on the annotated data, and require the qualified rate of manual annotation to reach more than 80% before it can be used as the subsequent model training data. The unqualified data must be re-annotated. Finally, form a high-quality manually annotated dataset D6. [[ID=-18]]

[0042] The step 4 includes: Constructing the dialect area classification dataset D7: In the original audio dataset D1, the audio clips have been preliminarily sorted according to the "language area-city-county" hierarchical structure. First, based on the classification experience of experts in the field of linguistics, the Mandarin dialects are systematically divided, and the audio samples of each county-level city are preliminarily classified into the dialect area language family. Then, as Figure 5 As shown in the figure, based on the high-dimensional semantic features extracted by the Paraformer model framework, t-SNE technology is used for visualization analysis, as shown in the figure. Figure 5 Cluster analysis, as shown, helps verify and optimize the accuracy of sub-dialectal region assignments for each county-level city audio sample, enabling a more data-driven region division. Finally, a corresponding region label is automatically generated for each audio sample, and a subdialect.txt file is constructed, with each line recording a corresponding "audio ID-dialect region" relationship, serving as a basis for subsequent modeling and management.

[0043] Building a sub-dialect classification algorithm: Based on the regionalized dataset D7, a multi-classification acoustic model was trained using the Paraformer model architecture to automatically identify sub-dialect regions. This model takes an audio sample as input and outputs the region label to which it belongs. This allows for the automatic classification of new audio recordings into dialect regions, providing a critical foundation for subsequent dialect recognition systems.

[0044] The step 5 includes: Character-level modeling of annotated text: For manually annotated text data, a character-level modeling strategy is used, using the "character" as the smallest morpheme unit. This strategy involves inserting spaces between each Chinese character, so that each character corresponds to a separate token. This approach not only significantly reduces the complexity of the model vocabulary but also improves decoding efficiency and model robustness during recognition. Character-level modeling demonstrates greater stability and adaptability, particularly in scenarios with significant pronunciation differences among dialects or lexical ambiguity.

[0045] Constructing the Dialect Recognition Model Training Dataset D8: Based on the manually annotated dataset D6, each "audio ID-text" pair was organized into a standardized JSON-formatted data item. Each JSON data item contains the following fields: the unique audio file identifier, the audio file path, the total duration of the audio, the corresponding annotated text content, and the total number of Chinese characters in the annotated text. Based on the statistical results of the total audio duration, the dataset was divided into a training set and a validation set in an 8:2 ratio, forming the complete training dataset D8 for subsequent modeling.

[0046] Training the sub-dialect recognition model: The Paraformer model serves as the foundation, having demonstrated strong performance in Mandarin speech recognition tasks. Based on this foundation, fine-tuning training was performed on speech data from different sub-dialect regions within D8 to adapt to the dialectal characteristics of each region. In terms of training strategy, to fully inherit the linguistic knowledge of the Mandarin model while simultaneously improving its dialect modeling capabilities, a "partial parameter freezing" approach was adopted. This involves freezing the underlying network parameters in the Paraformer encoder, preserving its basic modeling capabilities for Mandarin audio. Only the high-level structures within the encoder are trained to enhance the model's perception of dialect speech characteristics. This strategy allows the dialect recognition model to maintain its original basic capabilities while achieving superior dialect recognition and generalization performance, as shown in Table 1.

[0047] Table 1 Comparison of word error rates in various cities before and after fine-tuning of the present invention

[0048] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A non-autoregressive end-to-end dialect identification method for power supply service telephone system, characterized by: The following steps are involved: Step 1: Preprocessing of Dual-channel Telephone Recordings Based on the dual-channel recording data from the power supply service telephone system, an audio processing tool was used to split the audio into two mono channels, corresponding to the customer and customer service roles respectively, and unified them into WAV format with a 16kHz sampling rate. A voice activity detection model was applied to remove non-speech segments from the audio and construct valid audio segments for the left and right channels suitable for annotation. Step 2: Automatic pre-annotation and context building Combine the valid left and right channel audio clips constructed in step 1 into a complete conversation in the original call time order, construct a corpus dataset with contextual semantics, and automatically transcribe them based on the Mandarin dialect recognition base model to generate a dialect pre-annotated dataset of "audio ID-initial text"; Step 3: Manual annotation and dialect vocabulary construction Based on the dialect pre-annotated dataset in step 2, manually annotate according to the established dialect annotation standards to construct a prefecture-level dialect dictionary, thereby forming a standardized manually annotated dataset; the dialect annotation standards include annotation standards for dialect pronunciation differences, annotation standards for named entities and numbers, and contextual review standards for semantic ambiguity; Step 4: Sub-dialect area division and classification modeling Combining the experience of linguistic experts with acoustic feature cluster analysis, based on the standardized manually annotated dataset obtained in step 3, dialects with similar acoustic features are grouped into the same sub-dialect area, thus constructing a dialect area classification dataset. A multi-classification acoustic model for sub-dialect area identification is trained based on the dialect area classification dataset to automatically identify the dialect area to which the input audio belongs. Step 5: Dialect Recognition Model Training and Optimization Based on the standardized manually annotated speech dataset obtained in step 3, construct a character-level training corpus; Taking the Mandarin speech recognition pre-training model as the basic model, the low-level structure of its encoder is frozen to retain the general speech representation ability, and the high-level structure is fine-tuned to adapt to the dialect characteristics of each sub-region; for each sub-dialect region, the dialect recognition model is independently trained on its corresponding corpus; during recognition, the sub-dialect region recognition multi-classification acoustic model of step 4 is first used to determine the region to which the input audio belongs, and then the dialect recognition model of the corresponding sub-region is called for transcription; at the same time, a mechanism combining automatic annotation and manual review is introduced to continuously optimize the training corpus and model parameters to improve the overall recognition performance.

2. The non-autoregressive end-to-end dialect recognition method for a power supply service telephone system according to claim 1, characterized in that: The non-voice segments in step 1 include ringtones, silent segments, and background noise.

3. The non-autoregressive end-to-end dialect identification method for a power supply service telephone system according to claim 1, characterized in that: The dialect pronunciation differences in step 3 include differences in initial consonants, finals, and tones.

4. The non-autoregressive end-to-end dialect identification method for a power supply service telephone system according to claim 1, characterized in that: The step 1 of pre-processing the dual-channel telephone recording corpus includes the following steps: 11) Constructing the original audio dataset D1: Collecting the full set of telephone recordings aggregated from the power grid "cloud" platform, we organized the data into different levels, categorized by language region, prefecture, and county. We ensured that the total number of recordings was no less than 1 million, with no fewer than 150,000 recordings from each prefecture and city. This included recordings of calls from men, women, and children of all ages across all counties, ensuring language diversity, demographic diversity, and representativeness. 12) Constructing left and right mono audio dataset D2: Based on the original audio dataset D1, we used audio processing tools to split each two-channel phone recording into two mono tracks, one for each speaker (the customer) and one for the customer service representative. The generated left and right audio tracks were all in a unified WAV format with a sampling rate of 16kHz to ensure consistency in subsequent speech processing and model input. 13) Constructing valid short audio segments D3: For the left and right channel single-track dataset D2, a voice activity detection model is introduced to automatically extract the start and end timestamps of valid speech in each long audio segment. Based on the VAD detection results, the audio is sliced ​​to generate several short speech segments. Subsequently, the invalid segments are further filtered out by combining the time annotation information of non-speech events, retaining only segments containing clear and complete speech information, thereby constructing valid short audio segments D3 for subsequent pre-labeling and manual annotation.

5. The non-autoregressive end-to-end dialect identification method for a power supply service telephone system according to claim 4, characterized in that: The automatic pre-annotation and context building step 2 includes the following steps: 21) Constructing contextual semantics corpus dataset D4: Based on the valid left and right channel audio clips obtained in step 1, sort and merge the short audio clips of the left and right channels according to the chronological order of the original call to reconstruct the complete conversation sequence of the two parties. The result is shown on the right side of Figure 3, forming contextual semantics corpus dataset D4. 22) Constructing the dialect pre-annotated dataset D5: Based on the contextual semantics corpus dataset D4, we used the Mandarin dialect recognition base model to transcribe each valid short audio clip, generating the corresponding Chinese text. Finally, we generated a text.txt file containing the "audio ID - pre-annotated text" pair. That is, each phone call contained a text.txt file, thus constructing the dialect pre-annotated dataset D5.

6. The non-autoregressive end-to-end dialect identification method for a power supply service telephone system according to claim 5, characterized in that: The step 3 of manual annotation and dialect vocabulary construction includes the following steps: 31) Word segmentation to assist manual annotation: Based on the dialect pre-annotated dataset D5, preliminary automatic word segmentation is performed on the pre-annotated audio transcriptions in the text.txt file, with structured segmentation performed based on the vocabulary within the sentence. 32) Dialect lexicon construction: Based on the lexical differences between dialects and Mandarin identified during manual annotation, uncommon characters with similar pronunciations to standard Mandarin are used as replacements for annotations. A mapping relationship between Mandarin and dialect word pairs is established. Each city independently constructs a dialect lexicon based on local phonetic characteristics, and ultimately, the dialect lexicons from each city are compiled to form a dialect lexicon. 33) Construction of the manually annotated dataset D6: Local personnel were assigned to annotate the text.txt file corresponding to each phone call according to annotation standards that account for dialect pronunciation differences. The pre-annotated text corresponding to the audio file ID was modified to generate "audio ID-manually annotated text" pairs. These manually annotated "audio ID-manually annotated text" pairs were systematically post-processed to form the standardized manually annotated dataset D6.

7. The non-autoregressive end-to-end dialect identification method for a power supply service telephone system according to claim 6, characterized in that: The systematic post-processing includes: first, reading and applying the constructed dialect vocabulary to perform consistency verification, redundancy removal, and normalization on "Mandarin-dialect" word pairs; second, organizing reviewers with relevant experience to conduct a sampling quality assessment of the annotated data, requiring a manual annotation pass rate of more than 80% to be used as subsequent model training data; data that does not meet the standard must be re-annotated; finally, a standardized manually annotated dataset D6 is formed.

8. The non-autoregressive end-to-end dialect identification method for a power supply service telephone system according to claim 6, characterized in that: Standards for noting dialect pronunciation differences: When the audio transcription contains expressions that differ from Mandarin in terms of tone, initial consonants, or finals, rare characters with very low frequency of use are used to replace them to maximize the preservation of phonetic characteristics and ensure consistency of annotation; Named entity and number annotation standards: In scenarios involving standard named entities such as place names, personal names, time, and numbers, standard Mandarin Chinese characters must be used for annotation. The use of dialect words, colloquial symbols, or numerical symbols as substitutes is prohibited. Contextual review standards for semantically ambiguous audio clips: When faced with potentially ambiguous audio clips, annotators should combine the previously constructed dataset D4 with contextual semantic information and fully refer to the spoken context to avoid incorrect annotations due to being out of context.

9. The non-autoregressive end-to-end dialect identification method for a power supply service telephone system according to claim 1, characterized in that: Step 4: Sub-dialect area division and classification modeling, including the following steps: 41) Constructing the Dialect Area Classification Dataset D7: In the original audio dataset D1, the audio segments were initially organized according to the "language area - prefecture - county" hierarchical structure. This initial organization involved: first, systematically dividing Mandarin dialects based on the classification experience of linguistics experts, and preliminarily assigning audio samples from each county-level city to a dialect area language family. Then, based on the high-dimensional semantic features extracted by the Paraformer model framework, t-SNE technology was used for visualization analysis to assist in verifying and optimizing the accuracy of the attribution of audio samples from each county-level city to the sub-dialect area, achieving a more data-driven area classification. Finally, a corresponding area label was automatically generated for each audio sample, and the subdialect.txt file was constructed as the dialect area classification dataset D7. Each line in the file records a corresponding "audio ID - dialect area" relationship, which serves as the basis for subsequent modeling and management. 42) Constructing a sub-dialect classification algorithm: Based on the dialect area classification dataset D7, a multi-classification acoustic model for sub-dialect area identification is trained using the Paraformer model architecture to achieve automatic identification of sub-dialect areas. The multi-classification acoustic model takes audio samples as input and outputs the area label to which they belong, thereby supporting the automatic classification of dialect areas in newly recorded data and providing an important foundation for subsequent dialect recognition systems.

10. The non-autoregressive end-to-end dialect identification method for a power supply service telephone system according to claim 1, characterized in that: The step 5 of dialect recognition model training and optimization includes the following steps: 51) Character-level modeling of annotated text: For manually annotated text data, a character-level modeling strategy is used, using "character" as the smallest morpheme unit. This means that spaces are inserted between each Chinese character, so that each Chinese character corresponds to an independent token. 52) Constructing the dialect recognition model training dataset D8: Based on the standardized manually annotated dataset D6, each "audio ID - manually annotated text" pair was organized into a standardized JSON-formatted data item. Each JSON data item contained the following fields: unique audio file identifier, audio file path, total audio duration, corresponding annotated text content, and the total number of Chinese characters in the annotated text. Based on the statistical results of the total audio duration, dataset D6 was divided into a training set and a validation set in an 8:2 ratio, thus forming the complete dialect recognition model training dataset D8 for subsequent modeling. 53) Training the sub-dialect recognition model: Using the Paraformer model as the foundation, fine-tuning training is performed on the speech data of different sub-dialect regions in the dialect recognition model training dataset D8 to adapt to the dialect characteristics of each region. The fine-tuning training includes: in terms of training strategy, adopting a "partial parameter freezing" method, that is, freezing the underlying network parameters in the Paraformer encoder to retain its basic modeling ability for Mandarin audio; only training the high-level structures in the encoder to enhance the model's perception of dialect speech characteristics.

Images