Power supply service voice system supporting multi-language mixed recognition and processing

Abstract

According to the power supply service voice system supporting multi-language mixed recognition and processing, accurate recognition and natural synthesis of multi-language and multi-dialect mixed voice are achieved by fusing the Conformer acoustic model and the dynamic n-gram language model, the generalization ability of the model for regional dialects and accent is optimized, and the recognition accuracy of cross-language scenes is improved. Moreover, by integrating the power grid business term library and the knowledge graph, the semantic analysis capability of the model for the power industry scene is optimized, so that the technical term misjudgment rate can be reduced, the power business interaction accuracy can be improved, and the recognition accuracy can be further improved.

Description

Technical Field

[0001] This application relates to the field of power supply service technology, and in particular to a power supply service voice system that supports multilingual mixed recognition and processing. Background Technology

[0002] Against the backdrop of rapid global informatization and intelligentization, the level of intelligence in power supply services, as a core component of basic social services, directly impacts the quality of life for the public and regional economic development. However, current intelligent power supply services face numerous challenges, with language barriers being particularly prominent.

[0003] Most existing smart power supply service systems are developed based on a single language, with Mandarin as the primary service language. This makes it difficult to meet the language needs of different regions and user groups. The system has poor adaptability to regional dialects such as Cantonese and slang, as well as accents such as European Portuguese variants, resulting in significant semantic understanding errors in cross-language scenarios. This necessitates dedicated personnel to support both dialects and other languages. Furthermore, existing systems cannot achieve high recognition accuracy in multilingual scenarios, such as those involving mixed Chinese and English or Chinese and Portuguese. Therefore, it is clear that existing smart power supply service systems cannot cover international service needs, resulting in limited service coverage, poor user experience, and severely hindering the widespread adoption and application of smart power supply service systems. Summary of the Invention

[0004] The purpose of this application is to address at least one of the aforementioned technical deficiencies, particularly the low accuracy of identification in the prior art.

[0005] In some embodiments, this application provides a power supply service voice system that supports multilingual mixed recognition and processing, including: a multilingual and multi-dialect mixed speech recognition model, N natural language processing models and N speech synthesis models, where N is a positive integer greater than 1;

[0006] The multilingual and multi-dialect mixed speech recognition model includes a Conformer acoustic model and a dynamic n-gram language model. The multilingual and multi-dialect mixed speech recognition model and the natural language processing model are constructed based on a preset power grid business terminology database and power grid business knowledge graph.

[0007] The N natural language processing models and the N speech synthesis models correspond one-to-one to form N processing links, and any two processing links are used to process different languages ​​or dialects;

[0008] The multilingual and multi-dialect mixed speech recognition model is used to acquire speech requests for power supply services, perform speech recognition on the speech requests for power supply services to obtain speech text; perform language dialect type recognition on the speech requests for power supply services to obtain language dialect type; and output the speech text to the corresponding processing link according to the language dialect type.

[0009] The natural language processing model is used to perform intent recognition on the received speech text using a natural language understanding algorithm, generate service response text based on the intent recognition result, and input the service response text into the speech synthesis model in the same processing link.

[0010] The speech synthesis model is used to generate a corresponding speech response when the service response text is received.

[0011] In one embodiment, the speech synthesis model is implemented using Mel spectrum generation technology based on a fast diffusion model and a Vocos vocoder.

[0012] In one embodiment, the speech synthesis model includes an encoder that is implemented based on an unsupervised monotonic alignment algorithm.

[0013] In one embodiment, the multilingual and multi-dialect mixed speech recognition model includes a speech activity detection module based on a connection-time classification algorithm.

[0014] In one embodiment, the multilingual, multi-dialect mixed speech recognition model is used to acquire speech requests for power supply services, and the process of performing speech recognition on the speech requests for power supply services to obtain speech text includes:

[0015] The multilingual and multi-dialect mixed speech recognition model is used to enter a first transition state when the current state is non-speech state and a valid audio frame is received; if the duration of the first transition state exceeds a first preset duration threshold, it enters the speech state to perform speech recognition on the valid audio frame and obtain the speech text.

[0016] The multilingual and multi-dialect mixed speech recognition model is also used to enter a second transition state when the current state is the speech state and an invalid audio frame is received; if the duration of the second transition state exceeds a second preset duration threshold, it enters the non-speech state to stop speech recognition.

[0017] In one embodiment, the training process of the multilingual, multi-dialect mixed speech recognition model includes:

[0018] Obtain the first training set, which includes full training data of multilingual dialects;

[0019] The initial model is trained using the first training set to obtain a basic speech recognition model.

[0020] Supplementary training data corresponding to the target language dialect is generated based on audio cloning technology, and the supplementary training data is added to the first training set until the proportion of training data of various language dialects in the first training set after addition meets the preset proportion balance condition and a second training set is obtained; wherein, the target language dialect is a language or dialect whose proportion of training data in the first training set is lower than a preset threshold.

[0021] The basic speech recognition model is fine-tuned using the second training set to obtain the multilingual and multi-dialect mixed speech recognition model.

[0022] In one embodiment, the process of generating supplementary training data corresponding to the target language dialect based on audio cloning technology includes:

[0023] Multiple user question texts are constructed based on power supply service scenarios;

[0024] By performing batch semantic generalization on each user question text using a large language model, multiple generalized question texts corresponding to the target language dialect are obtained.

[0025] Using voiceprint cloning technology and speech synthesis model, supplementary training speech corresponding to each of the generalization problem texts is generated;

[0026] The supplementary training data is obtained based on each of the generalization problem texts and each of the supplementary training speech.

[0027] In one embodiment, the full training data includes full training speech obtained by denoising the original full speech using a target denoising algorithm, and the supplementary training data includes supplementary training speech obtained by denoising the original supplementary speech using the target denoising algorithm, wherein the target denoising algorithm is a denoising algorithm used by power supply service smart devices.

[0028] In one embodiment, the multilingual and multi-dialect mixed speech recognition model, the natural language processing model, and the speech synthesis model are deployed using a private deployment scheme.

[0029] The multilingual and multi-dialect mixed speech recognition model, the natural language processing model, and the speech synthesis model transmit data through the power grid.

[0030] In one embodiment, N is 4, and the 4 processing links are used to process Mandarin, English, Cantonese and Portuguese respectively.

[0031] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0032] This application achieves accurate recognition and natural synthesis of mixed speech in multiple languages ​​and dialects by integrating the Conformer acoustic model and a dynamic n-gram language model. It also optimizes the model's generalization ability to regional dialects and accents, improving recognition accuracy in cross-language scenarios. Furthermore, by integrating a power grid business terminology database and knowledge graph, this application optimizes the model's semantic parsing capabilities for power industry scenarios, thereby reducing the misjudgment rate of technical terms, improving the accuracy of power business interactions, and further enhancing recognition accuracy. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 Here is a system architecture diagram of the power supply service voice system in some embodiments;

[0035] Figure 2 This is a schematic diagram illustrating the training process of the acoustic model in some embodiments;

[0036] Figure 3 Here is a system architecture diagram of a speech synthesis model based on a fast diffusion model in some embodiments;

[0037] Figure 4 This describes the overall acquisition process of a multilingual and multi-dialect mixed speech recognition model in some embodiments;

[0038] Figure 5 This is a schematic diagram illustrating the workflow of a multilingual and multi-dialect mixed speech recognition model in some embodiments. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The technical terms used in this application are explained below:

[0041] Conformer acoustic model: A deep learning model that combines the advantages of Transformer (a neural network based on self-attention mechanism) and CNN (Convolutional Neural Network), suitable for speech recognition tasks. By capturing long-range dependencies (Transformer) and local features (CNN), it improves the recognition accuracy of multilingual mixed speech.

[0042] Dynamic n-gram language model: A statistical language model that dynamically adjusts the window size of the n-gram based on the context (e.g., 3-gram predicts the probability of the next word based on the first two words) to correct the output of the acoustic model and improve semantic accuracy in cross-language mixed scenarios.

[0043] Fast Diffusion Model: A generative model that generates high-quality speech features (such as Mel spectra) from noisy data through a stepwise back-diffusion process. Compared to traditional diffusion models, it employs optimal transport condition flow matching technology to accelerate inference speed while maintaining output quality.

[0044] ASR (Automatic Speech Recognition): A technology that converts speech signals into text, including modules such as acoustic models, language models, and decoders.

[0045] VAD (Voice Activity Detection): Detects the start and end points of valid speech segments in a speech signal, filters background noise, and improves the accuracy of speech recognition.

[0046] TTS (Text-to-Speech): A technique for converting text into natural speech, which includes steps such as text encoding, duration prediction, acoustic feature generation, and waveform synthesis.

[0047] Unsupervised Monotonic Alignment Search: A text-audio alignment method that requires no manual annotation. It automatically matches the temporal correspondence between text and speech through algorithms, simplifying the training data preparation process for multilingual speech synthesis.

[0048] Tone-Aware modeling: a speech recognition technology for tonal languages ​​(such as Mandarin and Cantonese) that improves the accuracy of dialect and accent recognition by modeling tone changes.

[0049] Knowledge graph: A structured semantic network that represents domain knowledge through entities, attributes, and relationships. This system integrates a knowledge graph from the power industry to enhance the professionalism of semantic parsing.

[0050] NLP (Natural Language Processing): The technology that enables computers to understand and generate natural language text, including two parts: Natural Language Understanding (NLU) and Natural Language Generation (NLG).

[0051] As mentioned in the background section, existing smart power supply service systems suffer from low recognition accuracy in multilingual mixed recognition scenarios, failing to meet international service needs. This results in limited service coverage, a poor user experience, and severely hinders the widespread adoption and application of smart power supply service systems. In addition, existing technologies also exhibit the following technical shortcomings:

[0052] (1) High risk of data security: In the existing technology, mainstream voice engines rely on public cloud services for voice processing, and user privacy data and sensitive business information are at risk of leakage, which makes it difficult to meet the strict requirements of the power industry for localized closed-loop management of data.

[0053] (2) Insufficient industry specialization adaptation: The existing technology uses a general speech engine, but the general speech engine lacks deep integration of professional knowledge base and terminology base in the power field. It has weak semantic understanding ability for professional scenarios in the power industry, high misjudgment rate, and cannot meet the complex interaction needs of the power industry.

[0054] (3) It is difficult to balance the efficiency and quality of speech synthesis: Traditional speech synthesis technology has the problems of large amount of computation and slow synthesis speed. High naturalness synthesis requires complex models, resulting in high resource consumption and long response delay, making it difficult to achieve real-time interaction in low-latency scenarios.

[0055] To address the aforementioned technical deficiencies, this application provides a power supply service voice system that supports multilingual mixed recognition and processing, and possesses at least one of the following advantages:

[0056] (1) Multilingual mixed recognition and dialect adaptation capabilities are significantly improved

[0057] Existing systems typically achieve an accuracy rate below 80% for multilingual mixed inputs (e.g., a mix of Mandarin and English, or Cantonese and English) and regional dialects (e.g., Cantonese variants, Portuguese variants). This application, however, significantly improves the generalization ability of multilingual and multi-dialect mixed inputs by integrating the Conformer acoustic model and a dynamic n-gram language model, combined with tone-aware modeling and cross-lingual prosodic processing, achieving a stable accuracy rate of ≥90% in cross-lingual scenarios. Furthermore, for terms specific to the power industry, the application further reduces the misjudgment rate of specialized semantics by integrating a power industry knowledge graph and terminology database.

[0058] (2) Data security and localized deployment

[0059] Existing systems rely on public cloud services, posing a risk of user privacy breaches. This application, however, adopts a closed-loop, end-to-end intranet processing architecture. All voice data collection, recognition, and synthesis are completed within the power grid's intranet, avoiding reliance on third-party public cloud services, ensuring zero user data leakage, and meeting the power industry's stringent requirements for data sovereignty and security. Furthermore, the private deployment solution supports flexible expansion and can adapt to the localized needs of different regional power grids (such as Cantonese-Portuguese bilingual services in the Qingdao-Macau region).

[0060] (3) Optimizing the balance between speech synthesis efficiency and naturalness

[0061] Traditional speech synthesis technology relies on complex models, resulting in high computational resource consumption and high response latency (typically ≥2000 milliseconds). In contrast, this application adopts Mel spectrum generation technology based on a fast diffusion model and an optimized Vocos vocoder, which significantly reduces computational complexity and synthesis latency while ensuring the naturalness of synthesized speech. The average response time is shortened to ≤1500 milliseconds, enabling efficient synthesis of multilingual speech, significantly improving the real-time interactive experience, and meeting the real-time interactive needs of power business scenarios.

[0062] (4) Deep adaptation to specialization in the power sector

[0063] Existing general-purpose voice engines lack targeted optimization for power industry scenarios. This application enhances the ability to parse industry-specific semantics by constructing a power grid business terminology database and a fault diagnosis knowledge graph. For example, when a user queries "peak-valley electricity pricing policy," the voice system can accurately link electricity policy documents with the user's electricity consumption data, outputting a structured explanation and reducing the misjudgment rate by 35%. Simultaneously, it supports multimodal interaction within the power industry (such as voice command control for meter data queries), improving the level of service intelligence.

[0064] (5) Innovation of hybrid data generation and enhancement strategies

[0065] To address the shortage of rare language data (such as mixed Chinese-Portuguese speech), this application innovatively combines large language model generalization generation, TTS synthesis, and speaker cloning techniques to efficiently construct a high-precision training dataset. Compared to traditional manual annotation methods, data generation efficiency is improved by 70%, and tone perception enhancement technology is used to solve the problem of dialect pronunciation differences, providing sufficient and diverse corpus support for model training.

[0066] (6) Innovative speech processing flow and modules

[0067] This application employs optimized VAD technology combined with an intelligent output result judgment mechanism to improve the accuracy of speech segment extraction and the reliability of recognition results. Precise time boundary normalization technology is used to ensure that the timestamp of the recognition result matches the original speech data, enhancing output usability. A text-audio alignment module based on unsupervised monotonic alignment search simplifies the training data preparation process for multilingual speech synthesis and improves system deployment efficiency.

[0068] The following is a description of the voice system for power supply service provided in this application.

[0069] In some embodiments, such as Figure 1 As shown, this application provides a power supply service voice system supporting multilingual mixed recognition and processing, including: a multilingual and multi-dialect mixed speech recognition model, N natural language processing models, and N speech synthesis models, where N is a positive integer greater than 1. In some examples, the system may also include a multilingual and multi-dialect mixed voice wake-up module, which is used to activate the entire voice system and put it into a standby state when a specific wake word or specific voice command is detected, preparing it to receive and process subsequent voice input.

[0070] The multilingual and multi-dialect mixed speech recognition model includes the Conformer acoustic model and the dynamic n-gram language model. Both the multilingual and multi-dialect mixed speech recognition model and the natural language processing model are constructed based on a pre-defined power grid business terminology database and a power grid business knowledge graph. The power grid business terminology database may include multiple terms related to power supply services and power grid business.

[0071] Specifically, the Conformer acoustic model refers to an acoustic model trained using the Conformer algorithm. Conformer combines the long-range dependency capture capability of the Transformer architecture with the local feature extraction advantages of convolutional neural networks. Through the synergistic effect of multi-head self-attention mechanisms and convolutional modules, it can efficiently process complex features in speech signals. This architecture is particularly suitable for end-to-end speech recognition systems, directly mapping speech to text, effectively improving recognition accuracy and robustness. It also demonstrates excellent adaptability in multilingual and multi-dialect recognition tasks, providing strong support for the efficient operation of the system in complex speech environments. In some embodiments, the training process of the acoustic model can be as follows: Figure 2 As shown.

[0072] Language models are used to model the probability distribution of language, helping to correct the output of acoustic models and improve overall recognition performance. This application can use a dynamic n-gram algorithm to implement the language model, resulting in a dynamic n-gram language model. The dynamic n-gram algorithm predicts the probability of the next word based on the previous n-1 words. The value of n can be determined according to the actual situation, and this application does not impose specific restrictions on it. Taking the dynamic 3-gram algorithm as an example, the dynamic 3-gram algorithm predicts the probability of the next word based on the previous two words. Assuming the word sequence is w1, w2, ... wn, the formula for calculating the occurrence of a word is as follows:

[0073]

[0074] In the formula, C(w) i-2 ,w i-1 ,w i ) is a word sequence w i-2 ,w i-1 ,w i The number of times it appears in the corpus. C(w) i-2 ,w i-1 ) is a word sequence w i-2 ,w i-1 The number of times it appears in the corpus.

[0075] In some examples, to achieve better low-latency, low-cost computational inference, the language model can be trained using a 3-gram extension to the first two consecutive terms.

[0076] N natural language processing models and N speech synthesis models are paired one-to-one to form N processing chains. Any two processing chains are used to process different languages ​​or dialects. It is understood that N can be set according to the specific circumstances.

[0077] In some examples, N=4, meaning the speech system provided in this application may include four natural language processing models and four speech synthesis models. These four natural language processing models and four speech synthesis models correspond one-to-one to form four processing links, which are respectively used to provide speech services in Mandarin, English, Cantonese, and Portuguese. For ease of explanation, the following embodiments use the example of a speech system providing Mandarin, English, Cantonese, and Portuguese.

[0078] In the process of providing voice services, a multilingual, multi-dialect hybrid speech recognition model is used to acquire the voice request for power supply services and perform speech recognition on it to obtain speech text. This model also identifies the language and dialect type of the voice request, determines the dialect type, and outputs the speech text to the corresponding processing link based on the dialect type. A natural language processing model, upon receiving the speech text, uses natural language understanding algorithms to perform intent recognition on the speech text and generates a service response text based on the intent recognition result. This service response text is then input into a speech synthesis model within the same processing link. Finally, a speech synthesis model, upon receiving the service response text, generates the corresponding response speech.

[0079] Specifically, the multilingual, multi-dialect hybrid speech recognition model can detect user voice input through a hybrid recognition engine. When valid voice input is detected, it determines whether the voice contains a wake word, thus determining whether the voice system is activated. If activated, the multilingual, multi-dialect hybrid speech recognition model can acquire the user's voice request for power service and perform speech recognition to obtain the speech text. Furthermore, when activated, the voice system enters a language determination phase to accurately identify the user's language, including Mandarin, Cantonese, English, and Portuguese, providing the correct processing direction for subsequent language understanding.

[0080] Based on the identified language and dialect type, the multilingual and multi-dialect mixed speech recognition model will send the speech text to the corresponding natural language processing module for processing.

[0081] These natural language processing modules include Mandarin, Cantonese, English, and Portuguese understanding modules. Each module is capable of processing input in its corresponding language and can understand the user's intent and needs based on the voice text. It then generates service response text to address the user's needs. Through this step, the voice system can accurately capture the user's instructions or questions and generate response text that matches the user's power supply service requirements, providing a foundation for subsequent operations.

[0082] After generating the service response text, the natural language understanding module sends it to the speech generation model in the same processing chain to generate a response speech in the corresponding language / dialect. After processing the user's voice input, the speech system can output the information to the user in a broadcast format. This output process may include text-to-speech synthesis, speech playback, and possible feedback prompts. Thus, users can perform operations through interactive control throughout the entire process, such as adjusting volume, selecting the broadcast language, and interrupting playback. This interactive control mechanism makes the system more flexible and user-friendly, meeting the needs and preferences of different users.

[0083] The above embodiments achieve accurate recognition and natural synthesis of mixed speech in multiple languages ​​and dialects by integrating the Conformer acoustic model and the dynamic n-gram language model. They also optimize the model's generalization ability to regional dialects and accents, improving recognition accuracy in cross-language scenarios. Furthermore, this application optimizes the model's semantic parsing ability for power industry scenarios by integrating a power grid business terminology database and knowledge graph, thereby reducing the misjudgment rate of technical terms, improving the accuracy of power business interactions, and further enhancing recognition accuracy.

[0084] In some embodiments, the speech synthesis model is implemented using Mel-spectrum generation technology based on a fast diffusion model and a Vocos vocoder. The speech synthesis model based on the fast diffusion model can be implemented as follows: Figure 3 As shown, phoneme input refers to the input scheme that uses Chinese characters to convert to phonemes, which facilitates fine-grained control over pronunciation. The text editor refers to the Transformer structure based on a multi-head attention mechanism. Duration prediction refers to predicting the pronunciation duration of each phoneme based on the input of the text encoder. Upsampling refers to length expansion of the hidden layer input based on the duration prediction result. The diffusion model refers to obtaining the final Mel spectrum by back-diffusion through several steps based on the mean of the input Mel spectrum. The vocoder is used to realize the fast conversion from Mel spectrum to speech.

[0085] Fast diffusion models are suitable for generating high-quality continuous features, but their slow inference process limits their wider application. This paper utilizes state-of-the-art optimal transport condition flow matching techniques to improve the inference speed of diffusion models while maintaining their original output quality.

[0086] The vocoder can employ the fast vocoder Vocos, and the training loss of Vocos can be adjusted to improve sound quality. The optimized Vocos vocoder used in this application significantly improves computational speed while maintaining speech quality.

[0087] In some embodiments, the speech synthesis model further includes an encoder, which is implemented based on an unsupervised monotonic alignment algorithm to simplify the preparation of training data.

[0088] In some embodiments, the multilingual and multi-dialect mixed speech recognition model includes a speech activity detection module based on the connection-time classification (CTC) algorithm. Specifically, the multilingual and multi-dialect mixed speech recognition model can process the input speech data through VAD technology to identify the start and end points of speech, thereby effectively extracting the speech segments to be recognized. These extracted speech segments are then decoded to convert the speech signals into recognizable text information.

[0089] Traditional Voice Awareness (VAD) technology segments audio based on audio energy points. However, the presence of noise in the audio can negatively impact VAD's segmentation accuracy, causing the model to detect longer segments with inherent noise, thus affecting recognition performance. This embodiment, however, uses a Critical Tone Classification (CTC) model for audio segmentation. It decodes and infers the attributes of each frame, identifying segments exceeding a threshold. This approach effectively distinguishes between valid and invalid speech. VAD segments invalid speech segments, ensuring that only valid audio fragments are recognized by the model, further improving accuracy.

[0090] In some embodiments, the process of using a multilingual, multi-dialect mixed speech recognition model to acquire speech requests for power supply services and to perform speech recognition on those requests to obtain speech text includes:

[0091] The multilingual and multi-dialect mixed speech recognition model is used to enter a first transition state when the current state is non-speech state and a valid audio frame is received; if the duration of the first transition state exceeds a first preset duration threshold, it enters the speech state to perform speech recognition on the valid audio frame and obtain speech text.

[0092] The multilingual and multi-dialect mixed speech recognition model is also used to enter a second transition state when the current state is speech state and an invalid audio frame is received; if the duration of the second transition state exceeds a second preset duration threshold, it enters a non-speech state to stop speech recognition.

[0093] Specifically, when the multilingual and multi-dialect mixed speech recognition model is in a non-speech state, i.e., the initial state, upon receiving a valid audio frame, it enters a non-speech to speech transition state (i.e., the first transition state). The model can count the duration of the first transition state to obtain its duration. If the duration of the first transition state exceeds a first preset duration threshold, it enters the speech state, the model begins speech recognition, and outputs the speech text.

[0094] When a multilingual, multi-dialect mixed speech recognition model is in speech mode, upon receiving a non-audio frame, it transitions from speech mode to non-speech mode (i.e., a second transition state). The model can calculate the duration of the second transition state. If the duration of the second transition state exceeds a second preset duration threshold, it enters non-speech mode and stops recognition.

[0095] In this way, precise time boundary normalization technology can ensure that the timestamp of the recognition result perfectly matches the original speech data, improving time accuracy and the usability of the recognition results. This innovation is particularly important for speech recognition applications that require precise timestamps.

[0096] In some embodiments, this application may also employ an intelligent output result judgment mechanism. Specifically, the multilingual and multi-dialect mixed speech recognition model is a model constructed by combining an acoustic model and a language model. The acoustic model is responsible for extracting audio sound features and outputting acoustic spikes. Combined with a given sequence from the language model, the engine continuously adjusts the recognition result based on the acoustic spikes. The engine has an output judgment mechanism; if the optimal result is achieved, the recognition result is returned, and the output is determined according to the set criteria, thereby improving the reliability and efficiency of recognition.

[0097] In some embodiments, the overall acquisition process of a multilingual and multi-dialect mixed speech recognition model can be as follows: Figure 4 As shown, it specifically includes:

[0098] 1. Data collection and processing

[0099] (1) Training audio data collection and annotation;

[0100] (2) Collect open-source shared datasets;

[0101] (3) Crawling data from data websites to obtain training audio data, provided that laws, regulations and public order are met;

[0102] (4) For scarce data (such as mixed Mandarin and English, or mixed Cantonese and English data), construct user questions based on actual business application scenarios, and generate high-quality questions in batches through large language models (such as DeepSeek-R1, Tongyi Thousand Questions).

[0103] (5) Data was obtained by cloning seed audio from different users using cosyvoice technology.

[0104] (6) Use TTS (text-to-speech) synthesis technology to generate training audio data.

[0105] (7) Data preprocessing: The collected data is processed by calling two sets of corresponding language ASR engines for recognition and comparing the processing results to quickly label a large amount of data.

[0106] (8) Data augmentation strategies are adopted, such as adding a small amount of noise, handling homonyms, cross-language prosody processing, and using tone-aware to achieve tone modeling; audio with severe background noise is removed or noise is reduced by adaptive algorithms; audio speech rate is normalized and the time axis of the audio signal is adjusted so that all audio segments have a similar average speech rate.

[0107] 2. Change the audio pitch and adjust the audio volume.

[0108] 3. Train the acoustic model and the language model.

[0109] 4. Verify the ASR recognition rate of the trained multilingual and multi-dialect mixed speech recognition model to ensure that its performance meets the requirements.

[0110] In some embodiments, the workflow of a multilingual, multi-dialect mixed speech recognition model can be as follows: Figure 5 As shown. Figure 5 As shown, the input speech data is processed using VAD technology to identify the start and end points of the speech, thereby effectively extracting the speech segments to be recognized. These extracted speech segments are then decoded, converting the speech signal into recognizable text information. After decoding, a decision step is performed to determine whether to output the current recognition result. This step can be based on some criterion or condition, such as the confidence level of the recognition result. If it is decided to output the recognition result, time boundary normalization is then performed to ensure that the timestamp of the recognition result matches the timestamp of the original speech data, thereby improving the accuracy of the recognition result. Finally, post-processing of the recognition result is performed, which may include noise removal, error correction, and formatted output, ultimately delivering the processed recognition result to the user.

[0111] In some embodiments, the training process of a multilingual and multi-dialect mixed speech recognition model includes:

[0112] Obtain the first training set, which includes full training data for multiple language dialects;

[0113] The initial model was trained using the first training set to obtain the basic speech recognition model.

[0114] Supplementary training data corresponding to the target language dialect is generated based on audio cloning technology, and the supplementary training data is added to the first training set until the proportion of training data of various language dialects in the first training set meets the preset proportion balance condition and a second training set is obtained; wherein, the target language dialect is the language or dialect whose proportion of training data in the first training set is lower than the preset threshold.

[0115] The basic speech recognition model was fine-tuned using the second training set to obtain a multilingual and multi-dialect mixed speech recognition model.

[0116] Specifically, the training of a multilingual, multi-dialect mixed speech recognition model can be divided into two progressive stages. The first stage is the construction of a basic model to obtain a foundational speech recognition model. In this stage, the model is trained using full training data for each language / dialect, allowing it to initially learn the common acoustic and linguistic features of different languages / dialects. However, the performance of the basic speech recognition model exhibits biases; languages / dialects with abundant data show better recognition results, while languages / dialects with scarce data tend to have their recognition results biased towards languages / dialects with high data resources due to insufficient feature learning. Therefore, the second stage, targeted optimization, involves reorganizing the training data.

[0117] In the second stage, by supplementing low-resource language / dialect samples and controlling the proportion of high-resource language / dialect samples, the training data ratio of each language / dialect can be made more balanced. This balanced data is then used to fine-tune the basic speech recognition model. After fine-tuning, the model captures features of each language more accurately, not only retaining the recognition advantages of high-resource languages / dialects but also significantly improving the bias problem of low-resource languages / dialects, ultimately achieving a comprehensive improvement in the recognition performance of all languages / dialects.

[0118] Considering that training data for languages / dialects such as Cantonese and Portuguese is difficult to obtain, which poses a challenge to data balance, this embodiment can generate training data for low-resource languages / dialects through audio cloning technology, thereby increasing the amount of training data for languages / dialects such as Cantonese and Portuguese, and thus achieving data balance between different languages.

[0119] In some embodiments, the process of generating supplementary training data corresponding to the target language dialect based on audio cloning technology includes:

[0120] Multiple user question texts are constructed based on power supply service scenarios;

[0121] By performing batch semantic generalization on each user's question text using a large language model, multiple generalized question texts corresponding to the target language dialect are obtained.

[0122] Using voiceprint cloning technology and speech synthesis model, supplementary training speech is generated for each generalization problem text.

[0123] Supplementary training data is obtained based on the generalization problem texts and supplementary training audios.

[0124] Specifically, given the high cost, long cycle, and limitations of traditional manual text data collection, this embodiment utilizes a large-scale language model for text generalization. This approach efficiently generates a wider range and richer variety of text questions, effectively improving the model's robustness and generalization ability.

[0125] In generating supplementary training data, this application can construct multiple user question texts based on power supply service scenarios. These user question texts can cover different power supply services and different user needs. For each user question text, a large-scale language model can be used to semantically generalize the user question text, obtaining the corresponding generalized question text. With each generalized question text obtained, voiceprint cloning technology and a speech synthesis model can be used to generate supplementary training speech corresponding to each generalized question text, and supplementary training data corresponding to the target language dialect can be obtained accordingly.

[0126] In some embodiments, the full training data includes the full training speech obtained by denoising the original full speech using a target denoising algorithm, and the supplementary training data includes the supplementary training speech obtained by denoising the original supplementary speech using a target denoising algorithm. The target denoising algorithm is a denoising algorithm used by a power supply service smart device, which may be a smart device used in a power service business hall.

[0127] In this embodiment, to improve the accuracy and robustness of speech recognition, the same noise reduction algorithm as in the actual application environment is used to process the original speech, and the noise-reduced speech is used as training data. During the data preprocessing stage, the same noise reduction algorithm and technology as those used in power supply service smart devices are applied to the original training speech to simulate speech signals in real-world scenarios, ensuring a high degree of consistency between the training scenario and the test recognition scenario, thereby improving the model's adaptability and recognition performance in actual business hall environments.

[0128] In some embodiments, to ensure high-quality generated data, this application may employ a dual-recognition engine comparison and filtering strategy. Specifically, two recognition engines targeting the corresponding language are used to perform cross-validation and comparative analysis on the generated data. Only when the recognition results of the two engines are highly consistent will the data be included in the training set. In this way, the quality of the training data can be controlled.

[0129] In some embodiments, the multilingual and multi-dialect mixed speech recognition model, natural language processing model, and speech synthesis model are deployed using a private deployment scheme. Data transmission for these models occurs through the power grid's intranet. This localized, private deployment framework enables the entire process of speech acquisition, recognition, synthesis, and semantic analysis to operate in a closed loop within the power grid's intranet, avoiding reliance on third-party public cloud services, ensuring zero leakage of user data, and meeting the power industry's stringent requirements for data sovereignty and security.

[0130] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the related listed items.

[0131] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0132] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A power supply service voice system supporting multilingual mixed recognition and processing, characterized in that, include: A multilingual and multi-dialect mixed speech recognition model, N natural language processing models, and N speech synthesis models, where N is a positive integer greater than 1; The multilingual and multi-dialect mixed speech recognition model includes a Conformer acoustic model and a dynamic n-gram language model. The multilingual and multi-dialect mixed speech recognition model and the natural language processing model are constructed based on a preset power grid business terminology database and power grid business knowledge graph. The N natural language processing models and the N speech synthesis models correspond one-to-one to form N processing links, and any two processing links are used to process different languages ​​or dialects; The multilingual and multi-dialect mixed speech recognition model is used to acquire speech requests for power supply services, perform speech recognition on the speech requests for power supply services to obtain speech text; perform language dialect type recognition on the speech requests for power supply services to obtain language dialect type; and output the speech text to the corresponding processing link according to the language dialect type. The natural language processing model is used to perform intent recognition on the received speech text using a natural language understanding algorithm, generate service response text based on the intent recognition result, and input the service response text into the speech synthesis model in the same processing link. The speech synthesis model is used to generate a corresponding speech response when the service response text is received.

2. The power supply service voice system supporting multilingual mixed recognition and processing according to claim 1, characterized in that, The speech synthesis model is implemented using Mel spectrum generation technology based on a fast diffusion model and a Vocos vocoder.

3. The power supply service voice system supporting multilingual mixed recognition and processing according to claim 2, characterized in that, The speech synthesis model includes an encoder, which is implemented based on an unsupervised monotonic alignment algorithm.

4. The power supply service voice system supporting multilingual mixed recognition and processing according to claim 1, characterized in that, The multilingual and multi-dialect mixed speech recognition model includes a speech activity detection module based on a connection-time classification algorithm.

5. The power supply service voice system supporting multilingual mixed recognition and processing according to claim 1, characterized in that, The multilingual, multi-dialect mixed speech recognition model is used to acquire speech requests for power supply services, and to perform speech recognition on the speech requests to obtain speech text. The process includes: The multilingual and multi-dialect mixed speech recognition model is used to enter a first transition state when the current state is non-speech state and a valid audio frame is received; if the duration of the first transition state exceeds a first preset duration threshold, it enters the speech state to perform speech recognition on the valid audio frame and obtain the speech text. The multilingual and multi-dialect mixed speech recognition model is also used to enter a second transition state when the current state is the speech state and an invalid audio frame is received; if the duration of the second transition state exceeds a second preset duration threshold, it enters the non-speech state to stop speech recognition.

6. The power supply service voice system supporting multilingual mixed recognition and processing according to any one of claims 1 to 5, characterized in that, The training process of the multilingual and multi-dialect mixed speech recognition model includes: Obtain the first training set, which includes full training data of multilingual dialects; The initial model is trained using the first training set to obtain a basic speech recognition model. Supplementary training data corresponding to the target language dialect is generated based on audio cloning technology, and the supplementary training data is added to the first training set until the proportion of training data of various language dialects in the first training set after addition meets the preset proportion balance condition and a second training set is obtained; wherein, the target language dialect is a language or dialect whose proportion of training data in the first training set is lower than a preset threshold. The basic speech recognition model is fine-tuned using the second training set to obtain the multilingual and multi-dialect mixed speech recognition model.

7. The power supply service voice system supporting multilingual mixed recognition and processing according to claim 6, characterized in that, The process of generating supplementary training data corresponding to the target language dialect based on audio cloning technology includes: Multiple user question texts are constructed based on power supply service scenarios; By performing batch semantic generalization on each user question text using a large language model, multiple generalized question texts corresponding to the target language dialect are obtained. Using voiceprint cloning technology and speech synthesis model, supplementary training speech corresponding to each of the generalization problem texts is generated; The supplementary training data is obtained based on each of the generalization problem texts and each of the supplementary training voices.

8. The power supply service voice system supporting multilingual mixed recognition and processing according to claim 6, characterized in that, The full training data includes full training speech obtained by denoising the original full speech using the target denoising algorithm, and the supplementary training data includes supplementary training speech obtained by denoising the original supplementary speech using the target denoising algorithm. The target denoising algorithm is the denoising algorithm used by the power supply service smart device.

9. The power supply service voice system supporting multilingual mixed recognition and processing according to any one of claims 1 to 5, characterized in that, The multilingual and multi-dialect mixed speech recognition model, the natural language processing model, and the speech synthesis model are deployed using a private deployment scheme. The multilingual and multi-dialect mixed speech recognition model, the natural language processing model, and the speech synthesis model transmit data through the power grid.

10. The power supply service voice system supporting multilingual mixed recognition and processing according to any one of claims 1 to 5, characterized in that, N is 4, and the 4 processing links are used to process Mandarin, English, Cantonese and Portuguese respectively.

Images