Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition

Abstract

The invention discloses a Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition, and relates to the technical field of voice processing, the Tibetan dialect protection and real-time cross-dialect translation system comprises a voiceprint recognition module, a voice recognition module, a cross-dialect translation module, a voice synthesis module and a safety and audit module, the speech recognition module writes an input dialect speech stream into a standard Tibetan text according to a judged dialect category, the cross-dialect translation module receives the Tibetan text and drives a Tibetan large language model to complete semantic conversion and text generation, and the speech synthesis module converts the dialect text into target Tibetan dialect speech. According to the method, voiceprint recognition and dialect classification are combined, so that the function of personalized accurate cross-dialect translation is realized, the problem of difficult communication caused by large difference of Tibetan dialects is effectively solved, users of different dialects can perform smooth and natural real-time dialogues, and language barriers are broken through.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, specifically to a Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition. Background Technology

[0002] Currently, there are many dialects of Tibetan, such as Ü-Tsang, Kham, and Amdo. The differences in pronunciation, vocabulary, and grammar between these dialects are significant, making cross-dialect communication difficult.

[0003] Existing translation systems are mostly based on text or general speech recognition, lacking acoustic feature modeling for specific dialects and unable to personalize translations by combining user identity and dialect background, resulting in unnatural and inaccurate translation results. In addition, most existing systems rely on cloud services, posing risks to data security and privacy leaks.

[0004] Patent document CN109979432B discloses a dialect translation method and device. The above patent enables more convenient and accurate translation of target speech of different dialect types without the need for frequent manual switching of different dialect translation modes, thereby improving the user translation experience.

[0005] The aforementioned patent combines the speech features of the target speech with the representation vectors of various dialects as a basis for translation, using speech features and pre-set dialect representation vectors of various dialects to translate the target speech and obtain the translation result. However, it still has shortcomings in the accurate translation of Tibetan dialects.

[0006] To this end, this application proposes a Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition for accurate cross-dialect translation. Summary of the Invention

[0007] The purpose of this invention is to provide a Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition, so as to solve the technical problem of inaccurate cross-Tibetan dialect translation results mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition, comprising a voiceprint recognition module, a speech recognition module, a cross-dialect translation module, a speech synthesis module, and a security and auditing module. The voiceprint recognition module extracts voiceprint feature vectors from user speech input and automatically determines the Tibetan dialect category through acoustic structure analysis of the speech. The speech recognition module converts the input dialect speech stream into standard Tibetan text based on the dialect category determined by the voiceprint recognition module. The cross-dialect translation module receives the text output by the speech recognition module and drives a Tibetan language model to complete semantic conversion and text generation from the source dialect to the target dialect. The speech synthesis module converts the dialect text output by the cross-dialect translation module into the target Tibetan dialect speech using an acoustic model and a vocoder. The security and auditing module is logically parallel to the entire processing chain and establishes monitoring and intervention connections with the voiceprint recognition module, speech recognition module, cross-dialect translation module, and speech synthesis module, respectively.

[0009] Preferably, the voiceprint recognition module adopts a hierarchical processing architecture, consisting of a voiceprint feature extraction unit, a dialect classification unit, and an adaptive learning unit. The voiceprint feature extraction unit employs a hybrid feature extraction strategy based on Mel-frequency cepstral coefficients combined with first-order and second-order difference features, and integrates deep embedding vectors based on d-vectors. The dialect classification unit uses an end-to-end deep neural network hybrid classification framework based on Gaussian mixture model-general background model and connectionist temporal classification to analyze the dialect attributes of the input speech, determine the dialect category of the input speech, and output the classification confidence. The adaptive learning unit incrementally learns and adjusts the voiceprint feature model and the dialect classification model based on the user's subsequent voice interactions.

[0010] Preferably, the output of the voiceprint recognition module is connected to the input of the speech recognition module. After the voiceprint verification is successful, the system enables the dialect recognition and text transcription functions. The logical connection between the voiceprint recognition module and the speech recognition module is a cascade activation mechanism. The voiceprint recognition module not only identifies the dialect category information of the user, but also triggers the activation of the dialect transcription function and outputs it to the speech recognition module as a priori parameter. The dialect category information acts on the decoding process of initializing the speech recognition model, constituting the knowledge transfer logic of the voiceprint information to standard Tibetan text transcription process.

[0011] Preferably, the output of the speech recognition module is connected to the input of the cross-dialect translation module. The standard Tibetan text obtained from the transcription is input into the cross-dialect translation module. The cross-dialect translation module adopts a logical routing architecture that uses a dual-model collaboration of 70B and 7B. This routing logic follows a division of labor mode of "heavy model for stable quality and light model for speed". Open question answering and long context synthesis are routed to 70B for deep semantic analysis and long text generation. Structured rewriting and short instructions are routed to 7B to directly generate short bridging answers. The bridging answers are not directly output to the user, but are used as auxiliary information and fed back to the terminology standardization and retrieval enhancement unit along with the original query. Together, they constitute the retrieval query conditions, forming a closed-loop optimization logic from lightweight model decision-making to retrieval enhancement process.

[0012] Preferably, the cross-dialect translation module drives the Tibetan language big data model to perform deep semantic understanding and context construction. The cross-dialect translation module integrates a terminology standardization and retrieval enhancement unit. The Tibetan language big data model and the terminology standardization and retrieval enhancement unit form a bidirectional logical interaction, specifically a "request-response-injection" synchronous protocol. During processing, the Tibetan language big data model triggers terminology query conditions and initiates a terminology retrieval request to the terminology standardization and retrieval enhancement unit. After receiving the request, the terminology standardization and retrieval enhancement unit performs dictionary matching based on Aho-Corasick automata and semantic similarity retrieval based on FAISS vectors in parallel, and performs score normalization and weighted fusion on the retrieval results of the two. The standardized term with the highest score after fusion is injected back into the context window currently generated by the Tibetan language big data model in the form of a structured prompt template.

[0013] Preferably, the execution logic of the cross-dialect translation module is controlled by the Tibetan language big data model. The translation triggering conditions include two situations: First, when the Tibetan language big data model determines that the user request is a cross-dialect translation task, a translation instruction mark is attached to the intermediate representation generated by the Tibetan language big data model, and the cross-dialect translation module starts accordingly and converts the standard Tibetan text into the specified target output; Second, when the security and auditing module detects non-target language characters and non-standard expressions in the generated content, it triggers the cross-dialect translation module to re-translate the abnormal paragraphs, which serves as a logical guarantee for the compliance repair of the output content.

[0014] Preferably, the interaction logic between the security and audit module and other modules in the system is based on a unified audit data bus. The audit data bus collects metadata from voice input, voiceprint verification results, transcribed text, retrieval candidate sets, intermediate results generated by the model, final output content, and various decision metadata. The security and audit module performs correlation analysis based on global data, conducts security intervention and audit traceability. In the input stage, the security and audit module scans and desensitizes sensitive information on the original voice and transcribed text. In the inference stage, it performs real-time security classification and rejection judgment on the generated content. In the output stage, it intercepts the final content of the output text and records audit information, forming a full-link security control logic covering data input, internal processing and result output.

[0015] Preferably, all modules of the system are designed and integrated based on the logical premise of local private deployment. The Tibetan language big language model, the terminology standardization and retrieval enhancement unit rely on the terminology library and retrieval index, the voiceprint recognition module relies on the voiceprint library, and the security and auditing module relies on the policy rule library. All of these are stored in the local environment of the running device. Data interaction between modules is completed through internal inter-process communication and local area network calls, without relying on external Internet services.

[0016] Preferably, the data governance and model training of the system are completed entirely in a local isolated environment. The data governance follows a pipeline of "cleaning-deduplication-language recognition-alignment-segmentation-acceptance". The model training follows a three-stage paradigm: the first stage is continuous pre-training based on Tibetan monolingual and parallel corpora; the second stage is supervised fine-tuning based on instruction samples; and the third stage is alignment training based on safety bias fine-tuning and order preference optimization.

[0017] Preferably, the system supports end-to-end full-link streaming processing. The speech recognition module supports streaming automatic speech recognition, which can receive audio streams online and output partial transcription results in real time without waiting for the end of the whole sentence. The cross-dialect translation module incrementally pushes the generation results of the Tibetan language model to the client in the form of a token stream through the server event protocol. The speech synthesis module has streaming synthesis capability, which converts the received text stream into the corresponding target dialect audio stream in real time.

[0018] Compared with the prior art, the beneficial effects of the present invention are:

[0019] 1. This invention combines voiceprint recognition with dialect classification to achieve personalized and accurate cross-dialect translation, effectively solving the communication difficulties caused by the large differences in Tibetan dialects, enabling users of different dialects to have fluent and natural real-time conversations and breaking through language barriers.

[0020] 2. This invention enhances the professionalism of translation by introducing retrieval-enhanced generation and terminology standardization units. During the translation process, the system can dynamically inject standardized terms that have been reviewed by experts to ensure the accuracy of the translation and the consistency of terminology. The output results are highly authoritative and readable, avoiding misunderstandings caused by inconsistent translations.

[0021] 3. By employing lightweight and large-scale models, this invention achieves the function of providing high-quality service assurance, supports streaming speech recognition, text translation and speech synthesis, has low end-to-end latency, and provides a natural and smooth interactive experience, ensuring high availability and high stability of the service;

[0022] 4. This invention achieves data security protection through local private deployment and end-to-end security auditing, reduces the risk of privacy leaks from cloud servers, provides real-time security intervention through end-to-end security auditing, and has post-event auditing and traceability capabilities, ensuring the security of user voiceprints and voice data. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0024] Figure 2 This is a schematic diagram of the overall architecture of the present invention;

[0025] Figure 3 This is a schematic diagram of the data governance process of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0028] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0029] Example 1: Please refer to Figure 1 and Figure 2 A Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition includes a voiceprint recognition module, a speech recognition module, a cross-dialect translation module, a speech synthesis module, and a security and auditing module. The voiceprint recognition module extracts voiceprint feature vectors from user speech input and automatically determines the Tibetan dialect category through acoustic structure analysis of the speech. The voiceprint recognition module adopts a hierarchical processing architecture, comprising a voiceprint feature extraction unit, a dialect classification unit, and an adaptive learning unit. The voiceprint feature extraction unit employs a hybrid feature extraction strategy based on Mel-frequency cepstral coefficients combined with first-order and second-order difference features, and integrates deep embedding vectors based on d-vectors. The dialect classification unit uses a hybrid classification framework based on Gaussian mixture model-general background model and connectionist temporal classification, performing dialect attribute analysis on the input speech, determining its dialect category, and outputting classification confidence. The adaptive learning unit incrementally learns and adjusts the voiceprint feature model and dialect classification model based on subsequent user speech interactions.

[0030] Furthermore, the voiceprint recognition module of this invention adopts a hybrid strategy based on Mel frequency cepstral coefficients combined with first-order and second-order difference features, and integrates deep embedding vectors based on d-vectors to construct a high-dimensional acoustic feature representation. It extracts voiceprint feature vectors from user speech input. This hybrid feature extraction strategy captures dialect-specific acoustic properties, such as the glottal stop features of the Amdo dialect and the tone patterns of the Ü-Tsang dialect, by analyzing the spectral envelope, fundamental frequency trajectory, and formant distribution of the speech signal. The voiceprint feature extraction unit performs frame-by-frame windowing processing on the input speech, with each frame being 25 milliseconds long and a step size of 10 milliseconds, calculating 39-dimensional Mel spectral features. Simultaneously, it extracts 128-dimensional d-vector embedding vectors through a pre-trained deep neural network, forming a 167-dimensional hybrid feature vector. The dialect classification unit adopts a hybrid classification framework of Gaussian mixture model-general background model and connectionist temporal classification end-to-end deep neural network to perform multi-level analysis of the input speech. The Gaussian mixture model-general background model is trained on 512 high-dimensional features based on the expectation-maximization algorithm. The CTC end-to-end network uses a 5-layer bidirectional LSTM structure with 512 hidden units per layer. It optimizes frame-level classification accuracy using a connectionist temporal classification loss function. This hybrid framework performs both frame-level and sentence-level dialect attribute analysis on the input speech. Frame-level analysis identifies the frequency and duration of specific phonemes, while sentence-level analysis combines contextual semantic information to calculate the dialect category probability distribution, ultimately outputting the dialect category and classification confidence. With user authorization, the adaptive learning unit uses an online learning algorithm to incrementally update the voiceprint feature model and dialect classification model. When the user provides new speech samples, the adaptive learning unit fine-tunes the model parameters using a momentum stochastic gradient descent algorithm, with a learning rate of 0.001 and a momentum parameter of 0.9. A model snapshot is taken every 100 samples to prevent forgetting. Simultaneously, the adaptive learning unit maintains a user-personalized voiceprint feature library, using k-means clustering to perform hierarchical indexing of voiceprint features, supporting fast retrieval and matching.

[0031] The output of the voiceprint recognition module is connected to the input of the speech recognition module through a cascade activation mechanism. When the voiceprint verification is successful and the confidence level exceeds the 0.85 threshold, the system enables the dialect recognition and text transcription functions. After successful verification, the voiceprint recognition module not only outputs the dialect category information, but also triggers the activation of the dialect transcription function, passing the dialect category as a priori parameter to the speech recognition module to initialize the decoding process of the speech recognition model and construct the knowledge transfer logic from voiceprint information to standardized Tibetan text transcription.

[0032] Example 2: Please refer to Figure 1 and Figure 2A Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition includes a voiceprint recognition module, a speech recognition module, a cross-dialect translation module, a speech synthesis module, and a security and auditing module. The speech recognition module transcribes the input dialect speech stream into standard Tibetan text based on the dialect category determined by the voiceprint recognition module. The output of the voiceprint recognition module is connected to the input of the speech recognition module. After the voiceprint verification is passed, the system enables the dialect recognition and text transcription functions. The logical connection between the voiceprint recognition module and the speech recognition module is a cascade activation mechanism. The voiceprint recognition module not only identifies the dialect category information of the user, but also triggers the activation of the dialect transcription function and outputs it as a priori parameter to the speech recognition module. The dialect category information acts on the initialization of the decoding process of the speech recognition model, constituting the knowledge transfer logic of the voiceprint information to standard Tibetan text transcription process.

[0033] Furthermore, the speech recognition module accurately transcribes the input dialect speech stream into standardized Tibetan text based on the dialect category determined by the voiceprint recognition module. The speech recognition module employs an end-to-end speech recognition architecture based on deep neural networks, integrating three core components: an acoustic model, a language model, and a decoder. The acoustic model is based on the Transformer architecture, containing a 12-layer encoder and a 6-layer decoder, with 512 hidden units per layer and an 8-head attention mechanism. It is specifically optimized for the acoustic characteristics of the Tibetan dialect. The model input is 80-dimensional spectral features, which, after layer normalization and positional encoding, learn inter-frame dependencies through a multi-head self-attention mechanism, combined with feedforward neural networks. The network computes acoustic likelihood. The language model is based on a hybrid architecture of n-gram and neural network, trained on a Tibetan corpus of 65 million words, covering vocabulary and grammatical patterns from dialect areas such as Central Tibet, Kham, and Amdo. The decoder uses a beam search algorithm with a beam width of 10. It combines the frame-level probabilities output by the acoustic model and the word-level probabilities output by the language model, and calculates the score of the candidate path by weighted summation. The weight parameters are determined through cross-validation on the development set. For different dialect categories, the module dynamically loads the corresponding acoustic model parameters and language model dictionaries. For example, when the input is Kham dialect, it loads the acoustic model specific to Kham dialect and the language model containing Kham-specific vocabulary.

[0034] During the transcription process, the speech recognition module implements a multi-level quality control mechanism: First, speech activity detection is performed, and the start and end points of speech segments are determined based on dual thresholds of energy and zero-crossing rate. Then, noise robustness processing is performed, using spectral subtraction to reduce the impact of environmental noise. Finally, confidence calibration is performed, calculating the confidence score of the transcription result based on the output probability of the flexible maximum transfer function. Segments with a confidence score below 0.7 trigger a re-recognition process. The transcription output is Tibetan encoded text with a unified character standard, following the national standard "Tibetan Encoded Character Set," while maintaining the semantic integrity of the original speech and dialectal characteristics. The speech recognition module supports streaming processing, processing the input audio stream in real time through a sliding window mechanism with a window size of 2 seconds and a step size of 1 second. Combined with bidirectional LSTM streaming recognition technology, the end-to-end latency is controlled within 500 milliseconds while maintaining high accuracy. The transcription result is passed to the cross-dialect translation module in a structured data format, including text content, timestamp, confidence score, and dialect category label, providing complete contextual information for the subsequent translation process.

[0035] Example 3: Please refer to Figure 1 and Figure 2 A Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition includes a voiceprint recognition module, a speech recognition module, a cross-dialect translation module, a speech synthesis module, and a security and auditing module. The cross-dialect translation module receives text output by the speech recognition module and drives a Tibetan language model to complete semantic conversion and text generation from the source dialect to the target dialect. The output of the speech recognition module is connected to the input of the cross-dialect translation module, inputting the transcribed standard Tibetan text into the cross-dialect translation module. The cross-dialect translation module adopts a logical routing architecture of 70B and 7B dual-model collaboration. This routing logic follows the division of labor mode of "heavy model for quality stability and light model for speed improvement". Open question answering and long context synthesis are routed to 70B for deep semantic analysis and long text generation, while structured rewriting and short command are routed to 7B to directly generate short bridging answers. The bridging answers are not directly output to the user, but are used as auxiliary information and fed back to the terminology standardization and retrieval enhancement unit along with the original query to jointly constitute the retrieval query conditions, forming a closed-loop optimization logic from lightweight model decision-making to retrieval enhancement process.

[0036] The cross-dialect translation module drives the Tibetan language big data model to perform deep semantic understanding and context construction. The cross-dialect translation module integrates a terminology standardization and retrieval enhancement unit. There is a two-way logical interaction between the Tibetan language big data model and the terminology standardization and retrieval enhancement unit, which is a synchronous protocol of "request-response-injection". During the processing, the Tibetan language big data model triggers terminology query conditions and sends a terminology retrieval request to the terminology standardization and retrieval enhancement unit. After receiving the request, the terminology standardization and retrieval enhancement unit performs dictionary matching based on Aho-Corasick automata and semantic similarity retrieval based on FAISS vector in parallel, and performs score normalization and weighted fusion on the retrieval results of the two. The standardized term with the highest score after fusion is injected back into the context window currently generated by the Tibetan language big data model in the form of a structured prompt template.

[0037] The execution logic of the cross-dialect translation module is controlled by the Tibetan language big data model. The translation is triggered under two conditions: First, when the Tibetan language big data model determines that the user request is a cross-dialect translation task, a translation instruction mark is attached to the intermediate representation generated by the Tibetan language big data model. The cross-dialect translation module is then activated and the standard Tibetan text is converted into the specified target output. Second, when the security and auditing module detects non-target language characters and non-standard expressions in the generated content, the cross-dialect translation module is triggered to re-translate the abnormal paragraphs, which serves as a logical guarantee for the compliance repair of the output content.

[0038] Furthermore, the cross-dialect translation module receives standard Tibetan text output from the speech recognition module and drives the Tibetan large language model to complete semantic conversion and text generation from the source dialect to the target dialect. The cross-dialect translation module adopts a logical routing architecture that utilizes both the 70B and 7B models collaboratively, dynamically allocating tasks based on request type and complexity. The 70B model is based on the Transformer architecture, containing an 80-layer attention network with 128 attention heads per layer and 8192 hidden dimensions. It is trained on 65 million Tibetan-Chinese parallel sentence pairs and a large-scale Tibetan monolingual corpus, undertaking… Open question answering and long context synthesis tasks; the 7B model adopts a streamlined architecture, containing a 32-layer attention network with 32 attention heads per layer and 4096 hidden dimensions. It is specifically designed to handle structured rewriting and short instruction tasks. The routing logic is based on multi-feature decision-making: analyzing the length of the input text, syntactic complexity, entity density, and intent category. When the text length exceeds 128 characters, contains multiple clause structures, the entity density is higher than 0.15, or the intent is open-ended, it is routed to the 70B model; otherwise, it is routed to the 7B model to ensure overall user experience and resource utilization.

[0039] The terminology standardization and retrieval enhancement unit integrated within the cross-dialect translation module employs a hybrid retrieval strategy to achieve terminology consistency control. When the Tibetan language model triggers terminology query conditions during processing, such as detecting specialized terms, culturally specific concepts, and ambiguous words, it initiates a terminology retrieval request to the terminology standardization and retrieval enhancement unit. This unit then performs parallel dictionary-based exact matching based on the Aho-Corasick automaton and semantic similarity retrieval based on FAISS vectors. Dictionary matching uses a pre-compiled Tibetan-Chinese terminology automaton containing 120,000 expert-reviewed terminology pairs. The matching process employs the maximum forward matching algorithm, supporting overlapping matching and... For partial matching, vector retrieval uses a local FAISS index and employs the HNSW graph algorithm to construct a multi-layer index structure. The embedding model is based on Tibetan BERT, outputting a 768-dimensional semantic vector. Similarity calculation uses the L2 distance metric. The scores of the two retrieval results are normalized to the [0,1] interval using min-max. Then, a biased dictionary hit α=0.7 is used, supplemented with a semantic relevance weighted fusion β=0.3. The highest-scoring standard term is selected. The retrieved terms are injected back into the context window of the large language model in the form of a structured prompt template. The template format is a tag sequence of “[TERM: source term → standard term]”, which influences the subsequent generation process through an attention mechanism.

[0040] The cross-dialect translation module also implements a translation triggering mechanism. When the Tibetan language model determines that a user has requested a cross-dialect translation task, it attaches a translation instruction marker to the intermediate representation it generates. When the security and auditing module detects non-target language characters or non-standard expressions, it triggers a re-translation process. The entire process is optimized by caching. An LRU cache is established for frequently requested translation patterns with a cache size of 10,000 entries and a hit rate of up to 35%, which significantly improves real-time performance.

[0041] Example 4: Please refer to Figure 1 and Figure 2 A Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition includes a voiceprint recognition module, a speech recognition module, a cross-dialect translation module, a speech synthesis module, and a security and auditing module. The speech synthesis module converts the dialect text output by the cross-dialect translation module into the target Tibetan dialect speech by using an acoustic model and a vocoder.

[0042] The system supports end-to-end full-link streaming processing: the speech synthesis module has streaming synthesis capabilities, which convert the received text stream into the corresponding target dialect audio stream in real time;

[0043] Furthermore, the speech synthesis module employs a deep learning-based acoustic model and vocoder technology to convert the dialect text output by the cross-dialect translation module into the target Tibetan dialect speech. The acoustic model is based on the FastSpeech 2 architecture and includes three components: an encoder, a variance adapter, and a decoder. The encoder uses a 6-layer Transformer structure with 256 hidden units per layer and a 4-head attention mechanism to map the input text sequence into a phoneme-level hidden representation. The variance adapter integrates three predictors for duration, pitch, and energy, each implemented using an independent 1D convolutional network with a kernel size of 3, a stride of 1, and padding of 1. Each predictor outputs frame-level acoustic features. The decoder uses a 4-layer Transformer structure to upsample the feature sequence output by the variance adapter into a frame-level Mel-spectrum image. The upsampling rate is determined based on the predicted phoneme duration. The vocoder uses a HiFi-GAN architecture. The generator is based on a multi-receptive field fusion structure and includes multiple convolutional layers with different dilation factors to capture multi-scale features of the speech waveform. The discriminator uses a combination of multi-period discrimination and multi-scale discrimination architecture to ensure the naturalness and fluency of the generated speech.

[0044] For different target dialects, the speech synthesis module loads corresponding acoustic model parameters and speech databases. For example, when the synthesis target is Kham dialect, it uses an acoustic model trained on 2000 hours of Kham dialect speech and loads a prosodic rule library unique to Kham dialect, including tone curve templates, syllable duration patterns, and pause distribution rules. Multi-level quality control is implemented during the synthesis process: word segmentation, phonetic transcription, and prosodic prediction are performed in the text front-end processing stage, and prosodic boundary detection is performed based on the conditional random field model; the acoustic feature generation stage optimizes spectral details through adversarial training; and the waveform generation stage ensures sound quality fidelity through multi-resolution spectral loss.

[0045] The speech synthesis module supports streaming synthesis, employing an incremental generation strategy based on a sliding window with a window size of 256 frames and a step size of 128 frames. Combined with an overlap-addition algorithm, it ensures audio coherence while enabling real-time parameter adjustment. It supports dynamic adjustment of speech rate from 0.5x to 2.0x, pitch of ±200 cents, and volume from -20dB to +6dB to meet personalized output needs. The synthesized output is 16kHz, 16-bit mono PCM audio, and it provides both parametric encoding and compressed encoding output formats to adapt to different transmission bandwidth requirements. The entire synthesis process is deeply integrated with the security and auditing module, which performs content compliance checks on the output audio, including acoustic feature matching of sensitive words and sentiment analysis, to ensure that the output content meets security standards.

[0046] Example 5: Please refer to Figure 1 and Figure 2A Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition includes a voiceprint recognition module, a speech recognition module, a cross-dialect translation module, a speech synthesis module, and a security and auditing module. The security and auditing module is logically parallel to the entire processing chain and establishes monitoring and intervention connections with the voiceprint recognition module, speech recognition module, cross-dialect translation module, and speech synthesis module, respectively. The interaction logic between the security and auditing module and other modules in the system is based on a unified audit data bus. The audit data bus collects metadata from voice input, voiceprint verification results, transcribed text, retrieval candidate sets, model generation intermediate results, final output content, and various decision metadata. The security and auditing module performs correlation analysis based on global data, performs security intervention and audit traceability. In the input stage, the security and auditing module scans and desensitizes sensitive information on the original speech and transcribed text. In the inference stage, it performs real-time security classification and rejection judgment on the generated content. In the output stage, it intercepts the final content of the output text and records audit information, forming a full-chain security control logic covering data input, internal processing, and result output.

[0047] Furthermore, the security and auditing module operates in parallel with the entire processing chain, establishing comprehensive monitoring and intervention connections with the voiceprint recognition module, speech recognition module, cross-dialect translation module, and speech synthesis module. Based on a unified auditing data bus architecture, the security and auditing module uses a publish-subscribe model to collect runtime data from each module, including the original audio features of the voice input, the confidence score of voiceprint verification, the intermediate results of the transcribed text, the sorted list of the retrieval candidate set, the attention distribution generated by the model, and the structured content of the final output. The auditing data bus achieves high-throughput data transmission, with a peak throughput of 10,000 messages / second and an average latency of less than 50 milliseconds. It uses the Apache Arrow memory format to achieve zero-copy serialization.

[0048] During the input phase, the security and auditing module implements multimodal sensitive information detection. For the original speech, it employs acoustic event detection based on deep neural networks to identify acoustic patterns of specific keywords. For the transcribed text, it uses multi-pattern matching based on the Aho-Corasick automaton, covering sensitive word lists and synonym variants in Chinese and Tibetan, with a vocabulary exceeding 100,000 entries and a matching latency of less than 5 milliseconds. Detected sensitive information is desensitized using differential privacy technology. Controllable noise conforming to the human auditory masking effect is added to the speech signal. Rule-based replacement and generalization are implemented for the text content. During the inference phase, the security and auditing module integrates a real-time security classifier, trained on 100,000 labeled security samples based on the BERT architecture. It includes classification capabilities for eight sensitive dimensions, including politics, religion, and ethnicity. The classifier performs real-time analysis of the generated content. The system generates risk levels and rejection suggestions. When the risk score exceeds the 0.75 threshold, a rejection template is triggered and a legitimate alternative suggestion is generated. In the output stage, the security and audit module performs final content interception and audit archiving, performs syntactic analysis and semantic role labeling on the output text to identify potential risk expressions, and performs acoustic feature matching on the output audio to detect abnormal prosodic patterns. All audit data adopts a structured disk storage strategy, recording timestamps, session IDs, module call chains, decision basis, and handling results. A data hash chain is generated based on the SHA-256 algorithm to ensure the immutability of audit records. The security and audit module also implements an intelligent early warning mechanism, which detects abnormal access patterns based on time series analysis. When high-risk requests occur continuously, the protection level is automatically upgraded, and it supports integration with manual review workflows to form a complete security protection closed loop.

[0049] Example 6: Please refer to Figure 1 , Figure 2 and Figure 3 The system for Tibetan dialect protection and real-time cross-dialect translation based on voiceprint recognition is designed and integrated on the premise that all modules of the system are deployed locally. The Tibetan language model, the terminology library and retrieval index on which the terminology standardization and retrieval enhancement unit depend, the voiceprint library on which the voiceprint recognition module depends, and the policy rule library on which the security and auditing module depends are all stored in the local environment of the running device. Data interaction between modules is completed through internal inter-process communication and local area network calls, without relying on external Internet services.

[0050] The system's data governance and model training are completed entirely in a local isolated environment. Data governance follows a pipeline of "cleaning-deduplication-language recognition-alignment-segmentation-acceptance". Model training follows a three-stage paradigm: the first stage is continuous pre-training based on Tibetan monolingual and parallel corpora; the second stage is supervised fine-tuning based on instruction samples; and the third stage is alignment training based on safety bias fine-tuning and order preference optimization.

[0051] Furthermore, in the construction of the Tibetan language model, data governance and model training constitute the core foundation for system performance and security compliance. Data governance strictly follows a localized pipeline, encompassing six key stages: cleaning, deduplication, language recognition, alignment, segmentation, and acceptance. Data sources include independently collected parallel and accurate Tibetan-Chinese corpora, legally authorized publicly available Tibetan texts, and a professional terminology database reviewed by experts. In the cleaning stage, rule-based and statistical methods are used to remove HTML tags, garbled characters, and non-text elements, while simultaneously standardizing the Tibetan text to ensure consistent character encoding. The deduplication stage implements a three-layer deduplication strategy: approximate document deduplication based on SimHash, paragraph-level deduplication based on MinHash, and instance-level deduplication based on precise hashing, effectively eliminating the interference of duplicate data on model training. The language recognition stage employs the FastText classifier and an n-gram language model. The combined approach accurately identifies Tibetan-Chinese bilingual content and filters out third-language impurities. In the alignment stage, a combination strategy of heuristic rules and neural machine translation models is employed. Initial screening is performed based on length ratio, lexical alignment, and syntactic similarity. Then, semantic similarity is calculated through a cross-language encoder, retaining only high-quality alignment pairs with a confidence score higher than 0.9. In the segmentation stage, long texts are processed into sentence units and paragraphs, with intelligent segmentation based on Tibetan punctuation and semantic boundaries to ensure the semantic integrity of training samples. In the acceptance stage, a quality scoring model is introduced, evaluating from three dimensions: language fluency, semantic accuracy, and cultural appropriateness. Samples failing to reach the 0.85 score threshold are discarded. The entire governance process is completed in a physically isolated local environment, with raw data and intermediate products strictly adhering to the "no data leaving the domain" principle, ultimately forming training resources containing 65 million Tibetan-Chinese aligned sentence pairs, a large-scale Tibetan monolingual corpus, and 120,000 professional terms.

[0052] The model training adopts a three-stage progressive paradigm. The first stage involves continuous pre-training on cleaned Tibetan monolingual texts and high-quality aligned corpora. The language prior of the model is optimized based on the causal language modeling objective. The training uses the QLoRA parameter-efficient method, configured with bf16 calculation precision, 8-bit weight loading, LoRA rank 64, and a learning rate of 1e-4, achieving an effective batch size of 8192 contexts. With a size of 256, the model's ability to model Tibetan grammatical structure and expression habits is significantly improved. The second stage, instruction alignment, uses Tibetan-based question-and-answer and task instruction samples for supervised fine-tuning, emphasizing factual accuracy, structural regularity, and neutral style. The training process employs grouped attention and gradient checkpointing techniques, reducing GPU memory usage by 40% while maintaining performance. The third stage, safety bias fine-tuning, is based on a sequential preference optimization method. The model is trained on policy-labeled data to learn to reject answers with a unified template when sensitive topics are triggered, and to provide legitimate alternative suggestions. The third stage introduces a safety classifier distillation technique, transforming rule signals into learnable risk perception capabilities, ensuring the stability and auditability of output behavior. The entire training system is implemented in a distributed parallel framework, combining data parallelism, tensor parallelism, and pipeline parallelism, along with gradient accumulation and dynamic loss scaling, to achieve efficient convergence while ensuring training stability. Ultimately, this results in a powerful and secure Tibetan language model.

[0053] Working principle: The voiceprint recognition module first extracts voiceprint features from the input speech and automatically determines the dialect category. Its output serves as a priori parameters to trigger subsequent processing. The speech recognition module transcribes the speech stream into standard Tibetan text in real time according to the dialect category. The cross-dialect translation module adopts a dual-model collaborative architecture of 70B and 7B. Through logical routing, complex tasks are assigned to large-scale models for processing, while simple tasks are handled by lightweight models. Combined with terminology standardization and retrieval enhancement generation technology, standardized terms are dynamically injected during the generation process. The speech synthesis module finally converts the translation result into the target dialect speech output. The security audit module monitors in parallel throughout the entire process, implementing sensitive information detection, real-time security classification, and content interception at the input, inference, and output stages to ensure compliance and controllability across the entire chain. All modules run in a local private environment, completing data interaction through internal communication without relying on external network services, achieving secure and reliable real-time cross-dialect translation.

[0054] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition, characterized in that: The system includes a voiceprint recognition module, a speech recognition module, a cross-dialect translation module, a speech synthesis module, and a security and auditing module. The voiceprint recognition module extracts voiceprint feature vectors from the user's voice input and automatically determines the Tibetan dialect category through acoustic structure analysis of the speech. The speech recognition module converts the input dialect speech stream into standard Tibetan text based on the dialect category determined by the voiceprint recognition module. The cross-dialect translation module receives the text output by the speech recognition module and drives the Tibetan language model to complete the semantic conversion and text generation from the source dialect to the target dialect. The speech synthesis module converts the dialect text output by the cross-dialect translation module into the target Tibetan dialect speech using an acoustic model and a vocoder. The security and auditing module is logically parallel to the entire processing chain and establishes monitoring and intervention connections with the voiceprint recognition module, speech recognition module, cross-dialect translation module, and speech synthesis module, respectively.

2. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: The voiceprint recognition module adopts a hierarchical processing architecture, consisting of a voiceprint feature extraction unit, a dialect classification unit, and an adaptive learning unit. The voiceprint feature extraction unit uses a hybrid feature extraction strategy based on Mel frequency cepstral coefficients combined with first-order and second-order difference features, and integrates deep embedding vectors based on d-vectors. The dialect classification unit uses an end-to-end deep neural network hybrid classification framework based on Gaussian mixture model-general background model and connectionist temporal classification to perform dialect attribute analysis on the input speech, determine the dialect category of the input speech, and output the classification confidence score. The adaptive learning unit incrementally learns and adjusts the voiceprint feature model and dialect classification model based on the user's subsequent voice interactions.

3. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: The output of the voiceprint recognition module is connected to the input of the speech recognition module. After the voiceprint verification is successful, the system enables the dialect recognition and text transcription functions. The logical connection between the voiceprint recognition module and the speech recognition module is a cascade activation mechanism. The voiceprint recognition module not only identifies the dialect category information of the user, but also triggers the activation of the dialect transcription function and outputs it to the speech recognition module as a priori parameter. The dialect category information acts on the initialization of the decoding process of the speech recognition model, constituting the knowledge transfer logic of the voiceprint information to standard Tibetan text transcription process.

4. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: The output of the speech recognition module is connected to the input of the cross-dialect translation module. The standard Tibetan text obtained from the transcription is input into the cross-dialect translation module. The cross-dialect translation module adopts a logical routing architecture that uses a dual-model collaboration of 70B and 7B. This routing logic follows a division of labor mode of "heavy model for stable quality and light model for speed". Open question answering and long context synthesis are routed to 70B for deep semantic analysis and long text generation. Structured rewriting and short instructions are routed to 7B to directly generate short bridging answers. The bridging answers are not directly output to the user, but are used as auxiliary information and fed back to the terminology standardization and retrieval enhancement unit along with the original query. Together, they constitute the retrieval query conditions, forming a closed-loop optimization logic from lightweight model decision-making to retrieval enhancement process.

5. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: The cross-dialect translation module drives the Tibetan language big data model to perform deep semantic understanding and context construction. The cross-dialect translation module integrates a terminology standardization and retrieval enhancement unit. There is a two-way logical interaction between the Tibetan language big data model and the terminology standardization and retrieval enhancement unit, which is a synchronous protocol of "request-response-injection". During the processing, the Tibetan language big data model triggers terminology query conditions and sends a terminology retrieval request to the terminology standardization and retrieval enhancement unit. After receiving the request, the terminology standardization and retrieval enhancement unit performs dictionary matching based on Aho-Corasick automata and semantic similarity retrieval based on FAISS vectors in parallel, and performs score normalization and weighted fusion on the retrieval results of the two. The standardized term with the highest score after fusion is injected back into the context window currently generated by the Tibetan language big data model in the form of a structured prompt template.

6. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: The execution logic of the cross-dialect translation module is controlled by the Tibetan language big model. The translation triggering conditions include two situations: First, when the Tibetan language big model determines that the user request is a cross-dialect translation task, a translation instruction mark is attached to the intermediate representation generated by the Tibetan language big model. The cross-dialect translation module is then started and the standard Tibetan text is converted into the specified target output. Secondly, when the security and auditing module detects non-target language characters and non-standard expressions in the generated content, it triggers the cross-dialect translation module to re-translate the abnormal paragraphs, thus serving as a logical safeguard for the compliance repair of the output content.

7. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: The security and audit module interacts with other modules in the system based on a unified audit data bus. The audit data bus collects data from voice input, voiceprint verification results, transcribed text, retrieval candidate sets, intermediate results generated by the model, final output content, and various decision metadata. The security and audit module performs correlation analysis based on global data to conduct security intervention and audit traceability. During the input stage, the security and audit module scans and desensitizes sensitive information on the original voice and transcribed text. During the inference stage, it performs real-time security classification and rejection judgment on the generated content. During the output stage, it intercepts the final content of the output text and records audit information, forming a full-link security control logic covering data input, internal processing, and result output.

8. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: All modules of the system are designed and integrated based on the premise of local private deployment. The Tibetan language big data model, the terminology standardization and retrieval enhancement unit rely on the terminology library and retrieval index, the voiceprint recognition module relies on the voiceprint library, and the security and auditing module relies on the policy rule library. All of these are stored in the local environment of the running device. Data interaction between modules is completed through internal inter-process communication and local area network calls, without relying on external Internet services.

9. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: The system's data governance and model training are completed entirely in a local isolated environment. Data governance follows a pipeline of "cleaning-deduplication-language recognition-alignment-segmentation-acceptance" steps. Model training follows a three-stage paradigm: the first stage is continuous pre-training based on Tibetan monolingual and parallel corpora. The second stage is supervised fine-tuning based on instruction samples; The third stage is alignment training based on safety bias fine-tuning and order preference optimization.

10. The Tibetan dialect protection and real-time cross-dialect translation system based on voiceprint recognition according to claim 1, characterized in that: The system supports end-to-end full-link streaming processing. The speech recognition module supports streaming automatic speech recognition, which can receive audio streams online and output partial transcription results in real time without waiting for the end of the whole sentence. The cross-dialect translation module incrementally pushes the generated results of the Tibetan language model to the client in the form of a token stream through a server-side event protocol; The speech synthesis module has streaming synthesis capabilities, which convert the received text stream into the corresponding target dialect audio stream in real time.

Images