Information processing method and system for voice conversion

Abstract

The invention relates to the technical field of voice signal processing and synthesis, and particularly discloses an information processing method and system for voice conversion, and the method comprises the steps: extracting a language feature vector of each word from an input text; calculating posterior probabilities of the words in different languages by combining a Bayesian reasoning mechanism, and generating language attribution confidence coefficient characteristic values; a Monte Carlo sampling method is adopted to carry out multiple times of context sensitive simulation, and pronunciation path selection probability distribution characteristic values are generated; further fusing the feature values into a multi-language pronunciation decision vector, inputting the multi-language pronunciation decision vector into a multi-language end-to-end speech synthesis model, calling a phoneme mapping rule of a corresponding language and an acoustic parameter prediction module, and generating a high-quality target speech spectrogram; and finally, dynamically adjusting Bayesian prior distribution and a language recognition threshold according to the output speech spectrogram.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing and synthesis, in particular to a speech conversion information processing method and system. BACKGROUND

[0002] With the acceleration of globalization and the rapid development of artificial intelligence technology, the demand for cross-language communication is increasing, and speech synthesis technology, as an important part of human-computer interaction, plays an increasingly important role in intelligent assistants, online education, international conference translation and other fields. Traditional speech synthesis systems are mainly designed for a single language and can achieve high-quality speech output. However, in real-world application scenarios, user input often contains mixed text of two or more languages, such as Chinese-English mixed conversations, multi-lingual social media content, etc. This multi-language mixing phenomenon puts higher requirements on the language recognition ability, pronunciation path selection mechanism and speech generation quality of existing speech synthesis systems. Therefore, building a speech conversion information processing system that can accurately recognize multi-language context, reasonably handle pronunciation ambiguity, and adaptively optimize model performance has become an important research direction in the current speech synthesis field.

[0003] The prior art has the following disadvantages:

[0004] In the speech synthesis scene of multi-language mixed text, such as international conferences, cross-language dialogue systems or global intelligent assistant applications, the system needs to output coherent and natural speech for a text containing Chinese, English and even small languages (such as French, Japanese). However, the existing technology faces two core challenges: language switching point recognition difficulty and pronunciation ambiguity caused by vocabulary table conflict. First, without explicit language labeling, the system has difficulty accurately determining the position of language switching in the text, resulting in confusion in pronunciation rules and intonation patterns for the speech synthesis model, such as pronouncing English words according to Chinese pinyin rules; second, there are a large number of "homonyms" (such as "subject" in English, which means "subject", while in the Latin language system it may correspond to completely different words) in different languages, which makes the model easily choose the wrong pronunciation path when lacking context understanding ability, affecting the accuracy and intelligibility of speech output. Therefore, how to achieve efficient and robust multi-language mixing processing through patent-level language recognition and context perception mechanism has become one of the key technical bottlenecks that need to be broken through in the current high-end speech synthesis system. SUMMARY

[0005] The purpose of the present application is to provide a speech conversion information processing method and system to solve the problems in the above background.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] An information processing method for speech conversion, comprising the following steps:

[0008] S1: receiving a mixed text input containing two or more languages, performing word segmentation, semantic unit segmentation and context window construction on the text, extracting language feature vectors of each word, including spelling structure, syllable distribution, letter frequency and syntax dependency;

[0009] S2: according to the pre-trained language prior probability model and conditional probability model, the posterior probability of each word under different languages is calculated, and the most possible language category of the current semantic segment is dynamically evaluated in combination with the sliding window mechanism to generate language attribution confidence characteristic value;

[0010] S3: for homonyms with multiple language interpretation possibilities, a Monte Carlo sampling method is used for multiple context-sensitive simulations, the frequency of each language label is counted, and pronunciation path selection probability distribution characteristic value is generated;

[0011] S4: the language attribution confidence characteristic value and the pronunciation path selection probability distribution characteristic value are fused into a multi-language pronunciation decision vector, and input into a multi-language end-to-end speech synthesis model, the phoneme mapping rule and acoustic parameter prediction module corresponding to the language are called to generate the target speech spectrum graph;

[0012] S5: according to the target speech spectrum graph, the Bayesian prior distribution and the language recognition threshold are dynamically adjusted to realize online fine-tuning and continuous optimization of the model, and the generalization ability and robustness of the system in a multi-language mixed environment are improved.

[0013] As a further scheme of the application: according to the pre-trained language prior probability model and conditional probability model, the posterior probability of each word under different languages is calculated, specifically including:

[0014] A language prior probability model is established, which records the basic appearance frequency of multiple candidate languages in the entire corpus; by statistically analyzing the global word frequency distribution of each language in a large-scale multi-language text data set and performing normalization processing, the basic use preference probability of each language is obtained;

[0015] A language conditional probability model is constructed to represent the possibility of the appearance of a certain word in a specific language; the model is generated based on the local lexical statistics of each language, and a smoothing mechanism is introduced to avoid the problem that the probability is zero due to the absence of some rare words in the training set;

[0016] The Bayesian inference method is used to comprehensively judge the possibility of each word in different languages, that is, the relative confidence of the word belonging to each language is calculated by combining the basic use frequency of the language and the possibility of the word appearing in the language; and a minimum confidence threshold is set, and when the confidence corresponding to a language is lower than the threshold, the language is excluded as the language to which the word belongs.

[0017] As a further scheme of the application, the combination sliding window mechanism dynamically evaluates the most possible language category to which the current semantic segment belongs, and comprises the following steps:

[0018] A sliding window with a fixed length and a proper sliding step are set, and continuous semantic segments in the input text are sequentially intercepted;

[0019] For the word set in each intercepted semantic segment, the average language confidence score of each word in each candidate language is calculated, as the overall matching degree of the semantic segment in different languages;

[0020] A context consistency weight factor is introduced, which is used to measure the continuity of the language category between the current semantic segment and the previous segment, and the language scores of the previous and next segments are fused by weighting, so that the system can better maintain the continuity of language use when judging the current language;

[0021] Finally, according to the weighted comprehensive language score, the language with the highest score is selected as the predicted language label of the current semantic segment.

[0022] As a further scheme of the application, the specific process of generating the pronunciation path selection probability distribution feature value is:

[0023] Based on the language score distribution of the current semantic segment in all languages, the uncertainty degree of language classification is analyzed, which is specifically whether the distribution is concentrated in a language or scattered among multiple languages;

[0024] The score gap between the dominant language and other languages is calculated to reflect the clarity of language recognition in the current semantic segment, and if the score of the dominant language is much higher than that of other languages, it means that the language recognition is clearer;

[0025] A fuzzy clustering method is used to recognize the language score of multiple continuous semantic segments, analyze whether there is a frequent switching trend in language use, identify the potential language boundary area, and adjust the confidence correction coefficient accordingly;

[0026] Based on the above information, a language attribution confidence feature value is generated, which reflects the reliability of language recognition in the current semantic segment, and the higher the value, the more accurate and stable the language recognition.

[0027] As a further scheme of the present application: the multiple context-sensitive simulation using the Monte Carlo sampling method specifically comprises:

[0028] A candidate language tag set is constructed, homophonic synonyms that can have multiple language interpretations in the current text are identified, and pronunciation rules and semantic roles in different languages are extracted;

[0029] A sampling frequency threshold is set, and a context constraint factor is introduced for each sampling process, which is dynamically generated based on the consistency degree of the language categories of the adjacent words before and after the current word, and is used to limit unreasonable language tag combinations in the sampling process;

[0030] In each sampling, a possible language tag path is selected according to the context semantic coherence model, and the frequency of each language tag in the path is recorded;

[0031] After all the sampling is completed, the proportion of each language tag in all sampling paths is counted to form a preliminary pronunciation path selection probability distribution characteristic value.

[0032] As a further scheme of the present application: the generation of the pronunciation path selection probability distribution characteristic value comprises the following steps:

[0033] A syntax structure matching degree evaluation module is established to analyze whether the current sampling path conforms to the typical syntax pattern of the target language, such as the subject-predicate-object sequence and the usage habit of prepositions;

[0034] A syntax adaptation weight λ∈[0, 1] is introduced, and the probability score of the current sampling path is weighted and corrected according to the syntax rationality of the path;

[0035] The language use preference database is combined to extract the language use tendency data of the user or the text source, and a language preference adjustment coefficient is set to further adjust the path score;

[0036] The syntax corrected path score and the user language preference influence are fused, and the final probability distribution of each language tag is recalculated and normalized to serve as the optimized pronunciation path selection probability distribution characteristic value.

[0037] As a further scheme of the present application: the process of generating the pronunciation path selection probability distribution characteristic value further comprises the following steps:

[0038] The multiple sampling results are subjected to cluster analysis to identify high-frequency language path combinations, which are defined as dominant pronunciation path clusters;

[0039] By measuring the distance between the dominant path and other paths, the distance is normalized to obtain a path stability index ρ∈[0, 1], which is used to evaluate the stability of the current vocabulary pronunciation selection;

[0040] A language switching cost function is set, if frequent language switching occurs in a certain sampling path, the overall path score is correspondingly reduced;

[0041] The path stability index and the language switching cost are integrated to generate a final pronunciation path selection probability distribution characteristic value, and output to a subsequent pronunciation decision module for guiding language label selection in a multi-language mixed speech synthesis process.

[0042] As a further scheme of the application: the language attribution confidence feature value and the pronunciation path selection probability distribution feature value are fused into a multi-language pronunciation decision vector, comprising the following steps:

[0043] A multi-language pronunciation decision vector space is constructed, wherein each dimension corresponds to a candidate language, and the initial weight of each dimension is initialized;

[0044] According to the language attribution confidence feature value, the basic weight of each language dimension is dynamically adjusted, so that the language with high confidence occupies a higher priority in subsequent decision-making;

[0045] Combined with the pronunciation path selection probability distribution feature value, a weighted fusion method is used to calculate the final multi-language pronunciation decision vector, which is used to represent the language most likely to be used for the current semantic unit and the corresponding pronunciation tendency strength;

[0046] A language activation threshold is set, if the decision vector component of a certain language dimension exceeds the threshold, it is output to the speech synthesis model as the dominant language label, otherwise the multi-language mixed pronunciation mode is triggered.

[0047] As a further scheme of the application: the corresponding language phoneme mapping rule and acoustic parameter prediction module are called to generate the target speech spectrum, specifically comprising:

[0048] A multi-language phoneme mapping rule library is established, the rule library stores the conversion rules of orthography to phoneme sequence of multiple candidate languages, and is indexed and managed according to language categories; according to the dominant language label or the mixed language combination, the corresponding phoneme mapping module is called from the rule library to generate the phoneme sequence representation of the current text unit;

[0049] A language adaptive acoustic parameter prediction module is constructed, which jointly predicts the key acoustic parameters of fundamental frequency trajectory, energy distribution, phoneme duration and pause position based on the prosodic features, context information and pronunciation path selection probability distribution feature value of the dominant language;

[0050] inputting the phoneme sequence and the predicted acoustic parameters into a multi-language compatible end-to-end speech synthesis model, which internally comprises a language-aware attention mechanism and a multi-language shared encoder for fusing language features and acoustic modeling information;

[0051] outputting a mel-spectrogram of the target speech from the speech synthesis model, wherein each time frame corresponds to a group of frequency band energy distribution vectors; and then restoring the mel-spectrogram into a time-domain speech waveform through a neural vocoder model, the mel-spectrogram of the target speech being a target speech spectrogram.

[0052] An information processing system for speech conversion, comprising:

[0053] a multi-language mixed text context modeling module, configured to receive a mixed text input containing two or more languages, perform word segmentation, semantic unit segmentation and context window construction on the text, and extract a language feature vector of each word, including spelling structure, syllable distribution, letter frequency and syntactic dependency;

[0054] a Bayesian posterior inference language recognition feature generation module, configured to calculate a posterior probability of each word under different languages according to a pre-trained language prior probability model and a conditional probability model, and dynamically evaluate a most likely language category to which a current semantic segment belongs in combination with a sliding window mechanism to generate a language attribution confidence feature value;

[0055] a Monte Carlo simulation pronunciation ambiguity resolution feature generation module, configured to, for a homograph with multiple language interpretation possibilities, perform multiple context-sensitive simulations by using a Monte Carlo sampling method, count the frequency of occurrence of each language label, and generate a pronunciation path selection probability distribution feature value;

[0056] a multi-language fusion speech synthesis decision module, configured to fuse the language attribution confidence feature value and the pronunciation path selection probability distribution feature value into a multi-language pronunciation decision vector, and input the multi-language pronunciation decision vector into a multi-language end-to-end speech synthesis model to call a phoneme mapping rule and an acoustic parameter prediction module corresponding to a language, and generate a target speech spectrogram;

[0057] a feedback-driven online adaptive optimization module, configured to dynamically adjust a Bayesian prior distribution and a language recognition threshold according to the target speech spectrogram, realize online fine-tuning and continuous optimization of the model, and improve the generalization ability and robustness of the system in a multi-language mixed environment.

[0058] The present application has the following advantages:

[0059] (1) The application introduces a systematic multilingual text modeling and intelligent recognition mechanism, significantly improving the accuracy and naturalness of speech conversion in complex language environment. Specifically, the system uses a multilingual compatible word segmentation engine to perform morpheme level segmentation on the input text, and combines a sliding window mechanism to realize dynamic division of semantic units, thereby constructing a language analysis framework with context awareness. On this basis, the system extracts language feature vectors of words from multiple dimensions such as spelling structure, syllable distribution, letter frequency and syntactic dependency, and eliminates the representation bias caused by writing system differences through a language feature normalization module, further improving the model generalization ability. Subsequently, the system combines the pre-trained language prior probability model and conditional probability model, uses Bayesian inference method to calculate the posterior probability of each word in different languages, and introduces context consistency weight factor to dynamically evaluate the language category of semantic fragments, significantly enhancing the coherence and stability of language recognition. In view of the homonym problem existing in multilingual mixed text, the system innovatively introduces the Monte Carlo sampling mechanism to simulate multiple paths under the context constraint, combines the syntactic matching degree evaluation, user language preference adjustment, path stability analysis and language switching cost function, generates the optimized pronunciation path selection probability distribution characteristic value, thereby effectively eliminating pronunciation ambiguity and ensuring the accuracy and fluency of the final speech output. This technical system is particularly suitable for complex interactive scenarios such as international conference translation, cross-language education and multilingual customer service, can provide users with highly natural, accurate and personalized speech conversion experience, greatly improving the practicality and user experience value of the system.

[0060] (2) The present invention innovatively introduces a feedback-driven online adaptive optimization module, constructs a closed-loop speech processing mechanism with continuous learning capabilities, and significantly improves the robustness and personalized service capabilities of the system in a dynamic multilingual environment. The module analyzes the generated target speech spectrogram in real time, compares it with the expected acoustic features, extracts error distribution information, and dynamically adjusts the Bayesian prior distribution parameters accordingly, so that the model can quickly adapt to new language usage trends and acoustic performance characteristics. At the same time, the system adaptively corrects the language recognition threshold based on the actual recognition results, and intelligently adjusts the judgment boundary according to the error recognition rate, frequency of occurrence and context consistency of different languages, thereby improving the stability and accuracy of language classification. In addition, the system also integrates a performance monitoring mechanism, which uses multi-dimensional indicators such as naturalness score, intelligibility assessment, and semantic consistency to quantitatively evaluate the speech output quality, and automatically triggers the optimization process when performance fluctuations or abnormal user feedback are detected. On this basis, the system further supports personalized model fine-tuning function, combines user historical usage data and preference model, dynamically adjusts language weight distribution strategy and pronunciation path sampling tendency, and realizes customized speech synthesis service for individual users. This self-evolving technical architecture not only enhances the system's adaptability and generalization capabilities in complex scenarios, but also greatly improves the consistency and immersion of the interactive experience. It provides solid support for the long-term deployment and continuous optimization of multilingual speech conversion technology in advanced application areas such as intelligent customer service, cross-language education, and international conference translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The present invention will be further described below with reference to the accompanying drawings.

[0062] Figure 1 It is a flowchart of an information processing method for voice conversion of the present invention;

[0063] Figure 2 This is a flow chart of an information processing system for voice conversion in the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0065] See also Figure 1 As shown, the present invention is an information processing method for voice conversion, comprising the following steps:

[0066] S1: receiving mixed text input containing two or more languages, performing word segmentation, semantic unit segmentation and context window construction on the text, extracting language feature vectors of each word, including spelling structure, syllable distribution, letter frequency and syntactic dependency;

[0067] S2: calculating the posterior probability of each word under different languages according to the pre-trained language prior probability model and conditional probability model, and dynamically evaluating the most likely language category of the current semantic segment combined with the sliding window mechanism to generate language attribution confidence feature values;

[0068] S3: for homonyms with multiple language interpretation possibilities, using Monte Carlo sampling method for multiple context-sensitive simulations, and generating pronunciation path selection probability distribution feature values by counting the frequency of each language label;

[0069] S4: fusing the language attribution confidence feature values and the pronunciation path selection probability distribution feature values into a multi-language pronunciation decision vector, and inputting it into a multi-language end-to-end speech synthesis model to call the phoneme mapping rules and acoustic parameter prediction module of the corresponding language to generate the target speech spectrogram;

[0070] S5: dynamically adjusting the Bayesian prior distribution and the language recognition threshold according to the target speech spectrogram to realize online fine-tuning and continuous optimization of the model, and improve the generalization ability and robustness of the system in a multi-language mixed environment.

[0071] In S1, the mixed text input containing two or more languages is received, the text is segmented, the semantic unit is segmented, and the context window is constructed, and the language feature vector of each word is extracted, including spelling structure, syllable distribution, letter frequency and syntactic dependency, specifically including:

[0072] In the speech conversion information processing method of the application, first, the mixed text input containing two or more languages is received, for example, the user input text mixed with Chinese, English and French. The system first preprocesses the text, including word segmentation, semantic unit segmentation and context window construction. Specifically, a multi-language compatible word segmentation engine is used to perform morpheme-level segmentation on the input text. The engine has a built-in vocabulary and morphological rule library for multiple languages, which can automatically identify cross-language boundaries and avoid ambiguity segmentation problems caused by language confusion, thereby ensuring the accuracy of subsequent language feature extraction.

[0073] Subsequently, the system performs semantic unit division on the segmented word sequence based on the sliding window mechanism. Each semantic unit is composed of a plurality of consecutive words, and a context window length is set to capture local language usage patterns. Through this mechanism, the language consistency relationship between the current word and its context can be effectively modeled, providing more rich context clues for subsequent language attribution judgment.

[0074] After completing the semantic unit division, the system enters the language feature extraction phase. For each word, its language feature vector is extracted, including but not limited to: spelling structure features (such as the proportion of vowels and consonants), syllable distribution features (such as average syllable number, stress position), letter frequency features (such as high-frequency character occurrence rate), and syntactic dependency features (such as the role type of the word in the dependency syntax tree). These features describe the language properties of the word from multiple dimensions, helping to improve the discriminability of the language recognition model.

[0075] To further eliminate the representation bias caused by differences in writing systems of different languages, the system constructs a language feature normalization module. This module maps the original feature values from different language systems to a unified dimensional space, ensuring that the feature representations of all languages are comparable and consistent, thereby improving the generalization performance and robustness of the subsequent language recognition model.

[0076] On this basis, the system also establishes a language feature weight distribution mechanism. This mechanism dynamically adjusts the weight coefficients of spelling structure, syllable distribution, and other features according to the discriminability of each language in the training corpus, so that those features with higher discriminability in a multilingual environment have higher priority in the subsequent language recognition process. For example, some languages may have significant differences in letter frequency, while others may be more unique in syllable structure, and the system will automatically adjust the influence of each feature accordingly.

[0077] To enhance the accuracy of language recognition, the system introduces a cross-language similarity matrix to quantify the degree of feature similarity between different languages. When the language features of a word are highly matched with multiple languages, the system analyzes the similarities and differences between the candidate languages in combination with the matrix, thereby narrowing down the language selection range and determining the most likely candidate language set.

[0078] In addition, the system also constructs a context consistency enhancement module. This module analyzes the correlation between the language feature vectors of the current word and its adjacent words, and if the language features of adjacent words are highly consistent, it increases the credibility of the language corresponding to the current word. This mechanism not only improves the stability of language recognition, but also effectively alleviates the misjudgment problem caused by individual word ambiguity or noise interference.

[0079] Finally, the system outputs the standardized language feature vector sequence for subsequent language recognition and pronunciation path prediction modules. At the same time, the system also records the language preference trend after context consistency enhancement as an auxiliary decision basis, providing more accurate language label prediction results for the multilingual mixed speech synthesis process.

[0080] In S2, the posterior probability of each word in different languages is calculated according to the pre-trained language prior probability model and conditional probability model, and the most likely language category of the current semantic segment is dynamically evaluated in combination with the sliding window mechanism to generate a language attribution confidence feature value, which specifically includes:

[0081] After completing the extraction of the language feature vector of the input multi-language mixed text, the system enters the language attribution recognition phase, which realizes intelligent judgment of the language to which the words belong through the construction of a statistical model and a context analysis mechanism, and generates a language attribution confidence feature value, providing a key decision basis for subsequent pronunciation path selection.

[0082] Firstly, the system calls the pre-trained language prior probability model and conditional probability model to calculate the posterior probability of each word in different candidate languages. Among them, the language prior probability model records the basic appearance frequency of multiple candidate languages in the entire corpus. Based on a large-scale multi-language text data set, the global word frequency distribution of each language is counted and normalized to obtain the basic usage preference probability of each language. This probability reflects the language usage tendency of different languages in general context, providing basic weight support for subsequent language recognition.

[0083] At the same time, the system constructs a language conditional probability model to represent the possibility of the appearance of a certain word in a specific language. This model is generated based on the local vocabulary statistics of each language, that is, for each candidate language, the appearance frequency of each word in this language is counted, and the Laplace smoothing mechanism is combined to compensate for the probability of rare words or out-of-vocabulary words, avoiding the problem that the probability of some words is zero because they do not appear in the training set. Thus, the system can more accurately evaluate the matching degree between words and languages.

[0084] On this basis, the system uses Bayesian inference method to comprehensively judge the possibility of each word in different languages. Specifically, the system combines the language prior probability and the language conditional probability to calculate the relative credibility of the word belonging to each language, i.e. the posterior probability. This process fully integrates the overall usage frequency of the language and its actual performance in the local context, making the language recognition more robust and adaptive. In addition, the system also sets a minimum credibility threshold, when the posterior probability corresponding to a language is lower than the threshold, the language is excluded as the language to which the current word belongs, thus effectively filtering low-confidence language options and improving recognition accuracy.

[0085] To further enhance the coherence and stability of language recognition, the system introduces a sliding window mechanism for dynamic language category evaluation of continuous semantic segments. Specifically, the system sets a fixed-length sliding window and an appropriate sliding step to sequentially intercept continuous semantic segments in the input text. For each set of words in the intercepted semantic segment, the system calculates their average language confidence scores under each candidate language as the overall matching degree of the semantic segment under different languages.

[0086] To further improve the contextual consistency of language recognition, the system introduces a context consistency weight factor. This factor is used to measure the coherence of language categories between the current semantic segment and its previous segment. The system better maintains the continuity of language use by weighting and fusing the language scores of the previous and next two segments when judging the current language, avoiding language misjudgment problems caused by ambiguity or noise of individual words. Finally, the system selects the language with the highest score as the predicted language label of the current semantic segment according to the weighted comprehensive language score.

[0087] Based on the completion of language classification, the system further generates a language attribution confidence feature value to quantify the reliability of language recognition of the current semantic segment. The system analyzes the uncertainty degree of language classification based on the language score distribution of the current semantic segment under all languages. If the scores are highly concentrated in a certain language, it indicates that the language recognition is more explicit; otherwise, if the scores are scattered among multiple languages, it indicates that there is high uncertainty.

[0088] In addition, the system also calculates the score gap between the dominant language and other languages to reflect the explicitness of language recognition in the current semantic segment. If the score of the dominant language is much higher than that of other languages, it indicates that the current language recognition is clear and reliable; otherwise, there may be language confusion.

[0089] To further enhance the stability of recognition, the system uses fuzzy clustering method to recognize the language scores of multiple continuous semantic segments, analyzes whether there is a frequent switching trend in language use, and identifies potential language boundary areas. The system adjusts the confidence correction coefficient accordingly to adapt to complex scenarios with frequent language switching.

[0090] Finally, the system generates a language attribution confidence feature value based on the above information, which reflects the reliability of language recognition of the current semantic segment. The higher the value, the more accurate and stable the language recognition. This feature value will be an important input parameter for the subsequent pronunciation path selection and speech synthesis module, supporting the natural and smooth generation of multi-language mixed speech.

[0091] In S3, for the same homonym with multiple language interpretation possibilities, a Monte Carlo sampling method is used for multiple context-sensitive simulations, the frequency of each language label is counted, and the pronunciation path selection probability distribution characteristic value is generated, specifically including:

[0092] After completing the language attribution recognition and confidence evaluation of the input text, the system enters the pronunciation path selection phase. Due to the existence of a large number of homonyms (i.e. words with the same spelling but belonging to different languages and having different pronunciation rules) in multi-language mixed text, the traditional single language mapping mechanism is difficult to accurately determine the true pronunciation intention. Therefore, the present application introduces a context-sensitive Monte Carlo sampling method to simulate multiple times of candidate language label selection, and combines multiple language features for comprehensive evaluation to generate pronunciation path selection probability distribution characteristic values, providing reliable language decision basis for subsequent speech synthesis.

[0093] Firstly, the system constructs a candidate language label set to record all language categories that may be involved in the current text. On this basis, the system identifies the homonyms that may have multiple language interpretations in the text, and extracts the pronunciation rules, phoneme mapping table and semantic role information of these words in different languages as basic data support for subsequent sampling.

[0094] Subsequently, the system sets a preset sampling frequency threshold to determine how many times of language path simulation for each homonym. To ensure that the sampling process conforms to the context logic, the system introduces a context constraint factor for each sampling. The factor is dynamically generated based on the consistency degree of the language categories of the adjacent words before and after the current word. If the adjacent words highly consistently use a language, the current word is also more inclined to the language; otherwise, if there is a language switching trend in the context, the language selection range is appropriately relaxed. Through this mechanism, the system effectively limits unreasonable language label combinations, improves the coherence and credibility of the sampling results.

[0095] In each sampling process, the system selects a possible language path from the candidate language labels according to the context semantic coherence model, and records the frequency of each language label in the path. This process is repeated until all sampling tasks are completed. Finally, the system counts the proportion of each language label in the entire sampling process to form the preliminary pronunciation path selection probability distribution characteristic value.

[0096] To further improve the rationality of pronunciation path selection, the system establishes a syntactic structure matching degree evaluation module. This module analyzes whether the current sampling path conforms to the typical syntactic patterns of the target language, such as the subject-predicate-object order, the use of prepositions, the position of verb tenses, etc. The system calculates the syntactic rationality score of each path accordingly, and introduces a syntactic adaptation weight λ∈[0,1] to weight and correct the path score, where λ close to 1 indicates that the path syntax structure is reasonable, and λ close to 0 indicates that there are syntax errors or structures that do not conform to language habits in the path.

[0097] In addition, the system also extracts language use tendency data of the user or the text source in combination with the historical language use preference database. For example, for a user who has been using English for a long time, the system can adjust the weight of English in the path score. For this purpose, the system sets a language preference adjustment coefficient to adjust the path score according to the user's language use habits, thereby realizing personalized language recognition and pronunciation selection.

[0098] In order to enhance the stability and robustness of path selection, the system performs clustering analysis on multiple sampling results, identifies high-frequency language path combinations, and defines them as "dominant pronunciation path clusters". Subsequently, the system measures the distance between the dominant path and other paths, normalizes the distance to obtain a path stability index ρ∈[0,1], which is used to evaluate the stability of the current word pronunciation selection. The higher the value of ρ, the more concentrated and stable the path selection is; the lower the value of ρ, the more uncertain the path.

[0099] At the same time, the system sets a language switching cost function to punish paths that frequently switch languages. Specifically, whenever a language switching behavior occurs in a sampling path, the system applies a certain percentage of deduction to the overall score of the path according to the switching frequency and the context support, preventing the model from frequently changing language labels without clear context support.

[0100] Finally, the system integrates the above factors including context consistency, syntactic matching degree, user language preference, path stability and language switching cost, and recalculates the final probability distribution of each language label by normalization, generating the optimized pronunciation path selection probability distribution feature value. This feature value not only reflects the most likely language and pronunciation path of the current word, but also contains the stability and context adaptability information of the path selection, providing comprehensive and accurate input basis for the subsequent pronunciation decision module.

[0101] In S4, the language attribution confidence feature value and the pronunciation path selection probability distribution feature value are fused into a multi-language pronunciation decision vector, which is input into the multi-language end-to-end speech synthesis model, and the corresponding phoneme mapping rules and acoustic parameter prediction module are called to generate the target speech spectrum graph, which specifically includes:

[0102] After completing the language recognition and pronunciation path modeling of the input text, the system enters the final speech generation stage. The core task of this stage is to fuse the language attribution confidence feature values obtained from the previous processing with the pronunciation path selection probability distribution feature values to form a multi-language pronunciation decision vector, which is used to guide the subsequent speech synthesis model to select appropriate language pronunciation rules and generate high-quality target speech output.

[0103] Firstly, the system constructs a multi-language pronunciation decision vector space, where each dimension corresponds to a candidate language (such as Chinese, English, French, etc.), and sets the basic weights for each language dimension in the initialization stage. These initial weights reflect the system's default preference for different languages in general scenarios, which can be preset according to the language distribution of the training corpus or the application scenario.

[0104] Subsequently, the system dynamically adjusts the weights of each language dimension according to the language attribution confidence feature values generated in the previous step. For languages with high confidence, the system appropriately increases their corresponding weights to give them higher priority in subsequent pronunciation path selection; for low-confidence languages, their weights are correspondingly reduced, or even completely excluded, thereby improving the accuracy and stability of language selection.

[0105] On this basis, the system combines the pronunciation path selection probability distribution feature values obtained from S3 step and uses a weighted fusion strategy to calculate the final multi-language pronunciation decision vector. This vector not only represents the most likely language category used by the current semantic unit, but also quantifies the pronunciation tendency strength of that language and whether the multi-language mixed pronunciation mode needs to be triggered. To further enhance the flexibility and adaptability of the decision, the system sets a language activation threshold: when the score of a language dimension in the decision vector exceeds this threshold, that language is determined as the dominant language of the current semantic unit and is passed as a label to the speech synthesis module; otherwise, the system will start the multi-language mixed pronunciation mechanism to support cross-language continuous expression.

[0106] After completing the language decision, the system enters the first step of phoneme mapping in the speech generation process. For this purpose, the system pre-establishes a multi-language phoneme mapping rule library, which stores the conversion rules from orthography to phoneme sequences for multiple candidate languages and is indexed and managed by language categories. The system calls the corresponding phoneme mapping module from the rule library according to the dominant language label or mixed language combination output in the previous stage, converts the current text unit into the corresponding phoneme sequence, and provides structured input for subsequent acoustic modeling.

[0107] To further improve the naturalness and language adaptation ability of the synthesized speech, the system constructs a language-adaptive acoustic parameter prediction module. Based on the prosodic features of the dominant language (such as stress position, intonation change), contextual information (such as language consistency of previous and subsequent words), and probability distribution feature values of pronunciation path selection, the module jointly predicts multiple key acoustic parameters, including pitch contour, energy profile, phoneme duration, and pause positions. By introducing a language perception mechanism, the module can automatically adjust the parameter prediction strategy according to the pronunciation habits of different languages, making the synthesized speech more consistent with the intonation and rhythm of the target language.

[0108] Next, the system inputs the above-mentioned phoneme sequence and predicted acoustic parameters into a multi-language compatible end-to-end speech synthesis model. The model internally contains a language-aware attention mechanism and a multi-language shared encoder, which can efficiently process input data from different language systems in a unified framework. The language-aware attention mechanism helps the model focus on context information highly relevant to the current language when generating speech, while the multi-language shared encoder ensures that the model still has good generalization ability when facing new languages or rare language combinations.

[0109] After model processing, the system outputs the mel-spectrogram of the target speech. The spectrogram is a two-dimensional matrix, where each column represents a time frame and each row represents the energy distribution of a frequency band, fully describing the time-frequency characteristics of the speech signal. To further restore the spectral information to actual audible speech waveforms, the system calls a high-performance neural vocoder model (such as HiFi-GAN, WaveGlow, or LPCNet) to convert the mel-spectrogram into high-quality time-domain speech waveforms.

[0110] Finally, the system can also perform post-processing operations on the output waveform, including but not limited to gain adjustment, background noise suppression, speech rate fine-tuning, etc., to further improve the naturalness, clarity, and user experience of the speech.

[0111] In S5, according to the target speech spectrogram, the Bayesian prior distribution and language recognition threshold are dynamically adjusted to realize online fine-tuning and continuous optimization of the model, improving the generalization ability and robustness of the system in a multi-language mixed environment, including:

[0112] Firstly, during the initial operation of the system or when processing a new language, the initial Bayesian prior distribution is set based on the parameters in the pre-trained model. These prior distributions are set based on the expected distribution of different acoustic features (such as spectrum, fundamental frequency, etc.) across different languages, laying the foundation for subsequent processing. After generating the target speech spectrum each time, the system automatically compares the difference between the actual output spectrum and the spectrum under ideal conditions. Based on the analysis results, the system can dynamically adjust the Bayesian prior distribution to make it more close to the actual situation, effectively improve the capture ability of language characteristics, and reduce errors caused by language differences. In addition, the system also designs a feedback loop to continuously correct the Bayesian prior distribution based on user evaluation or automatic evaluation indicators (such as naturalness score, understanding accuracy, etc.), ensuring that it is always in the optimal state.

[0113] Before starting to process multi-language input, the system needs to set initial recognition thresholds for each supported language. These thresholds determine how the system distinguishes between different languages and when to trigger the mixed pronunciation mode. As the system receives more real-world speech data, the original language recognition thresholds may no longer be applicable. Therefore, the system dynamically adjusts these thresholds based on the latest data flow. For example, when detecting that a certain language has significantly increased in frequency, the recognition threshold for that language is appropriately lowered to increase recognition sensitivity; conversely, if the misrecognition rate of a certain language is high, the threshold should be considered to be raised to reduce errors. In addition to this, the decision logic involving multi-language mixed pronunciation needs to be optimized, including determining when and how to smoothly transition from one language to another, ensuring that the entire speech output is smooth and natural.

[0114] To ensure that the system can continuously improve in actual use, an online learning method is used to update model parameters regularly. This method allows the system to adjust itself in real time using newly collected data without retraining the entire model, greatly improving efficiency and response speed. At the same time, a complete set of performance monitoring indicators is set up to track the system's performance under different conditions. Once performance degradation or other problems are found, appropriate optimization measures are immediately initiated. Considering that each user's preferences and habits may differ, personalized model fine-tuning can also be used to meet specific needs, providing a more customized service experience.

[0115] Referring to Figure 2 A speech conversion information processing system is shown, comprising:

[0116] A multi-language mixed text context modeling module is used to receive mixed text input containing two or more languages, perform word segmentation, semantic unit segmentation, and context window construction on the text, extract language feature vectors for each word, including spelling structure, syllable distribution, letter frequency, and syntactic dependency relationships;

[0117] A Bayesian posterior inference language recognition feature generation module calculates the posterior probability of each word under different languages according to a pre-trained language prior probability model and a conditional probability model, and dynamically evaluates the most possible language category to which a current semantic segment belongs in combination with a sliding window mechanism to generate a language attribution confidence feature value;

[0118] A Monte Carlo simulation pronunciation ambiguity resolution feature generation module adopts a Monte Carlo sampling method to perform multiple context-sensitive simulations for homophonic words that have multiple language interpretation possibilities, counts the frequency of occurrence of each language label, and generates a pronunciation path selection probability distribution feature value;

[0119] A multi-language fusion speech synthesis decision module fuses the language attribution confidence feature value and the pronunciation path selection probability distribution feature value into a multi-language pronunciation decision vector, inputs the vector into a multi-language end-to-end speech synthesis model, calls a phoneme mapping rule and an acoustic parameter prediction module corresponding to a language, and generates a target speech spectrogram;

[0120] A feedback-driven online adaptive optimization module dynamically adjusts a Bayesian prior distribution and a language recognition threshold according to the target speech spectrogram, realizes online fine-tuning and continuous optimization of the model, and improves the generalization ability and robustness of the system in a multi-language mixed environment.

[0121] The working principle of the application: aims to solve the problems of inaccurate language recognition, pronunciation path ambiguity and unnatural speech output in speech synthesis in cross-language scenarios. The system includes five core modules: multi-language mixed text context modeling module, Bayesian posterior inference language recognition feature generation module, Monte Carlo simulation pronunciation ambiguity resolution feature generation module, multi-language fusion speech synthesis decision module, and feedback-driven online adaptive optimization module. The system receives mixed text input containing two or more languages, and performs morpheme-level segmentation and semantic unit division through a multi-language compatible word segmentation engine, builds a context window and extracts the language feature vector of each word, covering spelling structure, syllable distribution, letter frequency and syntactic dependency. Then, combined with the language prior and conditional probability model, the posterior probability of the word in different languages is calculated by Bayesian inference, and the sliding window mechanism and context consistency weight factor are introduced to dynamically evaluate the language attribution of the semantic segment and generate language attribution confidence feature values. For homonym and other pronunciation ambiguity problems, the system uses the Monte Carlo sampling method for multiple context-sensitive simulations, counts the frequency of each language label, and combines the syntactic adaptation weight, user language preference coefficient, path stability index and language switching cost function to generate optimized pronunciation path selection probability distribution feature values. The feature values are further fused with the language attribution confidence to form a multi-language pronunciation decision vector, which guides the end-to-end speech synthesis model to call the corresponding phoneme mapping rules and acoustic parameter prediction module to generate high-quality mel spectrum, and restore it to natural speech through a neural vocoder. The system adjusts the Bayesian prior distribution and language recognition threshold dynamically according to the output speech spectrum, realizes online fine-tuning and continuous optimization of the model, and significantly improves its generalization ability and robustness in complex multilingual environments. The application is suitable for international conference translation, intelligent customer service, cross-language education and other application scenarios, and has high intelligence, personalization and scalability.

[0122] The above describes one embodiment of the application in detail, but the content described is only the preferred embodiment of the application and cannot be considered as limiting the scope of the application. Any equivalent changes and improvements made in the scope of the application should still be within the scope of the application.

Claims

1. A method for processing information for speech conversion, characterized in that: The following steps are involved: S1: Receive mixed text input containing two or more languages, perform word segmentation, semantic unit segmentation, and context window construction on the text, and extract the language feature vector of each word, including spelling structure, syllable distribution, letter frequency, and syntactic dependency; S2: Based on the pre-trained language prior probability model and conditional probability model, the posterior probability of each word in different languages ​​is calculated. In combination with the sliding window mechanism, the most likely language category of the current semantic segment is dynamically evaluated to generate a language attribution confidence feature value. S3: For homographs with multiple possible interpretations in different languages, we use the Monte Carlo sampling method to perform multiple context-sensitive simulations, count the frequency of occurrence of each language label, and generate the probability distribution feature value of the pronunciation path selection; S4: The language attribution confidence feature value and the pronunciation path selection probability distribution feature value are combined into a multilingual pronunciation decision vector, which is then input into a multilingual end-to-end speech synthesis model. The phoneme mapping rules and acoustic parameter prediction module of the corresponding language are called to generate a target speech spectrogram. S5: Dynamically adjust the Bayesian prior distribution and language recognition threshold based on the target speech spectrogram to achieve online fine-tuning and continuous optimization of the model, improving the system's generalization and robustness in multilingual mixed environments.

2. The information processing method for voice conversion according to claim 1, characterized in that: The calculation of the posterior probability of each word in different languages ​​based on the pre-trained language prior probability model and conditional probability model specifically includes: A language prior probability model is established, which records the basic frequency of occurrence of multiple candidate languages ​​in the entire corpus. By statistically analyzing the global word frequency distribution of each language in a large-scale multilingual text dataset and performing normalization, the basic usage preference probability of each language is obtained. A linguistic conditional probability model is constructed to represent the probability of a word appearing in a specific language. The model is generated based on local lexical statistics within each language and incorporates a smoothing mechanism to avoid zero probabilities caused by rare words not appearing in the training set. The Bayesian inference method is used to comprehensively judge the possibility of each word in different languages. That is, the basic usage frequency of the language and the possibility of the word appearing in the language are combined to calculate the relative credibility of the word belonging to each language; and a minimum credibility threshold is set. When the credibility corresponding to a language is lower than the threshold, the possibility of the language as the language to which the word belongs is excluded.

3. The information processing method for voice conversion according to claim 1, characterized in that: The method of dynamically evaluating the most likely language category of the current semantic segment by combining the sliding window mechanism includes the following steps: Set a fixed-length sliding window and an appropriate sliding step size to sequentially capture continuous semantic segments in the input text; For each word set in the intercepted semantic segment, calculate their average language credibility scores in each candidate language as the overall matching degree of the semantic segment in different languages; A contextual consistency weight factor is introduced to measure the coherence of the language categories between the current semantic segment and its previous segment. By weightedly combining the language scores of the two segments, the system can better maintain the continuity of language usage when judging the current language. Finally, based on the weighted comprehensive language score, the language with the highest score is selected as the predicted language label for the current semantic segment.

4. The information processing method for voice conversion according to claim 1, characterized in that: The specific process of generating the pronunciation path selection probability distribution feature value is as follows: Based on the distribution of language scores of the current semantic segment across all languages, analyze the degree of uncertainty in its language classification, specifically whether the distribution is concentrated in a certain language or dispersed across multiple languages; The score difference between the dominant language and other languages ​​is calculated to reflect the clarity of language recognition in the current semantic segment. If the dominant language score is much higher than other languages, it means that the language recognition is relatively clear. The fuzzy clustering method is used to perform pattern recognition on the language scores of multiple continuous semantic segments to analyze whether there is a trend of frequent switching in language use, thereby identifying potential language boundary areas and adjusting the confidence correction coefficient accordingly. Based on the above information, a language attribution confidence feature value is generated. The language attribution confidence feature value reflects the reliability of language recognition of the current semantic segment. The higher the value, the more accurate and stable the language recognition.

5. The information processing method for voice conversion according to claim 1, characterized in that: The Monte Carlo sampling method is used to perform multiple context-sensitive simulations, specifically including: Construct a set of candidate language tags, identify homographs in the current text that may have multiple language interpretations, and extract their pronunciation rules and semantic roles in different languages; A sampling threshold is set, and a context constraint factor is introduced for each sampling process. This factor is dynamically generated based on the degree of consistency in the language categories of the adjacent words before and after the current word, and is used to limit unreasonable language label combinations during the sampling process. In each sampling, a possible language tag path is selected based on the contextual semantic coherence model, and the frequency of occurrence of each language tag in the path is recorded; After completing all sampling, the proportion of each language label in all sampling paths is counted to form a preliminary pronunciation path selection probability distribution feature value.

6. The information processing method for voice conversion according to claim 1, characterized in that: The method of generating a pronunciation path selection probability distribution feature value comprises the following steps: Establish a syntactic structure matching evaluation module to analyze whether the current sampling path conforms to the typical syntactic patterns of the target language, such as subject-verb-object order and preposition usage habits; The syntax adaptation weight λ∈[0,1] is introduced to perform weighted correction on the probability score of the current sampling path according to the syntax rationality of the path. Combined with the historical language usage preference database, the language usage tendency data of users or text sources is extracted, and the language preference adjustment coefficient is set to further adjust the path score; The path score after syntactic correction is integrated with the influence of user language preference, and the final probability distribution of each language label is renormalized and calculated as the characteristic value of the optimized pronunciation path selection probability distribution.

7. The information processing method for voice conversion according to claim 1, characterized in that: The process of generating the pronunciation path selection probability distribution feature value further includes the following steps: Cluster analysis was performed on the results of multiple samplings to identify the frequently occurring language path combinations and define them as the dominant pronunciation path clusters. By measuring the distance between the dominant path and other paths, the distance is normalized to obtain the path stability index ρ∈[0,1], which is used to evaluate the stability of the current vocabulary pronunciation selection; Set a language switching cost function. If frequent language switching occurs in a sampling path, the overall path score will be reduced accordingly. The path stability index and language switching cost are combined to generate the final pronunciation path selection probability distribution feature value, which is output to the subsequent pronunciation decision module to guide the language label selection in the multilingual mixed speech synthesis process.

8. The information processing method for voice conversion according to claim 1, characterized in that: The method of fusing the language attribution confidence feature value and the pronunciation path selection probability distribution feature value into a multilingual pronunciation decision vector comprises the following steps: Construct a multilingual pronunciation decision vector space, where each dimension corresponds to a candidate language, and initialize the initial weights of each dimension; Dynamically adjust the basic weights of each language dimension based on the language attribution confidence feature value, so that high-confidence languages ​​have higher priority in subsequent decision-making; Combined with the pronunciation path selection probability distribution eigenvalues, a weighted fusion method is used to calculate the final multilingual pronunciation decision vector, which is used to represent the most likely language used by the current semantic unit and its corresponding pronunciation tendency strength; Set a language activation threshold. If the decision vector component of a language dimension exceeds the threshold, it will be output to the speech synthesis model as the dominant language label. Otherwise, the multilingual mixed pronunciation mode will be triggered.

9. The information processing method for voice conversion according to claim 1, characterized in that: The calling of the phoneme mapping rules and acoustic parameter prediction module of the corresponding language to generate the target speech spectrogram specifically includes: Establishing a multilingual phoneme mapping rule base, the rule base stores the conversion rules of orthography to phoneme sequences of multiple candidate languages ​​and manages indexes by language category; calling the corresponding phoneme mapping module from the rule base based on the dominant language label or mixed language combination to generate a phoneme sequence representation of the current text unit; A language-adaptive acoustic parameter prediction module is constructed. This module selects probability distribution features based on the prosodic characteristics of the dominant language, contextual information, and articulatory path, and jointly predicts key acoustic parameters such as fundamental frequency trajectory, energy distribution, phoneme duration, and pause position. Inputting the phoneme sequence and the predicted acoustic parameters into a multi-language compatible end-to-end speech synthesis model, which includes a language-aware attention mechanism and a multi-language shared encoder for fusing language features with acoustic modeling information; The speech synthesis model outputs a mel-spectrogram of the target speech, where each time frame corresponds to a set of frequency band energy distribution vectors; the mel-spectrogram is then restored to a time-domain speech waveform through a neural vocoder model, and the mel-spectrogram of the target speech is the target speech spectrogram.

10. A voice conversion information processing system, characterized in that: An information processing method for speech conversion according to any one of claims 1 to 9, comprising: A multilingual mixed text context modeling module, which is used to receive mixed text input containing two or more languages, perform word segmentation, semantic unit segmentation, and context window construction on the text, and extract the language feature vector of each word, including spelling structure, syllable distribution, letter frequency, and syntactic dependency; A Bayesian posterior reasoning language identification feature generation module, which calculates the posterior probability of each word in different languages ​​based on a pre-trained language prior probability model and conditional probability model, and dynamically evaluates the most likely language category of the current semantic segment using a sliding window mechanism to generate a language attribution confidence feature value; A Monte Carlo simulation pronunciation ambiguity resolution feature generation module, wherein the Monte Carlo simulation pronunciation ambiguity resolution feature generation module uses a Monte Carlo sampling method to perform multiple context-sensitive simulations for homographs with multiple possible language interpretations, counts the frequency of occurrence of each language label, and generates a pronunciation path selection probability distribution feature value; A multilingual fusion speech synthesis decision module, which fuses the language attribution confidence feature value and the pronunciation path selection probability distribution feature value into a multilingual pronunciation decision vector, inputs the vector into the multilingual end-to-end speech synthesis model, calls the phoneme mapping rules and acoustic parameter prediction module of the corresponding language, and generates a target speech spectrogram; A feedback-driven online adaptive optimization module dynamically adjusts the Bayesian prior distribution and language recognition threshold based on the target speech spectrogram, enabling online fine-tuning and continuous optimization of the model, thereby improving the system's generalization and robustness in a multilingual mixed environment.

Images