Southern Fujian Chinese opera voice changing method and system based on RVC network

Abstract

The invention provides a southern Fujian Chinese opera voice changing method and system based on an RVC network in the technical field of artificial intelligence and voice crossing. The method comprises the steps that S1, a large number of historical Chinese audios and historical southern Fujian Chinese opera audios are acquired to construct a data set; s2, creating a semantic dictionary used for storing the corresponding relationship among the southern Fujian pinyin, the Chinese pinyin and the digital sequence, and the corresponding relationship between the pitch and the digital sequence; the corresponding relations are separated through preset section separators; s3, creating a southern Fujian opera sound change model, and setting a loss function of the southern Fujian opera sound change model; s4, training a southern Fujian Chinese opera voice change model through the data set and the loss function; and step S5, deploying the trained Southern Fujian Chinese opera voice changing model, and carrying out Southern Fujian Chinese opera voice changing operation through the deployed Southern Fujian Chinese opera voice changing model. The method has the advantage that the sound change quality of the Chinese opera of the southern Fujian is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and speech intersection technology, and in particular to a method and system for voice changing of Minnan opera based on an RVC network. Background Art

[0002] With the booming digital music industry, cross-lingual musical dissemination has become a crucial channel for cultural output. According to statistics, over 37% of users on global music streaming platforms actively search for songs in different languages. This demand is particularly prominent in the preservation and inheritance of local dialect culture. Traditional music production models require re-creation and recording for different language markets after recording a song. This not only exponentially increases production costs but also hinders the maximization of the cultural value of a large stock of musical works due to language barriers.

[0003] Current mainstream speech conversion technology primarily focuses on building technical systems around major languages. Deep learning models such as WaveNet and Tacotron have reached commercial standards for processing common languages like English and Mandarin. Their technical approach typically involves: 1) extracting speech features based on large-scale corpora (typically requiring over 100 hours of standard speech data); 2) using generative adversarial networks for timbre transfer; and 3) maintaining the natural fluency of the language through prosody prediction models. However, this technical paradigm systematically fails when dealing with Minnan dialect, as shown by the following:

[0004] 1) Phonological dimension: Minnan dialect fully preserves the "eight tones" system of ancient Chinese (yinping, yangping, shangsheng, yinqu, yangqu, yinru, yangru, and yangdi). Its tone variation complexity is 3.2 times that of Mandarin. Existing tone-shifting models cannot accurately capture the rules of continuous tone shifting.

[0005] 2) Differences in vocalization mechanisms: The glottal stop sounds unique to Minnan opera (e.g., the pronunciation of the word “duck” [ah]) result in 20-30% information loss in traditional MFCC feature extraction.

[0006] 3) Cultural context specificity: The acoustic characteristics of the emotional expressions unique to opera singing, such as the "crying tune" and the "seven-character tune", are significantly different from those in daily conversations. Existing emotion transfer models cannot effectively isolate these artistic vocal features.

[0007] It is worth noting that directly migrating a large language model to Minnan opera voice changing will result in chain errors: first, in the phoneme alignment stage, due to the lack of a dialect-specific phoneme set (Mandarin has only more than 400 phonemes, while Minnan dialect has 620), about 28% of syllables have mapping errors; second, in the rhythm modeling link, the unique dragging of opera (the duration of a single word can reach 3.5 seconds) will destroy the timing prediction mechanism of the LSTM network; the final generated audio will have an obvious "mechanical cavity" phenomenon.

[0008] Therefore, how to provide a method and system for voice changing of Minnan opera based on RVC network to improve the voice changing quality of Minnan opera has become a technical problem to be solved urgently. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a method and system for voice changing of Minnan opera based on RVC network, so as to improve the voice changing quality of Minnan opera.

[0010] In a first aspect, the present invention provides a method for voice changing of Minnan opera based on an RVC network, comprising the following steps:

[0011] Step S1: obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, and constructing a data set after preprocessing and annotating the historical Chinese audio and historical Minnan opera audio;

[0012] Step S2: creating a semantic dictionary for storing the correspondence between Minnan dialect pinyin, Chinese pinyin and number sequences, and the correspondence between pitch and number sequences; each of the correspondences is separated by a preset measure separator;

[0013] Step S3: creating a Minnan opera voice change model based on the input module, the semantic conversion module, and the RVC network, and setting a loss function of the Minnan opera voice change model;

[0014] The input module is used to extract Chinese pinyin from the input Chinese audio and input the Chinese pinyin into the semantic conversion module; the semantic conversion module is used to map the Chinese pinyin to obtain a corresponding digital sequence through a semantic dictionary, and input the digital sequence into the RVC network; the RVC network is used to generate the Minnan opera audio corresponding to the Chinese audio based on the digital sequence and the semantic dictionary;

[0015] Step S4: training the Minnan opera voice change model using the data set and the loss function;

[0016] Step S5: deploying the trained Minnan opera voice change model, and performing Minnan opera voice change operations using the deployed Minnan opera voice change model.

[0017] Furthermore, the step S1 is specifically as follows:

[0018] Obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, performing noise reduction on each of the historical Chinese audio and historical Minnan opera audio, segmenting each of the historical Chinese audio and historical Minnan opera audio based on rhythm or beat to obtain a plurality of Chinese sub-audios and opera sub-audios, and associating each of the Chinese sub-audios with the opera sub-audios to complete preprocessing of each of the historical Chinese audio and historical Minnan opera audio;

[0019] Each of the Chinese sub-audio codes is annotated with Chinese pinyin; each of the opera sub-audio codes is annotated with lyrics, duration, and pitch; the lyrics are in Minnan pinyin and separated by preset measure delimiters; the duration is used to identify the duration of each Minnan pinyin in the lyrics in seconds; and the pitch is used to identify the melody pitch of each Minnan pinyin in the lyrics;

[0020] A data set is constructed based on the labeled Chinese sub-audios and opera sub-audios.

[0021] Furthermore, in step S3, the input module is constructed based on a convolutional neural network and adopts a CTC loss function;

[0022] The semantic conversion module is constructed based on a recurrent neural network.

[0023] Furthermore, the step S4 is specifically as follows:

[0024] Dividing the data set into a training set, a validation set, and a test set based on a ratio of 8:1:1, and training the Minnan opera voice change model using the training set until the loss value of the loss function is less than a preset loss threshold;

[0025] The trained Minnan Opera voice change model is verified using the verification set to determine whether the voice change accuracy is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training. If so, the verification passes, and:

[0026] The verified Minnan opera voice change model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test passes and the training ends.

[0027] Furthermore, the step S5 is specifically as follows:

[0028] The trained Minnan opera voice changing model is deployed to a server through containerization technology, and an API interface for calling the Minnan opera voice changing model is set to obtain real-time Chinese audio. After pre-processing the real-time Chinese audio, the Minnan opera voice changing model is input through the API interface to obtain the Minnan opera audio corresponding to the real-time Chinese audio, so as to perform the Minnan opera voice changing operation.

[0029] In a second aspect, the present invention provides a voice changing system for Minnan opera based on an RVC network, comprising the following modules:

[0030] A data set construction module is used to obtain a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, and to construct a data set after preprocessing and annotating the historical Chinese audio and historical Minnan opera audio;

[0031] A semantic dictionary creation module is used to create a semantic dictionary for storing the correspondence between Minnan Pinyin, Chinese Pinyin and number sequences, and the correspondence between pitch and number sequences; each of the correspondences is separated by a preset section separator;

[0032] A Minnan opera voice change model creation module is used to create a Minnan opera voice change model based on the input module, the semantic conversion module and the RVC network, and set the loss function of the Minnan opera voice change model;

[0033] The input module is used to extract Chinese pinyin from the input Chinese audio and input the Chinese pinyin into the semantic conversion module; the semantic conversion module is used to map the Chinese pinyin to obtain a corresponding digital sequence through a semantic dictionary, and input the digital sequence into the RVC network; the RVC network is used to generate the Minnan opera audio corresponding to the Chinese audio based on the digital sequence and the semantic dictionary;

[0034] A Minnan opera voice change model training module, used to train the Minnan opera voice change model using the data set and loss function;

[0035] The Minnan opera voice changing module is used to deploy the trained Minnan opera voice changing model and perform Minnan opera voice changing operations through the deployed Minnan opera voice changing model.

[0036] Furthermore, the dataset construction module is specifically used to:

[0037] Obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, performing noise reduction on each of the historical Chinese audio and historical Minnan opera audio, segmenting each of the historical Chinese audio and historical Minnan opera audio based on rhythm or beat to obtain a plurality of Chinese sub-audios and opera sub-audios, and associating each of the Chinese sub-audios with the opera sub-audios to complete preprocessing of each of the historical Chinese audio and historical Minnan opera audio;

[0038] Each of the Chinese sub-audio codes is annotated with Chinese pinyin; each of the opera sub-audio codes is annotated with lyrics, duration, and pitch; the lyrics are in Minnan pinyin and separated by preset measure delimiters; the duration is used to identify the duration of each Minnan pinyin in the lyrics in seconds; and the pitch is used to identify the melody pitch of each Minnan pinyin in the lyrics;

[0039] A data set is constructed based on the labeled Chinese sub-audios and opera sub-audios.

[0040] Furthermore, in the Minnan opera voice change model creation module, the input module is constructed based on a convolutional neural network and adopts a CTC loss function;

[0041] The semantic conversion module is constructed based on a recurrent neural network.

[0042] Furthermore, the Minnan Opera voice change model training module is specifically used to:

[0043] Dividing the data set into a training set, a validation set, and a test set based on a ratio of 8:1:1, and training the Minnan opera voice change model using the training set until the loss value of the loss function is less than a preset loss threshold;

[0044] The trained Minnan Opera voice change model is verified using the verification set to determine whether the voice change accuracy is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training. If so, the verification passes, and:

[0045] The verified Minnan opera voice change model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test passes and the training ends.

[0046] Furthermore, the Minnan Opera voice changing module is specifically used for:

[0047] The trained Minnan opera voice changing model is deployed to a server through containerization technology, and an API interface for calling the Minnan opera voice changing model is set to obtain real-time Chinese audio. After pre-processing the real-time Chinese audio, the Minnan opera voice changing model is input through the API interface to obtain the Minnan opera audio corresponding to the real-time Chinese audio, so as to perform the Minnan opera voice changing operation.

[0048] The advantages of the present invention are:

[0049] 1. By obtaining a large amount of historical Chinese audio and the historical Minnan opera audio corresponding to the historical Chinese audio, each historical Chinese audio and historical Minnan opera audio is preprocessed and annotated to construct a data set; then a semantic dictionary is created to store Minnan pinyin, the correspondence between Chinese pinyin and digital sequences, and the correspondence between pitch and digital sequences; a Minnan opera voice change model is created based on the input module, the semantic conversion module and the RVC network, and the loss function of the Minnan opera voice change model is set; the input module is used to extract Chinese pinyin from the input Chinese audio; the semantic conversion module is used to map the Chinese pinyin to the corresponding digital sequence through the semantic dictionary; RV The C network is used to generate Minnan opera audio corresponding to Chinese audio based on a digital sequence and a semantic dictionary; then, the Minnan opera voice change model is trained using a data set and a loss function, the trained Minnan opera voice change model is deployed, and the Minnan opera voice change operation is performed using the deployed Minnan opera voice change model; that is, the Minnan opera voice change model is trained using a data set constructed from historical Chinese audio and historical Minnan opera audio. The data set is labeled based on the characteristics of the Minnan dialect. During the training process of the Minnan opera voice change model, the semantic dictionary is combined to effectively identify and convert the semantics of the audio, thereby greatly improving the quality of Minnan opera voice change.

[0050] 2. By collecting historical Chinese audio and corresponding historical Minnan opera audio and performing associated segmentation, we constructed a high-quality aligned dataset. This data pairing solves the problem of ambiguous data mapping in traditional speech conversion and ensures that the model learns accurate cross-language phoneme correspondences.

[0051] 3. By annotating the opera audio with Minnan dialect pinyin, duration, and pitch, the unique melodic elements (pitch) and duration (rhythm) of opera are incorporated into the training data for the first time, enabling the model to capture the rhythmic characteristics of opera singing, breaking through the limitation of traditional voice changing technology that only focuses on timbre conversion.

[0052] 4. Through the semantic dictionary, Chinese pinyin, Minnan pinyin, digital sequences and pitch mapping are integrated, and different elements are distinguished by section separators, which solves the problem of joint encoding of multimodal features (language, pitch, rhythm), and significantly improves the information integrity and accuracy during the conversion process.

[0053] 5. Through the bidirectional mapping of Chinese pinyin and Minnan pinyin, combined with pitch digital serialization, a nonlinear conversion from Mandarin phonemes to operatic phonemes is achieved. This is particularly suitable for scenarios where there are complex pronunciation differences between Minnan and Chinese.

[0054] 6. By adopting a hierarchical structure of CNN (input module) + RNN (semantic conversion module) + RVC network, CNN efficiently extracts audio spectrum features, RNN handles sequence mapping tasks, and the RVC network integrates the semantic dictionary to generate target audio. Each module has a clear division of labor and collaborative optimization, taking into account both feature extraction and sequence modeling capabilities.

[0055] 7. During the training process, if the accuracy of the validation set or the confidence of the test set does not meet the standards, the training set will be automatically expanded. This dynamic feedback mechanism effectively solves the overfitting problem under small sample data and improves the generalization ability of the model.

[0056] 8. Through phased verification of training sets, validation sets, and test sets, the model is ensured to achieve a balance in training adequacy, parameter adjustment rationality, and final performance reliability, which is superior to traditional evaluation methods that only rely on a single test set.

[0057] 9. Deploy models through containerization technology to achieve rapid service encapsulation and resource isolation, support high-concurrency real-time reasoning, and meet the low-latency requirements of opera voice changing in scenarios such as mobile terminals and performance systems.

[0058] 10. By forming a complete closed loop from real-time audio input, preprocessing to API interface calls, users do not need to manually intervene in complex sound feature processing steps, which greatly reduces the usage threshold and improves the universality of the technology.

[0059] 11. In view of the complex tones and unique intonations of Minnan opera, the pitch mapping and time value annotation in the semantic dictionary are used to preserve the ups and downs of opera singing, filling the gap in the application of traditional voice changing technology in the field of intangible cultural heritage.

[0060] 12. By combining noise reduction technology with rhythm-based segmentation methods in the data preprocessing stage, we can not only eliminate background interference and improve the purity of speech, but also ensure the temporal consistency of Chinese and opera audio through beat alignment. This dual processing significantly enhances the model's ability to capture key acoustic features (such as pitch and resonance peaks) and avoids mapping deviations caused by noise pollution; the rhythm-based segmentation strategy (rather than fixed-duration cutting) is more in line with the ups and downs of opera singing, and is especially suitable for long-tone processing such as dragging and throwing, enabling the model to learn the unique elastic rhythm pattern of opera and enhance the artistic expression of the generated audio.

[0061] 13. By constructing a dual-source aligned dataset and a multi-dimensional annotation system, combined with a structured semantic dictionary, accurate mapping of cross-language phonemes, pitch, and rhythm is achieved. A CNN-RNN-RVC collaborative network is used for hierarchical extraction of acoustic features and sequence modeling, and dynamic data enhancement and a three-stage verification mechanism are used to improve model robustness. Containerized deployment and an end-to-end preprocessing pipeline support real-time and efficient voice changing. Its modular design is scalable to multiple dialects and opera types, and a composite loss function is used to optimize the consistency of timbre, pitch, and duration. Ultimately, while addressing the lack of artistic expression of traditional voice changing technology, high-fidelity conversion of ordinary speech into Minnan opera singing is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] Figure 1 The present invention is a flow chart of a method for voice changing of Minnan opera based on RVC network.

[0064] Figure 2 The present invention is a structural schematic diagram of a Minnan opera voice changing system based on an RVC network. DETAILED DESCRIPTION

[0065] The technical solution in the embodiments of the present application has the following overall idea: a Minnan opera voice change model is trained using a dataset constructed using historical Chinese audio and historical Minnan opera audio. The dataset is annotated based on the characteristics of the Minnan dialect. During the training process of the Minnan opera voice change model, a semantic dictionary is combined to effectively identify and convert the semantics of the audio, thereby improving the quality of Minnan opera voice change.

[0066] Please refer to Figures 1 to 2 As shown, a preferred embodiment of the present invention's method for voice changing of Minnan opera based on RVC network includes the following steps:

[0067] Step S1: obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, and constructing a data set after preprocessing and annotating the historical Chinese audio and historical Minnan opera audio;

[0068] Step S2: creating a semantic dictionary for storing the correspondence between Minnan Pinyin, Chinese Pinyin and numerical sequences, and the correspondence between pitches and numerical sequences; each of the correspondences is separated by a preset measure separator; the measure separator is sp; for example, put is Minnan Pinyin, representing "no", and two uts represent a singing of two consecutive tones; for example, pitch C0 corresponds to 1, measure separator sp corresponds to 127, and Minnan Pinyin i corresponds to 131;

[0069] Through the semantic dictionary, Chinese pinyin, Minnan pinyin, digital sequences and pitch mapping are integrated, and different elements are distinguished by section separators, which solves the problem of joint encoding of multimodal features (language, pitch, rhythm) and significantly improves the information integrity and accuracy in the conversion process.

[0070] Through the bidirectional mapping of Chinese pinyin and Minnan pinyin, combined with pitch digital serialization, a nonlinear conversion from Mandarin phonemes to operatic phonemes is achieved. This is particularly suitable for scenarios where there are complex pronunciation differences between Minnan and Chinese.

[0071] Step S3: creating a Minnan opera voice change model based on the input module, the semantic conversion module, and the RVC network, and setting a loss function of the Minnan opera voice change model;

[0072] The input module is used to extract Chinese pinyin from the input Chinese audio and input the Chinese pinyin into the semantic conversion module; the semantic conversion module is used to map the Chinese pinyin to obtain a corresponding digital sequence through a semantic dictionary, and input the digital sequence into the RVC network; the RVC network is used to generate the Minnan opera audio corresponding to the Chinese audio based on the digital sequence and the semantic dictionary;

[0073] The RVC network can convert acoustic features (such as pitch, loudness, spectrum, etc.) into converted speech using waveform synthesis technology. Since the RVC network cannot directly recognize non-numeric characters in feature text, it is necessary to convert the semantic vectors in the feature text into digital sequences, that is, to perform the conversion using a semantic dictionary.

[0074] Step S4: training the Minnan opera voice change model using the data set and the loss function;

[0075] Step S5: deploying the trained Minnan opera voice change model, and performing Minnan opera voice change operations using the deployed Minnan opera voice change model.

[0076] By constructing a dual-source aligned dataset and a multi-dimensional annotation system, combined with a structured semantic dictionary, accurate mapping of cross-language phonemes, pitch, and rhythm is achieved. The CNN-RNN-RVC collaborative network is used for hierarchical extraction of acoustic features and sequence modeling, and the model robustness is improved based on dynamic data enhancement and a three-stage verification mechanism. Through containerized deployment and an end-to-end preprocessing pipeline, real-time and efficient voice changing is supported. Its modular design can be expanded to multiple dialects and opera types, and a composite loss function is used to optimize the consistency of timbre, pitch, and duration. Ultimately, on the basis of solving the lack of artistic expression of traditional voice changing technology, high-fidelity conversion of ordinary speech into Minnan opera singing is achieved.

[0077] The step S1 is specifically as follows:

[0078] Obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, performing noise reduction on each of the historical Chinese audio and historical Minnan opera audio, segmenting each of the historical Chinese audio and historical Minnan opera audio based on rhythm or beat to obtain a plurality of Chinese sub-audios and opera sub-audios, and associating each of the Chinese sub-audios with the opera sub-audios to complete preprocessing of each of the historical Chinese audio and historical Minnan opera audio;

[0079] Each of the Chinese sub-audio sounds is annotated with Chinese pinyin; each of the opera sub-audio sounds is annotated with lyrics (such as a, e, o), duration, and pitch (such as A1, A#1); the lyrics are in Minnan pinyin and separated by preset measure delimiters; the duration is used to identify the duration of each Minnan pinyin in the lyrics in seconds; and the pitch is used to identify the melody pitch of each Minnan pinyin in the lyrics;

[0080] A data set is constructed based on the labeled Chinese sub-audios and opera sub-audios.

[0081] By collecting historical Chinese audio and corresponding historical Minnan opera audio, and performing associated segmentation, we constructed a high-quality aligned dataset. This data pairing solves the problem of ambiguous data mapping in traditional speech conversion and ensures that the model learns accurate cross-language phoneme correspondences.

[0082] By annotating the opera audio with Minnan dialect pinyin, duration, and pitch, the unique melodic elements (pitch) and duration (rhythm) of opera were incorporated into the training data for the first time, enabling the model to capture the rhythmic characteristics of opera singing, breaking through the limitations of traditional voice changing technology that only focuses on timbre conversion.

[0083] In view of the complex tones and unique accents of Minnan opera, the pitch mapping and time value marking in the semantic dictionary are used to retain the ups and downs of opera singing, filling the gap in the application of traditional voice changing technology in the field of intangible cultural heritage inheritance.

[0084] By combining noise reduction technology with rhythm-based segmentation methods in the data preprocessing stage, we can not only eliminate background interference and improve speech purity, but also ensure the temporal consistency of Chinese and opera audio through beat alignment. This dual processing significantly enhances the model's ability to capture key acoustic features (such as pitch and resonance peaks) and avoids mapping deviations caused by noise pollution; the rhythm-based segmentation strategy (rather than fixed-duration cutting) is more in line with the ups and downs of opera singing, and is especially suitable for long-tone processing such as dragging and throwing, enabling the model to learn the unique elastic rhythm pattern of opera and enhance the artistic expression of the generated audio.

[0085] In step S3, the input module is constructed based on a convolutional neural network and adopts a CTC loss function;

[0086] The semantic conversion module is constructed based on a recurrent neural network.

[0087] By adopting a hierarchical structure of CNN (input module) + RNN (semantic conversion module) + RVC network, CNN efficiently extracts audio spectrum features, RNN handles sequence mapping tasks, and RVC network integrates semantic dictionaries to generate target audio. Each module has a clear division of labor and collaborative optimization, taking into account both feature extraction and sequence modeling capabilities.

[0088] The step S4 is specifically as follows:

[0089] Divide the dataset into a training set, a validation set, and a test set based on a ratio of 8:1:1, and train the Minnan opera voice change model using the training set until the loss value of the loss function is less than a preset loss threshold; in specific implementation, 1000 epochs can also be directly trained;

[0090] The trained Minnan Opera voice change model is verified using the verification set to determine whether the voice change accuracy is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training. If so, the verification passes, and:

[0091] The verified Minnan opera voice change model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test passes and the training ends.

[0092] During the training process, if the accuracy of the validation set or the confidence of the test set does not meet the standards, the training set will be automatically expanded. This dynamic feedback mechanism effectively solves the overfitting problem under small sample data and improves the model's generalization ability.

[0093] Through phased verification of training sets, validation sets, and test sets, the model is ensured to achieve a balance in training adequacy, parameter adjustment rationality, and final performance reliability, which is superior to traditional evaluation methods that only rely on a single test set.

[0094] The step S5 is specifically as follows:

[0095] The trained Minnan opera voice changing model is deployed to a server through containerization technology, and an API interface for calling the Minnan opera voice changing model is set to obtain real-time Chinese audio. After pre-processing the real-time Chinese audio, the Minnan opera voice changing model is input through the API interface to obtain the Minnan opera audio corresponding to the real-time Chinese audio, so as to perform the Minnan opera voice changing operation.

[0096] By deploying models using containerization technology, we achieve rapid service encapsulation and resource isolation, support high-concurrency real-time reasoning, and meet the low-latency requirements of opera voice changing in scenarios such as mobile terminals and performance systems.

[0097] By forming a complete closed loop from real-time audio input, preprocessing to API interface calls, users do not need to manually intervene in complex sound feature processing steps, which greatly reduces the usage threshold and improves the universality of the technology.

[0098] A preferred embodiment of the Minnan Opera voice changing system based on the RVC network of the present invention includes the following modules:

[0099] A data set construction module is used to obtain a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, and to construct a data set after preprocessing and annotating the historical Chinese audio and historical Minnan opera audio;

[0100] A semantic dictionary creation module is used to create a semantic dictionary for storing the correspondence between Minnan Pinyin, Chinese Pinyin and numerical sequences, and the correspondence between pitches and numerical sequences; each of the correspondences is separated by a preset section separator; the section separator is sp; for example, put is Minnan Pinyin, representing "no", and two uts represent a singing of two consecutive tones; for example, pitch C0 corresponds to 1, the section separator sp corresponds to 127, and the Minnan Pinyin i corresponds to 131;

[0101] Through the semantic dictionary, Chinese pinyin, Minnan pinyin, digital sequences and pitch mapping are integrated, and different elements are distinguished by section separators, which solves the problem of joint encoding of multimodal features (language, pitch, rhythm) and significantly improves the information integrity and accuracy in the conversion process.

[0102] Through the bidirectional mapping of Chinese pinyin and Minnan pinyin, combined with pitch digital serialization, a nonlinear conversion from Mandarin phonemes to operatic phonemes is achieved. This is particularly suitable for scenarios where there are complex pronunciation differences between Minnan and Chinese.

[0103] A Minnan opera voice change model creation module is used to create a Minnan opera voice change model based on the input module, the semantic conversion module and the RVC network, and set the loss function of the Minnan opera voice change model;

[0104] The input module is used to extract Chinese pinyin from the input Chinese audio and input the Chinese pinyin into the semantic conversion module; the semantic conversion module is used to map the Chinese pinyin to obtain a corresponding digital sequence through a semantic dictionary, and input the digital sequence into the RVC network; the RVC network is used to generate the Minnan opera audio corresponding to the Chinese audio based on the digital sequence and the semantic dictionary;

[0105] The RVC network can convert acoustic features (such as pitch, loudness, spectrum, etc.) into converted speech using waveform synthesis technology. Since the RVC network cannot directly recognize non-numeric characters in feature text, it is necessary to convert the semantic vectors in the feature text into digital sequences, that is, to perform the conversion using a semantic dictionary.

[0106] A Minnan opera voice change model training module, used to train the Minnan opera voice change model using the data set and loss function;

[0107] The Minnan opera voice changing module is used to deploy the trained Minnan opera voice changing model and perform Minnan opera voice changing operations through the deployed Minnan opera voice changing model.

[0108] By constructing a dual-source aligned dataset and a multi-dimensional annotation system, combined with a structured semantic dictionary, accurate mapping of cross-language phonemes, pitch, and rhythm is achieved. The CNN-RNN-RVC collaborative network is used for hierarchical extraction of acoustic features and sequence modeling, and the model robustness is improved based on dynamic data enhancement and a three-stage verification mechanism. Through containerized deployment and an end-to-end preprocessing pipeline, real-time and efficient voice changing is supported. Its modular design can be expanded to multiple dialects and opera types, and a composite loss function is used to optimize the consistency of timbre, pitch, and duration. Ultimately, on the basis of solving the lack of artistic expression of traditional voice changing technology, high-fidelity conversion of ordinary speech into Minnan opera singing is achieved.

[0109] The dataset construction module is specifically used for:

[0110] Obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, performing noise reduction on each of the historical Chinese audio and historical Minnan opera audio, segmenting each of the historical Chinese audio and historical Minnan opera audio based on rhythm or beat to obtain a plurality of Chinese sub-audios and opera sub-audios, and associating each of the Chinese sub-audios with the opera sub-audios to complete preprocessing of each of the historical Chinese audio and historical Minnan opera audio;

[0111] Each of the Chinese sub-audio sounds is annotated with Chinese pinyin; each of the opera sub-audio sounds is annotated with lyrics (such as a, e, o), duration, and pitch (such as A1, A#1); the lyrics are in Minnan pinyin and separated by preset measure delimiters; the duration is used to identify the duration of each Minnan pinyin in the lyrics in seconds; and the pitch is used to identify the melody pitch of each Minnan pinyin in the lyrics;

[0112] A data set is constructed based on the labeled Chinese sub-audios and opera sub-audios.

[0113] By collecting historical Chinese audio and corresponding historical Minnan opera audio, and performing associated segmentation, we constructed a high-quality aligned dataset. This data pairing solves the problem of ambiguous data mapping in traditional speech conversion and ensures that the model learns accurate cross-language phoneme correspondences.

[0114] By annotating the opera audio with Minnan dialect pinyin, duration, and pitch, the unique melodic elements (pitch) and duration (rhythm) of opera were incorporated into the training data for the first time, enabling the model to capture the rhythmic characteristics of opera singing, breaking through the limitations of traditional voice changing technology that only focuses on timbre conversion.

[0115] In view of the complex tones and unique accents of Minnan opera, the pitch mapping and time value marking in the semantic dictionary are used to retain the ups and downs of opera singing, filling the gap in the application of traditional voice changing technology in the field of intangible cultural heritage inheritance.

[0116] By combining noise reduction technology with rhythm-based segmentation methods in the data preprocessing stage, we can not only eliminate background interference and improve speech purity, but also ensure the temporal consistency of Chinese and opera audio through beat alignment. This dual processing significantly enhances the model's ability to capture key acoustic features (such as pitch and resonance peaks) and avoids mapping deviations caused by noise pollution; the rhythm-based segmentation strategy (rather than fixed-duration cutting) is more in line with the ups and downs of opera singing, and is especially suitable for long-tone processing such as dragging and throwing, enabling the model to learn the unique elastic rhythm pattern of opera and enhance the artistic expression of the generated audio.

[0117] In the Minnan opera voice change model creation module, the input module is constructed based on a convolutional neural network and adopts a CTC loss function;

[0118] The semantic conversion module is constructed based on a recurrent neural network.

[0119] By adopting a hierarchical structure of CNN (input module) + RNN (semantic conversion module) + RVC network, CNN efficiently extracts audio spectrum features, RNN handles sequence mapping tasks, and RVC network integrates semantic dictionaries to generate target audio. Each module has a clear division of labor and collaborative optimization, taking into account both feature extraction and sequence modeling capabilities.

[0120] The Minnan Opera voice change model training module is specifically used for:

[0121] Divide the dataset into a training set, a validation set, and a test set based on a ratio of 8:1:1, and train the Minnan opera voice change model using the training set until the loss value of the loss function is less than a preset loss threshold; in specific implementation, 1000 epochs can also be directly trained;

[0122] The trained Minnan Opera voice change model is verified using the verification set to determine whether the voice change accuracy is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training. If so, the verification passes, and:

[0123] The verified Minnan opera voice change model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test passes and the training ends.

[0124] During the training process, if the accuracy of the validation set or the confidence of the test set does not meet the standards, the training set will be automatically expanded. This dynamic feedback mechanism effectively solves the overfitting problem under small sample data and improves the model's generalization ability.

[0125] Through phased verification of training sets, validation sets, and test sets, the model is ensured to achieve a balance in training adequacy, parameter adjustment rationality, and final performance reliability, which is superior to traditional evaluation methods that only rely on a single test set.

[0126] The Minnan opera voice changing module is specifically used for:

[0127] The trained Minnan opera voice changing model is deployed to a server through containerization technology, and an API interface for calling the Minnan opera voice changing model is set to obtain real-time Chinese audio. After pre-processing the real-time Chinese audio, the Minnan opera voice changing model is input through the API interface to obtain the Minnan opera audio corresponding to the real-time Chinese audio, so as to perform the Minnan opera voice changing operation.

[0128] By deploying models using containerization technology, we achieve rapid service encapsulation and resource isolation, support high-concurrency real-time reasoning, and meet the low-latency requirements of opera voice changing in scenarios such as mobile terminals and performance systems.

[0129] By forming a complete closed loop from real-time audio input, preprocessing to API interface calls, users do not need to manually intervene in complex sound feature processing steps, which greatly reduces the usage threshold and improves the universality of the technology.

[0130] In summary, the advantages of the present invention are:

[0131] 1. By obtaining a large amount of historical Chinese audio and the historical Minnan opera audio corresponding to the historical Chinese audio, each historical Chinese audio and historical Minnan opera audio is preprocessed and annotated to construct a data set; then a semantic dictionary is created to store Minnan pinyin, the correspondence between Chinese pinyin and digital sequences, and the correspondence between pitch and digital sequences; a Minnan opera voice change model is created based on the input module, the semantic conversion module and the RVC network, and the loss function of the Minnan opera voice change model is set; the input module is used to extract Chinese pinyin from the input Chinese audio; the semantic conversion module is used to map the Chinese pinyin to the corresponding digital sequence through the semantic dictionary; RV The C network is used to generate Minnan opera audio corresponding to Chinese audio based on a digital sequence and a semantic dictionary; then, the Minnan opera voice change model is trained using a data set and a loss function, the trained Minnan opera voice change model is deployed, and the Minnan opera voice change operation is performed using the deployed Minnan opera voice change model; that is, the Minnan opera voice change model is trained using a data set constructed from historical Chinese audio and historical Minnan opera audio. The data set is labeled based on the characteristics of the Minnan dialect. During the training process of the Minnan opera voice change model, the semantic dictionary is combined to effectively identify and convert the semantics of the audio, thereby greatly improving the quality of Minnan opera voice change.

[0132] 2. By collecting historical Chinese audio and corresponding historical Minnan opera audio and performing associated segmentation, we constructed a high-quality aligned dataset. This data pairing solves the problem of ambiguous data mapping in traditional speech conversion and ensures that the model learns accurate cross-language phoneme correspondences.

[0133] 3. By annotating the opera audio with Minnan dialect pinyin, duration, and pitch, the unique melodic elements (pitch) and duration (rhythm) of opera are incorporated into the training data for the first time, enabling the model to capture the rhythmic characteristics of opera singing, breaking through the limitation of traditional voice changing technology that only focuses on timbre conversion.

[0134] 4. Through the semantic dictionary, Chinese pinyin, Minnan pinyin, digital sequences and pitch mapping are integrated, and different elements are distinguished by section separators, which solves the problem of joint encoding of multimodal features (language, pitch, rhythm), and significantly improves the information integrity and accuracy during the conversion process.

[0135] 5. Through the bidirectional mapping of Chinese pinyin and Minnan pinyin, combined with pitch digital serialization, a nonlinear conversion from Mandarin phonemes to operatic phonemes is achieved. This is particularly suitable for scenarios where there are complex pronunciation differences between Minnan and Chinese.

[0136] 6. By adopting a hierarchical structure of CNN (input module) + RNN (semantic conversion module) + RVC network, CNN efficiently extracts audio spectrum features, RNN handles sequence mapping tasks, and the RVC network integrates the semantic dictionary to generate target audio. Each module has a clear division of labor and collaborative optimization, taking into account both feature extraction and sequence modeling capabilities.

[0137] 7. During the training process, if the accuracy of the validation set or the confidence of the test set does not meet the standards, the training set will be automatically expanded. This dynamic feedback mechanism effectively solves the overfitting problem under small sample data and improves the generalization ability of the model.

[0138] 8. Through phased verification of training sets, validation sets, and test sets, the model is ensured to achieve a balance in training adequacy, parameter adjustment rationality, and final performance reliability, which is superior to traditional evaluation methods that only rely on a single test set.

[0139] 9. Deploy models through containerization technology to achieve rapid service packaging and resource isolation, support high-concurrency real-time reasoning, and meet the low-latency requirements of opera voice changing in scenarios such as mobile terminals and performance systems.

[0140] 10. By forming a complete closed loop from real-time audio input, preprocessing to API interface calls, users do not need to manually intervene in complex sound feature processing steps, which greatly reduces the usage threshold and improves the universality of the technology.

[0141] 11. In view of the complex tones and unique intonations of Minnan opera, the pitch mapping and time value annotation in the semantic dictionary are used to preserve the ups and downs of opera singing, filling the gap in the application of traditional voice changing technology in the field of intangible cultural heritage.

[0142] 12. By combining noise reduction technology with rhythm-based segmentation methods in the data preprocessing stage, we can not only eliminate background interference and improve the purity of speech, but also ensure the temporal consistency of Chinese and opera audio through beat alignment. This dual processing significantly enhances the model's ability to capture key acoustic features (such as pitch and resonance peaks) and avoids mapping deviations caused by noise pollution; the rhythm-based segmentation strategy (rather than fixed-duration cutting) is more in line with the ups and downs of opera singing, and is especially suitable for long-tone processing such as dragging and throwing, enabling the model to learn the unique elastic rhythm pattern of opera and enhance the artistic expression of the generated audio.

[0143] 13. By constructing a dual-source aligned dataset and a multi-dimensional annotation system, combined with a structured semantic dictionary, accurate mapping of cross-language phonemes, pitch, and rhythm is achieved. A CNN-RNN-RVC collaborative network is used for hierarchical extraction of acoustic features and sequence modeling, and dynamic data enhancement and a three-stage verification mechanism are used to improve model robustness. Containerized deployment and an end-to-end preprocessing pipeline support real-time and efficient voice changing. Its modular design is scalable to multiple dialects and opera types, and a composite loss function is used to optimize the consistency of timbre, pitch, and duration. Ultimately, while addressing the lack of artistic expression of traditional voice changing technology, high-fidelity conversion of ordinary speech into Minnan opera singing is achieved.

[0144] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for voice changing of Minnan opera based on RVC network, characterized by: The steps include: Step S1: obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, and constructing a data set after preprocessing and annotating the historical Chinese audio and historical Minnan opera audio; Step S2: creating a semantic dictionary for storing the correspondence between Minnan dialect pinyin, Chinese pinyin and number sequences, and the correspondence between pitch and number sequences; each of the correspondences is separated by a preset measure separator; Step S3: creating a Minnan opera voice change model based on the input module, the semantic conversion module, and the RVC network, and setting a loss function of the Minnan opera voice change model; The input module is used to extract Chinese pinyin from the input Chinese audio and input the Chinese pinyin into the semantic conversion module; the semantic conversion module is used to map the Chinese pinyin to obtain a corresponding digital sequence through a semantic dictionary, and input the digital sequence into the RVC network; the RVC network is used to generate the Minnan opera audio corresponding to the Chinese audio based on the digital sequence and the semantic dictionary; Step S4: training the Minnan opera voice change model using the data set and the loss function; Step S5: deploying the trained Minnan opera voice change model, and performing Minnan opera voice change operations using the deployed Minnan opera voice change model.

2. The method for voice changing of Minnan opera based on RVC network as claimed in claim 1, characterized in that: The step S1 is specifically as follows: Obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, performing noise reduction on each of the historical Chinese audio and historical Minnan opera audio, segmenting each of the historical Chinese audio and historical Minnan opera audio based on rhythm or beat to obtain a plurality of Chinese sub-audios and opera sub-audios, and associating each of the Chinese sub-audios with the opera sub-audios to complete preprocessing of each of the historical Chinese audio and historical Minnan opera audio; Each of the Chinese sub-audio sounds is annotated with Chinese pinyin; each of the opera sub-audio sounds is annotated with lyrics, duration, and pitch; the lyrics are in Minnan pinyin and separated by preset measure delimiters; the duration is used to identify the duration of each Minnan pinyin in the lyrics in seconds; The pitch is used to mark the melody pitch of each Minnan pinyin in the lyrics; A data set is constructed based on the annotated Chinese sub-audios and opera sub-audios.

3. The method for voice changing of Minnan opera based on RVC network as claimed in claim 1, characterized in that: In step S3, the input module is constructed based on a convolutional neural network and adopts a CTC loss function; The semantic conversion module is constructed based on a recurrent neural network.

4. The method for voice changing of Minnan opera based on RVC network as claimed in claim 1, characterized in that: The step S4 is specifically as follows: Dividing the data set into a training set, a validation set, and a test set based on a ratio of 8:1:1, and training the Minnan opera voice change model using the training set until the loss value of the loss function is less than a preset loss threshold; The trained Minnan Opera voice change model is verified using the verification set to determine whether the voice change accuracy is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training. If so, the verification passes, and: The verified Minnan opera voice change model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test passes and the training ends.

5. The method for voice changing of Minnan opera based on RVC network as claimed in claim 1, characterized in that: The step S5 is specifically as follows: The trained Minnan opera voice changing model is deployed to a server through containerization technology, and an API interface for calling the Minnan opera voice changing model is set to obtain real-time Chinese audio. After pre-processing the real-time Chinese audio, the Minnan opera voice changing model is input through the API interface to obtain the Minnan opera audio corresponding to the real-time Chinese audio, so as to perform the Minnan opera voice changing operation.

6. A voice changing system for Minnan opera based on RVC network, characterized by: Includes the following modules: A data set construction module is used to obtain a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, and to construct a data set after preprocessing and annotating the historical Chinese audio and historical Minnan opera audio; A semantic dictionary creation module is used to create a semantic dictionary for storing the correspondence between Minnan Pinyin, Chinese Pinyin and number sequences, and the correspondence between pitch and number sequences; each of the correspondences is separated by a preset section separator; A Minnan opera voice change model creation module is used to create a Minnan opera voice change model based on the input module, the semantic conversion module and the RVC network, and set the loss function of the Minnan opera voice change model; The input module is used to extract Chinese pinyin from the input Chinese audio and input the Chinese pinyin into the semantic conversion module; the semantic conversion module is used to map the Chinese pinyin to obtain a corresponding digital sequence through a semantic dictionary, and input the digital sequence into the RVC network; the RVC network is used to generate the Minnan opera audio corresponding to the Chinese audio based on the digital sequence and the semantic dictionary; A Minnan opera voice change model training module, used to train the Minnan opera voice change model using the data set and loss function; The Minnan opera voice changing module is used to deploy the trained Minnan opera voice changing model and perform Minnan opera voice changing operations through the deployed Minnan opera voice changing model.

7. The RVC network-based voice changing system for Minnan opera according to claim 6, characterized in that: The dataset construction module is specifically used for: Obtaining a large amount of historical Chinese audio and historical Minnan opera audio corresponding to the historical Chinese audio, performing noise reduction on each of the historical Chinese audio and historical Minnan opera audio, segmenting each of the historical Chinese audio and historical Minnan opera audio based on rhythm or beat to obtain a plurality of Chinese sub-audios and opera sub-audios, and associating each of the Chinese sub-audios with the opera sub-audios to complete preprocessing of each of the historical Chinese audio and historical Minnan opera audio; Each of the Chinese sub-audio sounds is annotated with Chinese pinyin; each of the opera sub-audio sounds is annotated with lyrics, duration, and pitch; the lyrics are in Minnan pinyin and separated by preset measure delimiters; the duration is used to identify the duration of each Minnan pinyin in the lyrics in seconds; The pitch is used to mark the melody pitch of each Minnan pinyin in the lyrics; A data set is constructed based on the labeled Chinese sub-audios and opera sub-audios.

8. The RVC network-based voice changing system for Minnan opera according to claim 6, characterized in that: In the Minnan opera voice change model creation module, the input module is constructed based on a convolutional neural network and adopts a CTC loss function; The semantic conversion module is constructed based on a recurrent neural network.

9. The RVC network-based voice changing system for Minnan opera according to claim 6, characterized in that: The Minnan Opera voice change model training module is specifically used for: Dividing the data set into a training set, a validation set, and a test set based on a ratio of 8:1:1, and training the Minnan opera voice change model using the training set until the loss value of the loss function is less than a preset loss threshold; The trained Minnan Opera voice change model is verified using the verification set to determine whether the voice change accuracy is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training. If so, the verification passes, and: The verified Minnan opera voice change model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test passes and the training ends.

10. The RVC network-based voice changing system for Minnan opera according to claim 6, characterized in that: The Minnan opera voice changing module is specifically used for: The trained Minnan opera voice changing model is deployed to a server through containerization technology, and an API interface for calling the Minnan opera voice changing model is set to obtain real-time Chinese audio. After pre-processing the real-time Chinese audio, the Minnan opera voice changing model is input through the API interface to obtain the Minnan opera audio corresponding to the real-time Chinese audio, so as to perform the Minnan opera voice changing operation.

Images