Sound customization method, device, equipment and storage medium
Through the improved VITS model, the fusion vector module and iSTFT decoder module are used to integrate phoneme sequences, pronunciation structures and word segmentation information, solving the naturalness and low efficiency of Chinese pronunciation synthesis, and achieving more efficient emotional information integration and personalized speech synthesis.
Patent Information
- Application Number
- CN202410280928.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-03-12
AI Technical Summary
Chinese pronunciation synthesis faces complex pronunciation rules, polyphonic characters, erhua pronunciation and tone change phenomena, resulting in low naturalness and efficiency, and lack of rich integration of emotional information, making it difficult to meet personalized needs.
The improved VITS model is adopted, combining prosthetic enhanced fusion vector module and iSTFT-based decoder module to integrate phoneme sequences, prosthetic structures and word segmentation information to improve the naturalness and efficiency of speech synthesis.
It improves the naturalness and efficiency of Chinese pronunciation synthesis, can better integrate rich emotional information, meet personalized needs, and improve interaction quality and sense of reality.
Smart Images

Figure CN118053416B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of sound processing technology, and in particular to a sound customization method, device, equipment and storage medium. Background Art
[0002] At present, in the field of digital broadcasting, voice customization of digital broadcasters has become a key technical challenge and research hotspot. With the advancement of digital broadcasting technology and the rapid growth of market demand, users not only expect digital broadcasters to be visually realistic, but also hope that their voices can provide a rich and personalized auditory experience. This demand has prompted researchers to develop advanced speech systems that can synthesize specific emotions, intonations, and personality styles. In the application of digital broadcasters, the research on voice customization not only focuses on making the synthesized speech more natural and realistic, but also focuses on how to adjust the sound attributes according to different broadcasting scenarios and the personalized needs of users. This research direction is not only crucial to improving the interactive quality and realism of digital broadcasters, but also has a significant impact on promoting their application potential in fields such as news, education, entertainment, and customer service.
[0003] Speech synthesis technology is a key technology for voice customization, but Chinese speech synthesis faces several challenges. First, the pronunciation rules of Chinese are much more complex than those of English. As a four-tone language, the pronunciation of Chinese depends on the combination of initials and finals and tones, and there are many polyphones, erhua sounds, and tone changes. These pronunciation characteristics are closely related to the context and rhythmic structure of the sentence, requiring the speech synthesis model to effectively extract context and semantic information. Secondly, the emotional color in Chinese expression is richer and more varied. The same sentence can convey different meanings according to different tones and emotional colors, which requires Chinese speech synthesis to integrate richer emotional information to improve the naturalness and expressiveness of pronunciation. In addition, although a large number of high-quality data sets have been accumulated in the field of English speech synthesis, Chinese speech data sets are relatively small, smaller in scale, and face the challenge of dialect diversity. These factors increase the difficulty and complexity of modeling Chinese speech synthesis models.
[0004] Therefore, there is an urgent need for a sound customization method that can effectively improve the naturalness and synthesis efficiency of Chinese speech synthesis. Summary of the invention
[0005] The main purpose of the present invention is to provide a sound customization method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the naturalness and synthesis efficiency of Chinese speech synthesis.
[0006] To achieve the above object, the present invention provides a sound customization method, which comprises the following steps:
[0007] Acquire text information input by a user, and pre-process the text information to generate a phoneme sequence, a rhythmic structure, and word segmentation information;
[0008] The phoneme sequence, the prosodic structure and the word segmentation information are input into an improved VITS model to generate target audio. The improved VITS model includes a prosodic enhanced fusion vector module and an iSTFT-based decoder module. The prosodic enhanced fusion vector module is used to integrate the phoneme sequence, the prosodic structure and the word segmentation information.
[0009] Optionally, the word segmentation information includes a word segmentation text and a word segmentation label, and the step of preprocessing the text information to generate a phoneme sequence, a prosodic structure and word segmentation information includes:
[0010] Perform sentence processing on the text information to obtain sentence units;
[0011] Performing word segmentation processing on each of the sentence units to obtain an initial word segmentation text, and performing part-of-speech tagging on each of the initial word segmentation texts to generate an initial word segmentation label;
[0012] Performing feature symbol conversion and standardization processing on each of the initial segmentation texts and each of the initial segmentation labels to obtain segmentation texts and segmentation labels;
[0013] Predicting the prosodic structure of the text information by a prosodic structure prediction model according to the segmented text and the segmented label;
[0014] Based on the word segmentation text and the word segmentation label, the text information is converted into a phoneme sequence through a pinyin-phoneme converter.
[0015] Optionally, before the step of obtaining text information input by the user and preprocessing the text information to generate a phoneme sequence, a prosodic structure and word segmentation information, the step further includes:
[0016] Constructing a VITS model, and improving the VITS model to obtain an initial VITS model;
[0017] Training the initial VITS model to obtain an improved VITS model;
[0018] Accordingly, the steps of constructing a VITS model and improving the VITS model to obtain an initial VITS model include:
[0019] Constructing a VITS model, and adding a prosody-enhanced fusion vector module on the basis of the VITS model, wherein the prosody-enhanced fusion vector module is used to input a phoneme sequence, a prosody structure, and word segmentation information into a text encoder to generate a fusion vector;
[0020] After adding the rhythmically enhanced fusion vector module, the iSTFT model is used to replace some repeated network layers in HiFi-GAN, and a multi-subband parallel generation strategy is introduced to replace the decoder module of the VITS model with a decoder module based on iSTFT to obtain the initial VITS model.
[0021] Optionally, before the step of predicting the prosodic structure of the text information by a prosodic structure prediction model according to the segmented text and the segmented label, the step further includes:
[0022] Acquire corpus data, wherein the corpus data includes prosodic features, phoneme sequences, and audio data, wherein the prosodic features include duration, pitch, and intensity;
[0023] Using the SpanPSP model as an initial prosodic structure prediction model, and inputting the corpus data into the initial prosodic structure prediction model for model training to obtain a training result;
[0024] Based on the training results, the initial prosodic structure prediction model is optimized to obtain a prosodic structure prediction model.
[0025] Optionally, after the step of converting and normalizing the initial segmentation texts and initial segmentation labels to obtain the segmentation texts and segmentation labels, the step further includes:
[0026] According to the segmented text and the segmented label, the polyphone problem is processed by a polyphone disambiguation model to ensure correct pinyin conversion.
[0027] Optionally, before the step of segmenting the text information into sentences to obtain sentence units, the step further includes:
[0028] Based on the text information, determining whether to perform sentence processing on the text information;
[0029] If the text information does not need to be processed into sentences, the text information is processed into words to obtain initial word segmentation texts, and each initial word segmentation text is tagged with a part of speech to generate an initial word segmentation tag;
[0030] Correspondingly, if it is necessary to perform sentence processing on the text information, the step of performing sentence processing on the text information to obtain sentence units is performed.
[0031] Optionally, after the step of inputting the phoneme sequence, the prosodic structure and the word segmentation information into the improved VITS model to generate the target audio, the method further includes:
[0032] Based on the target audio, generating a 3D facial animation through a Faceformer model;
[0033] According to user operation instructions, the playback display of the target audio and the visual display of the 3D face animation are controlled.
[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a sound customization device, which includes:
[0035] An information acquisition module is used to acquire text information input by a user, and pre-process the text information to generate a phoneme sequence, a rhythmic structure, and word segmentation information;
[0036] An audio generation module is used to input the phoneme sequence, the prosodic structure and the word segmentation information into an improved VITS model to generate target audio. The improved VITS model includes a prosodic enhanced fusion vector module and an iSTFT-based decoder module. The prosodic enhanced fusion vector module is used to integrate the phoneme sequence, the prosodic structure and the word segmentation information.
[0037] In addition, to achieve the above-mentioned purpose, the present invention also proposes a sound customization device, which includes: a memory, a processor, and a sound customization program stored in the memory and executable on the processor, wherein the sound customization program is configured to implement the steps of the sound customization method described above.
[0038] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a sound customization program is stored, and when the sound customization program is executed by a processor, the steps of the sound customization method described above are implemented.
[0039] The present invention obtains text information input by a user and pre-processes the text information to generate a phoneme sequence, a rhythmic structure and word segmentation information; the phoneme sequence, the rhythmic structure and the word segmentation information are input into an improved VITS model to generate a target audio, wherein the improved VITS model includes a rhythmically enhanced fusion vector module and an iSTFT-based decoder module, wherein the rhythmically enhanced fusion vector module is used to integrate the phoneme sequence, the rhythmic structure and the word segmentation information. Since the present invention integrates the phoneme sequence, the rhythmic structure and the word segmentation information through the rhythmically enhanced fusion vector module and introduces the iSTFT-based decoder module, compared with the prior art, the present invention not only improves the naturalness of Chinese speech synthesis, but also improves the efficiency of Chinese speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a structural schematic diagram of a sound customization device for a hardware operating environment involved in an embodiment of the present invention;
[0041] Figure 2 It is a flowchart of a first embodiment of the sound customization method of the present invention;
[0042] Figure 3 A schematic diagram of the processing flow of a fusion vector module for prosody enhancement in the sound customization method of the present invention;
[0043] Figure 4 It is a structural schematic diagram of a decoder module based on iSTFT in the sound customization method of the present invention;
[0044] Figure 5 A schematic diagram of a flow chart of a second embodiment of a sound customization method of the present invention;
[0045] Figure 6 A schematic diagram of rhythmic symbols in the sound customization method of the present invention;
[0046] Figure 7 A schematic diagram of a flow chart of a third embodiment of a sound customization method of the present invention;
[0047] Figure 8 It is a schematic diagram of the detailed flow of the system implementation in the sound customization method of the present invention;
[0048] Fig. 9 A user interaction interface design diagram in the sound customization method of the present invention;
[0049] Fig.10 It is a structural block diagram of the first embodiment of the sound customization device of the present invention.
[0050] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0052] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of a sound customization device for a hardware operating environment according to an embodiment of the present invention.
[0053] like Figure 1As shown, the sound customization device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0054] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation of the sound customization device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0055] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a sound customization program.
[0056] exist Figure 1 In the sound customization device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the sound customization device of the present invention can be set in the sound customization device, and the sound customization device calls the sound customization program stored in the memory 1005 through the processor 1001, and executes the sound customization method provided by the embodiment of the present invention.
[0057] The embodiment of the present invention provides a sound customization method, referring to Figure 2 , Figure 2 FIG. 1 is a flow chart of a first embodiment of a sound customization method according to the present invention.
[0058] In this embodiment, the sound customization method includes the following steps:
[0059] Step S10: Acquire text information input by the user, and pre-process the text information to generate a phoneme sequence, a prosodic structure, and word segmentation information.
[0060] It should be noted that the execution subject of this embodiment can be a computer server device with data processing, network communication and program running functions, such as a server, a tablet computer, a personal computer, an iPad, etc., or an electronic device that can realize the above functions, a sound customization device, etc. The following takes the sound customization device as an example, and the sound customization device includes a sound customization system, and the sound customization system can be applied in the scene of speech synthesis. Here, the above-mentioned sound customization system (hereinafter referred to as the system) is used to specifically describe this embodiment and the following embodiments.
[0061] It should be explained that the user interaction interface in the above system includes a text input panel, where the user can input the text that needs to be synthesized as the input information for sound customization, and then the input information is used as the text information input by the user, and the text information is preprocessed to generate phoneme sequences, rhythmic structures and word segmentation information.
[0062] It should be noted that the above-mentioned phoneme sequence can be a pinyin sequence converted from text information by a pinyin-phoneme converter. The prosodic structure can be the stress and pause between words, phrases and short sentences. Word segmentation is the operation of dividing a continuous text into independent words. The word segmentation information can include word segmentation text and word segmentation label. The word segmentation label can be a label generated by annotating the part of speech of the word segmentation text.
[0063] Step S20: Input the phoneme sequence, the prosodic structure and the word segmentation information into an improved VITS model to generate target audio, wherein the improved VITS model includes a prosodic enhanced fusion vector module and an iSTFT-based decoder module, wherein the prosodic enhanced fusion vector module is used to integrate the phoneme sequence, the prosodic structure and the word segmentation information.
[0064] It should be understood that before step S10: obtaining text information input by the user and preprocessing the text information to generate a phoneme sequence, a prosodic structure and word segmentation information, the following steps are also included:
[0065] Step S01: construct a VITS model, and improve the VITS model to obtain an initial VITS model.
[0066] Step S02: training the initial VITS model to obtain an improved VITS model.
[0067] In a specific implementation, a VITS model is constructed, and a rhythmically enhanced fusion vector module is added to the VITS model, wherein the rhythmically enhanced fusion vector module is used to input a phoneme sequence, a rhythmic structure, and word segmentation information into a text encoder to generate a fusion vector; after adding the rhythmically enhanced fusion vector module, the iSTFT model is used to replace some repeated network layers in HiFi-GAN, and a multi-subband parallel generation strategy is introduced to replace the decoder module of the VITS model with a decoder module based on iSTFT, thereby obtaining an initial VITS model.
[0068] It should be noted that the decoder module based on iSTFT is designed to solve the problem of the decoder in the VITS model as a time-consuming bottleneck. The decoder in the VITS model is based on the HiFi-GAN vocoder and uses an overlapping convolutional network to sample input acoustic features. In order to optimize this process, this embodiment draws on the design of iSTFTNet, and effectively reduces the amount of calculation and accelerates the speech synthesis process by applying the iSTFT model to replace some repeated network layers in HiFi-GAN and introducing a multi-subband parallel generation strategy.
[0069] It is understandable that human language is not just a simple series of phonemes, but also contains multi-dimensional rhythmic features such as emotion, emphasis, and rhythm. At the same time, word segmentation information can help the model better understand the structure and semantics of the language, thereby producing more natural and fluent speech output. The phoneme sequence, rhythmic structure, and word segmentation information are then integrated through the rhythm-enhanced fusion vector module.
[0070] For example, refer to Figure 3 , Figure 3 The figure is a processing flow diagram of the rhythm-enhanced fusion vector module in the sound customization method of the present invention. After obtaining text information, the text information is pre-processed to generate phonemes (phoneme sequences), rhythms (rhythmic structures) and segmentation words (segmentation word information), and the phoneme sequences, rhythmic structures and segmentation word information are integrated by the text encoder in the rhythm-enhanced fusion vector module to generate a fusion vector.
[0071] Further, refer to Figure 4 , Figure 4The schematic diagram of the structure of the decoder module based on iSTFT in the sound customization method of the present invention. The decoder module based on iSTFT first upsamples the latent variable z in the residual block, and the upsampled z, that is, the output convolution is further mapped to the amplitude and phase variables of each sub-band signal. Then, the amplitude and phase of these sub-bands are converted into corresponding sub-band signals using iSTFT. Each sub-band signal is upsampled to match the sampling rate of the original signal, and the specific operation can be to insert zeros between samples. Finally, these sub-band signals can be synthesized into full-band waveform signals through a trainable synthesis filter, such as pseudo-QMF.
[0072] This embodiment constructs a VITS model, and improves the VITS model to obtain an initial VITS model; trains the initial VITS model to obtain an improved VITS model; obtains text information input by a user, and pre-processes the text information to generate a phoneme sequence, a prosodic structure, and word segmentation information; inputs the phoneme sequence, the prosodic structure, and the word segmentation information into the improved VITS model to generate a target audio, wherein the improved VITS model includes a prosodic enhanced fusion vector module and an iSTFT-based decoder module, wherein the prosodic enhanced fusion vector module is used to integrate the phoneme sequence, the prosodic structure, and the word segmentation information. Since this embodiment integrates the phoneme sequence, the prosodic structure, and the word segmentation information through the prosodic enhanced fusion vector module, and introduces the iSTFT-based decoder module, compared to the prior art, this embodiment not only improves the naturalness of Chinese speech synthesis, but also improves the efficiency of Chinese speech synthesis.
[0073] refer to Figure 5 , Figure 5 FIG. 4 is a flow chart of a second embodiment of the sound customization method of the present invention.
[0074] Based on the above first embodiment, in this embodiment, step S10 includes:
[0075] Step S101: Segment the text information to obtain sentence units.
[0076] It should be understood that the above sentence segmentation process may be to cut the text information into a plurality of small sentence units.
[0077] It should be explained that, before step S101, the following steps are also included:
[0078] Based on the text information, determining whether to perform sentence processing on the text information;
[0079] If the text information does not need to be processed into sentences, the text information is processed into words to obtain initial word segmentation texts, and each initial word segmentation text is tagged with a part of speech to generate an initial word segmentation tag;
[0080] Correspondingly, if it is necessary to perform sentence processing on the text information, the step of performing sentence processing on the text information to obtain sentence units is performed.
[0081] Step S102: performing word segmentation processing on each of the sentence units to obtain initial word segmentation texts, and performing part-of-speech tagging on each of the initial word segmentation texts to generate initial word segmentation tags.
[0082] In a specific implementation, the HanLP model may be used to perform word segmentation processing on each sentence unit and to perform part-of-speech tagging on the word segmentation text, or the ALBert tagging model may be used to perform part-of-speech tagging on the word segmentation text, which is not limited in this embodiment.
[0083] It should be noted that the above-mentioned part-of-speech tagging can be to mark the part-of-speech type of the segmented text through specific characters, and the part-of-speech types include verb (v), noun (n), localizer (LC), pronoun (PN), demonstrative word (DT), quantifier (M), etc.
[0084] It should be explained that this embodiment uses the BMES four-digit sequence tagging method in the process of word segmentation of each sentence unit. "B" represents the beginning of a word, "M" represents the middle part of a word, "E" represents the end of a word, and "S" is used to represent a single word. This method allows accurate tagging according to the length of each word, thereby effectively identifying the lexical boundaries in Chinese sentences.
[0085] Step S103: performing feature symbol conversion and standardization processing on each of the initial segmentation texts and each of the initial segmentation labels to obtain segmentation texts and segmentation labels.
[0086] In a specific implementation, the text regularizer may be used to convert and standardize each initial word segmentation text and each initial word segmentation label to obtain the word segmentation text and the word segmentation label.
[0087] Step S104: predicting the prosodic structure of the text information through a prosodic structure prediction model according to the segmented text and the segmented label.
[0088] It should be noted that before step S104, the method also includes: acquiring corpus data, the corpus data including prosodic features, phoneme sequences and audio data, the prosodic features including duration, pitch and intensity; using the SpanPSP model as an initial prosodic structure prediction model, and inputting the corpus data into the initial prosodic structure prediction model for model training to obtain training results; and optimizing the initial prosodic structure prediction model based on the training results to obtain a prosodic structure prediction model.
[0089] In a specific implementation, in order to ensure the subsequent correct pinyin conversion, after step S104, it also includes: processing the polyphone problem through a polyphone disambiguation model according to the word segmentation text and the word segmentation label to ensure the correct pinyin conversion.
[0090] Understandably, prosody support is crucial in speech synthesis. From the perspective of text information, prosody is reflected in the boundaries of words and phrases; from the perspective of speech, prosody usually includes three main features: duration, pitch, and intensity. The ability to control these features is crucial for speech synthesis, because prosody not only connects text information and speech, but also helps to clarify the meaning of text information and guides features such as the pronunciation duration of syllables when synthesizing speech.
[0091] In this embodiment, the core task of the prosodic structure prediction model is to complete the prediction of the three-layer prosodic hierarchy tree. The three layers are prosodic words (PW), prosodic phrases (PPH) and intonational phrases (IPH). They are different in terms of stress and pause time. For example, Figure 6 , Figure 6 This is a schematic diagram of prosodic symbols in the sound customization method of the present invention. The prosodic symbols include: #0, #1, #2, #3 and #4, and the prosodic symbols represent: character boundary, word boundary (word segmentation result), phrase boundary (between 1 and 3), short sentence boundary (with auditory pause) and long sentence boundary (long pause, usually at the end of the sentence).
[0092] In the specific implementation, in order to more accurately simulate natural speech, the prosodic marker #4 can be added at the end of the sentence. For the prediction of prosodic structure, the deep learning-based method currently performs best. The SpanPSP model proposed at ICASSP2022 and its pre-trained parameter file can be used. In the project, the SpanPSP model demonstrated excellent performance and effectively predicted the prosodic structure in text information, thus providing support for generating more natural and fluent speech synthesis.
[0093] Step S105: Based on the word segmentation text and the word segmentation label, the text information is converted into a phoneme sequence by a pinyin-phoneme converter.
[0094] For example, after the user enters text information in the "text input area", the long text is first cut into smaller sentence units through sentence segmentation for subsequent processing. Next, the text is segmented using ALBert-based technology, which is implemented using the HanLP model and can be annotated using the BMES four-bit sequence tagging method. For example, if the user enters "I am from Zhejiang", the segmentation result is "I / am / Zhejiang", and the text processed by the part-of-speech tagging algorithm is "I / S is / S Zhejiang / B Jiang / M people / E". After the segmentation and part-of-speech tagging are completed, the characters and their part-of-speech tag pairs are input into the text regularizer for conversion and standardization of special symbols. Subsequently, the polyphone disambiguation model is used to handle the polyphone problem, and finally, the pinyin-phoneme converter is used to convert it into a phoneme sequence, paving the way for subsequent speech synthesis.
[0095] In this embodiment, the text information is processed by sentence segmentation to obtain sentence units; the sentence units are processed by word segmentation to obtain initial word segmentation texts, and the initial word segmentation texts are tagged with parts of speech to generate initial word segmentation tags; the initial word segmentation texts and the initial word segmentation tags are converted and standardized with characteristic symbols to obtain word segmentation texts and word segmentation tags; the rhythmic structure of the text information is predicted by a rhythmic structure prediction model based on the word segmentation texts and the word segmentation tags; based on the word segmentation texts and the word segmentation tags, the text information is converted into a phoneme sequence by a pinyin-phoneme converter; the phoneme sequence, the rhythmic structure and the word segmentation information are input into the improved VITS model to generate target audio. Since the present embodiment obtains the phoneme sequence, rhythmic structure and word segmentation information by performing operations such as sentence segmentation, word segmentation and part of speech tagging, polyphone disambiguation and rhythmic structure prediction on the text information, and then performs speech synthesis based on the phoneme sequence, rhythmic structure and word segmentation information, compared with the prior art, the present embodiment effectively improves the naturalness and fluency of the synthesized speech.
[0096] refer to Figure 7 , Figure 7 FIG. 4 is a flow chart of a third embodiment of the sound customization method of the present invention.
[0097] Based on the above embodiments, in this embodiment, after step S20, the following steps are further included:
[0098] Step S30: Based on the target audio, generate a 3D facial animation through a Faceformer model.
[0099] Step S40: According to the user operation instruction, the playback display of the target audio and the visual display of the 3D face animation are controlled.
[0100] It should be noted that after generating the target audio, the 3D facial animation will be generated. Based on the target audio and the 3D face model, the system will call the Faceformer model to generate the 3D facial animation and use the pre-trained wav2vec2.0 weights to initialize the encoder. The system uses faceverse, lsfm, bfm 3DMM models to represent facial movements. And use Pytorch3D for visualization. After generating the 3D face animation, you can use the "3D Visualization Display Area" section to perform 3D visualization and audio playback. In addition, users can use the left and right buttons and the scroll wheel of the mouse to perform translation, rotation and scaling of the 3D posture.
[0101] In a specific implementation, for example, refer to Figure 8 , Figure 8 The figure is a detailed flow chart of the implementation of the system in the sound customization method of the present invention. After obtaining the text information input by the user, the text information is preprocessed to generate phonemes (phoneme sequences), rhythms (rhythmic structures) and tones (word segmentation information). Then, the target audio is generated through the speech generation model (the improved VITS model). Based on the target audio and the face model, the mesh sequence of the 3D face animation is generated through the Faceformer model, and finally a visual display is performed.
[0102] refer to Fig. 9 , Fig. 9 The user interaction interface design diagram in the sound customization method of the present invention, the user interaction interface input should include:
[0103] Select the file and click the "File" button to expand the drop-down menu. The user can choose the save location of the file and whether to read the previous data for display. Then a file browser will pop up to allow the user to specify the save or read location of the file.
[0104] Clicking the "Synthesis" button will expand the drop-down menu, where users can choose "Synthesize audio alone" or "Generate 3D face animation". After clicking, the corresponding synthesis core algorithm will be displayed. Generating 3D face animation can only be performed after synthesizing audio;
[0105] After the user clicks the "text input area" panel, he can directly enter the text prompt string, and the input content will be saved by the system in real time for subsequent synthesis operations.
[0106] After the user clicks the "Audio Selection" section, he can directly enter a text prompt string, and the input content will be saved by the system in real time for subsequent synthesis operations.
[0107] The "3D Visualization Display Area" section displays the visualization results of 3D face animation generation, which is convenient for users to quickly preview. Users can use the left and right buttons of the mouse and the scroll wheel to perform translation, rotation and scaling of the 3D posture.
[0108] The "Playback Progress Bar" section provides time series visualization operations for 3D motion sequences, including the functions of play / pause, stop, and jump to a specific time frame. These operations allow users to flexibly conduct time series review of results.
[0109] This embodiment obtains text information input by the user and pre-processes the text information to generate a phoneme sequence, a rhythmic structure and word segmentation information; the phoneme sequence, the rhythmic structure and the word segmentation information are input into the improved VITS model to generate a target audio, and based on the target audio, a 3D facial animation is generated through the Faceformer model; according to the user's operation instructions, the playback display of the target audio and the visualization display of the 3D facial animation are controlled. Since this embodiment generates the target audio through the improved VITS model, and then generates the 3D facial animation through the Faceformer model, and then according to the user's operation instructions, the playback display of the target audio and the visualization display of the 3D facial animation are controlled, thereby realizing the synchronous playback and visualization of the facial expressions and audio of the 3D facial animation.
[0110] In addition, an embodiment of the present invention further provides a storage medium, on which a sound customization program is stored. When the sound customization program is executed by a processor, the steps of the sound customization method described above are implemented.
[0111] Reference Fig.10 , Fig.10 It is a structural block diagram of the first embodiment of the sound customization device of the present invention.
[0112] like Fig.10 As shown, the sound customization device proposed in the embodiment of the present invention includes: an information acquisition module 501 and an audio generation module 502.
[0113] The information acquisition module 501 is used to acquire text information input by a user, and pre-process the text information to generate a phoneme sequence, a prosodic structure and word segmentation information.
[0114] The audio generation module 502 is used to input the phoneme sequence, the prosodic structure and the word segmentation information into the improved VITS model to generate target audio. The improved VITS model includes a prosodic enhanced fusion vector module and an iSTFT-based decoder module. The prosodic enhanced fusion vector module is used to integrate the phoneme sequence, the prosodic structure and the word segmentation information.
[0115] The audio generation module 502 is also used to construct a VITS model, and improve the VITS model to obtain an initial VITS model; and train the initial VITS model to obtain an improved VITS model.
[0116] The audio generation module 502 is also used to construct a VITS model, and a rhythmically enhanced fusion vector module is added on the basis of the VITS model. The rhythmically enhanced fusion vector module is used to input the phoneme sequence, rhythmic structure and word segmentation information into a text encoder to generate a fusion vector; after adding the rhythmically enhanced fusion vector module, the iSTFT model is used to replace some repeated network layers in HiFi-GAN, and a multi-subband parallel generation strategy is introduced to replace the decoder module of the VITS model with a decoder module based on iSTFT to obtain an initial VITS model.
[0117] This embodiment obtains text information input by a user and pre-processes the text information to generate a phoneme sequence, a prosodic structure, and word segmentation information; the phoneme sequence, the prosodic structure, and the word segmentation information are input into an improved VITS model to generate a target audio, wherein the improved VITS model includes a prosodic enhanced fusion vector module and an iSTFT-based decoder module, wherein the prosodic enhanced fusion vector module is used to integrate the phoneme sequence, the prosodic structure, and the word segmentation information. Since this embodiment integrates the phoneme sequence, the prosodic structure, and the word segmentation information through a prosodic enhanced fusion vector module and introduces an iSTFT-based decoder module, compared to the prior art, this embodiment not only improves the naturalness of Chinese speech synthesis, but also improves the efficiency of Chinese speech synthesis.
[0118] Based on the above-mentioned first embodiment of the sound customization device of the present invention, a second embodiment of the sound customization device of the present invention is proposed.
[0119] In this embodiment, the information acquisition module 501 is also used to perform sentence processing on the text information to obtain sentence units; perform word segmentation processing on each of the sentence units to obtain an initial word segmentation text, and perform part-of-speech tagging on each of the initial word segmentation texts to generate an initial word segmentation label; perform feature symbol conversion and standardization on each of the initial word segmentation texts and each of the initial word segmentation labels to obtain a word segmentation text and a word segmentation label; predict the rhythmic structure of the text information through a rhythmic structure prediction model based on the word segmentation text and the word segmentation label; based on the word segmentation text and the word segmentation label, convert the text information into a phoneme sequence through a pinyin-phoneme converter.
[0120] The information acquisition module 501 is further used to process polyphone problems through a polyphone disambiguation model according to the segmentation text and the segmentation label to ensure correct pinyin conversion.
[0121] The information acquisition module 501 is also used to determine whether to perform sentence processing on the text information based on the text information; if the text information does not need to be sentence processed, then the text information is segmented to obtain initial segmented texts, and each initial segmented text is tagged with a part of speech to generate an initial segmented label; accordingly, if the text information needs to be sentence processed, the step of segmenting the text information to obtain sentence units is executed.
[0122] The information acquisition module 501 is also used to acquire corpus data, which includes prosodic features, phoneme sequences and audio data, and the prosodic features include duration, pitch and intensity; using the SpanPSP model as an initial prosodic structure prediction model, and inputting the corpus data into the initial prosodic structure prediction model for model training to obtain training results; based on the training results, optimizing the initial prosodic structure prediction model to obtain a prosodic structure prediction model.
[0123] Based on the above-mentioned embodiments of the sound customization device of the present invention, a third embodiment of the sound customization device of the present invention is proposed.
[0124] In this embodiment, the audio generation module 502 is also used to generate a 3D facial animation through a Faceformer model based on the target audio; and control the playback display of the target audio and the visual display of the 3D facial animation according to user operation instructions.
[0125] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0126] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0128] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A sound customization method, characterized in that: The sound customization method comprises the following steps: Acquire text information input by a user, and pre-process the text information to generate a phoneme sequence, a rhythmic structure, and word segmentation information; Inputting the phoneme sequence, the prosodic structure and the word segmentation information into an improved VITS model to generate a target audio, wherein the improved VITS model includes a prosodic enhanced fusion vector module and an iSTFT-based decoder module, wherein the prosodic enhanced fusion vector module is used to integrate the phoneme sequence, the prosodic structure and the word segmentation information; Before the step of obtaining text information input by the user and preprocessing the text information to generate a phoneme sequence, a prosodic structure and word segmentation information, the method further includes: Constructing a VITS model, and adding a prosody-enhanced fusion vector module on the basis of the VITS model, wherein the prosody-enhanced fusion vector module is used to input a phoneme sequence, a prosody structure, and word segmentation information into a text encoder to generate a fusion vector; After adding the rhythm-enhanced fusion vector module, the iSTFT model is used to replace some repeated network layers in HiFi-GAN, and a multi-subband parallel generation strategy is introduced to replace the decoder module of the VITS model with a decoder module based on iSTFT, thereby obtaining an initial VITS model; The initial VITS model is trained to obtain an improved VITS model.
2. The sound customization method according to claim 1, characterized in that: The word segmentation information includes word segmentation text and word segmentation tags. The step of preprocessing the text information to generate a phoneme sequence, a prosodic structure and word segmentation information includes: Perform sentence processing on the text information to obtain sentence units; Performing word segmentation processing on each of the sentence units to obtain an initial word segmentation text, and performing part-of-speech tagging on each of the initial word segmentation texts to generate an initial word segmentation label; Performing feature symbol conversion and standardization processing on each of the initial segmentation texts and each of the initial segmentation labels to obtain segmentation texts and segmentation labels; Predicting the prosodic structure of the text information by a prosodic structure prediction model according to the segmented text and the segmented label; Based on the word segmentation text and the word segmentation label, the text information is converted into a phoneme sequence through a pinyin-phoneme converter.
3. The sound customization method according to claim 2, characterized in that: Before the step of predicting the prosodic structure of the text information by a prosodic structure prediction model according to the segmented text and the segmented label, the method further includes: Acquire corpus data, wherein the corpus data includes prosodic features, phoneme sequences, and audio data, wherein the prosodic features include duration, pitch, and intensity; Using the SpanPSP model as an initial prosodic structure prediction model, and inputting the corpus data into the initial prosodic structure prediction model for model training to obtain a training result; Based on the training results, the initial prosodic structure prediction model is optimized to obtain a prosodic structure prediction model.
4. The sound customization method according to claim 2, characterized in that: After the step of converting and normalizing the initial segmentation texts and initial segmentation labels to obtain the segmentation texts and segmentation labels, the method further includes: According to the segmented text and the segmented label, the polyphone problem is processed by a polyphone disambiguation model to ensure correct pinyin conversion.
5. The sound customization method according to claim 2, characterized in that: Before the step of performing sentence processing on the text information to obtain sentence units, the method further includes: Based on the text information, determining whether to perform sentence processing on the text information; If the text information does not need to be processed into sentences, the text information is processed into words to obtain initial word segmentation texts, and each initial word segmentation text is tagged with a part of speech to generate an initial word segmentation tag; Correspondingly, if it is necessary to perform sentence processing on the text information, the step of performing sentence processing on the text information to obtain sentence units is performed.
6. The sound customization method according to any one of claims 1 to 5, characterized in that: After the step of inputting the phoneme sequence, the prosodic structure and the word segmentation information into the improved VITS model to generate the target audio, the method further includes: Based on the target audio, generating a 3D facial animation through a Faceformer model; According to user operation instructions, the playback display of the target audio and the visual display of the 3D face animation are controlled.
7. A sound customization device, characterized in that: The sound customization device comprises: An information acquisition module is used to acquire text information input by a user, and pre-process the text information to generate a phoneme sequence, a rhythmic structure, and word segmentation information; An audio generation module, used for inputting the phoneme sequence, the prosodic structure and the word segmentation information into an improved VITS model to generate a target audio, wherein the improved VITS model includes a prosodic enhanced fusion vector module and an iSTFT-based decoder module, wherein the prosodic enhanced fusion vector module is used for integrating the phoneme sequence, the prosodic structure and the word segmentation information; The audio generation module is also used to construct a VITS model, and a rhythmically enhanced fusion vector module is added on the basis of the VITS model, wherein the rhythmically enhanced fusion vector module is used to input a phoneme sequence, a rhythmic structure and word segmentation information into a text encoder to generate a fusion vector; after adding the rhythmically enhanced fusion vector module, the iSTFT model is used to replace some repeated network layers in HiFi-GAN, and a multi-subband parallel generation strategy is introduced to replace the decoder module of the VITS model with a decoder module based on iSTFT to obtain an initial VITS model; the initial VITS model is trained to obtain an improved VITS model.
8. A sound customization device, characterized in that: The device comprises: a memory, a processor, and a sound customization program stored in the memory and executable on the processor, wherein the sound customization program is configured to implement the steps of the sound customization method according to any one of claims 1 to 7.
9. A storage medium, characterized in that: The storage medium stores a sound customization program, and when the sound customization program is executed by the processor, the steps of the sound customization method according to any one of claims 1 to 7 are implemented.
