Personalized speech synthesis method, electronic device, server, and storage medium

Abstract

Embodiments of the present application provide a personalized speech synthesis method, an electronic device, a server and a storage medium. The method comprises: in the case that the speech synthesis function of the electronic device is turned on, obtaining text information in the electronic device that needs to be synthesized by speech, the text information comprising user input text information and electronic device generated text information; inputting the text information and the audio features corresponding to the user into an audio model for speech synthesis, and outputting personalized speech, wherein the audio features corresponding to the user are features generated according to the speech information input by the user when registering the speech synthesis function, and the audio model is a text-independent model obtained by training first text information and first speech information in the training data. The method can improve the processing efficiency of speech synthesis and improve the user experience.

Description

Technical Field

[0001] This application relates to the field of electronic technology, specifically to a personalized speech synthesis method, electronic device, server, and storage medium. Background Technology

[0002] With the continuous development of electronic devices, their rich functions have greatly facilitated users' lives. For example, when an electronic device has its voice assistant function enabled, users can have voice conversations with it. If the user inputs a voice message, the device can respond with a voice message. Another example is when an electronic device has its artificial intelligence (AI) calling function enabled. If an incoming call is received, the user can input text on the call screen, and the device will synthesize the text into speech and send it to the other party.

[0003] In the scenarios described above, whether electronic devices provide voice responses or synthesize text into speech, the text-to-speech (TTS) process is involved. To improve the user experience, electronic devices also offer personalized speech synthesis functions, allowing users to record their own voices so that the output voice from the device reflects their own tone. How to enable users to conveniently and quickly record their own voices is a problem that needs to be solved for this function. Summary of the Invention

[0004] This application provides a personalized speech synthesis method, electronic device, server, and storage medium, which can improve the processing efficiency of speech synthesis and enhance the user experience.

[0005] Firstly, this application provides a personalized speech synthesis method, which includes: when the speech synthesis function of an electronic device is enabled, acquiring text information in the electronic device that needs to be synthesized, the text information including text information input by the user and text information generated by the electronic device; inputting the text information and the user-corresponding audio features into an audio model for speech synthesis, and outputting personalized speech, wherein the user-corresponding audio features are features generated based on the speech information recorded by the user when registering for the speech synthesis function, and the audio model is a text-independent model trained based on the first text information and the first speech information in the training data.

[0006] When users use the speech synthesis function of electronic devices, they can make the electronic devices output the voice tone that the user wants to use (i.e., personalized voice). For example, in the scenario of voice assistant function, after the user inputs a sentence by voice, the electronic device can use personalized voice to reply. Another example is in the scenario of AI call function, where the user inputs text information, and the electronic device can convert the text information into personalized voice (such as the user's own voice) to reply to the other party.

[0007] It should be noted that the personalized speech synthesis method in this application can be executed by an electronic device or by a server (such as a cloud server). The first aspect will be described using the execution by an electronic device as an example.

[0008] In this application, when a user registers for speech synthesis (i.e., custom voice registration), the user can read aloud prompt phrases or sentences. The electronic device can extract the user's corresponding audio features based on the user's input voice information. Then, after receiving text information, the electronic device can process the text information and the user's corresponding audio features based on the trained audio model to obtain personalized speech with the user's timbre. Here, the audio model is a text-independent model trained based on the first text information and the first voice information in the training data. Therefore, when processing the text information and the user's corresponding audio features, the audio model can use the user's corresponding audio features as prompts to synthesize speech with the user's timbre for output. That is, the audio model in this application does not need further fine-tuning based on the user's corresponding audio features, thus improving the processing efficiency of speech synthesis.

[0009] For text information that requires speech synthesis in the aforementioned electronic devices, in the scenario of voice assistant function, the text information can be the information that the electronic device queries and replies to in response to the user's voice input. In the scenario of AI call function, the text information can be the information that the user inputs to reply to the other party.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the above-mentioned input of text information and user-corresponding audio features into an audio model for speech synthesis and output of personalized speech includes: extracting features from text information to obtain semantic features corresponding to the text information; and synthesizing speech using the semantic features and user-corresponding audio features through an audio model to output personalized speech.

[0011] In electronic devices, when using audio models to synthesize speech from text information and user-corresponding audio features, features can be extracted from the text information first to obtain corresponding semantic features, and then speech synthesis can be performed based on the semantic features and user-corresponding audio features.

[0012] In some implementations, electronic devices can first perform front-end processing on text information such as text regularization, prosody prediction, and phonetic processing to obtain the phoneme information corresponding to the text information. Then, the phoneme information can be processed by dimension mapping, encoding, and duration stretching to obtain the semantic features corresponding to the text information.

[0013] Among them, dimension mapping processing refers to the process of mapping high-dimensional data (such as text, images, and audio) to a low-dimensional space; encoding processing refers to the process of integrating features of input text information and transforming it into high-dimensional abstract feature blocks; duration stretching processing refers to stretching the reading duration of text information, that is, predicting the duration of reading each character (or phoneme) and adding the duration information to the text information.

[0014] When synthesizing speech using audio models by combining semantic features and user-specific audio features, the user-specific audio features can be used as timbre cues. This allows for the fusion and decoding of semantic and audio features, converting text information into personalized speech that matches the user's timbre. Therefore, regardless of the type of text information, as long as the user's audio features from the registration process are extracted, the text information can be converted into personalized speech, improving the processing efficiency of speech synthesis.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, before performing duration stretching processing on the phoneme information, the above method further includes: predicting the stretching duration corresponding to the phoneme information based on a preset duration prediction model, wherein the duration prediction model is a model trained based on the encoding result corresponding to the first text information and the acoustic features corresponding to the first speech information in the training data.

[0016] When electronic devices perform duration stretching on text information (or phoneme information), since there is no corresponding speech information to determine the stretching duration, a duration prediction model can be used to predict the stretching duration. This duration prediction model can be a network model trained in parallel during the audio model training process. Therefore, in the process of inferring and generating personalized speech, the electronic device can use the duration prediction model, inputting the user's corresponding audio features and the encoded results of the text information, to obtain the stretching duration, which can then be used for subsequent duration stretching of the text information (or phoneme information).

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the training method of the audio model includes: performing front-end processing, dimension mapping processing, encoding processing, and duration stretching processing on the first text information to obtain the semantic features corresponding to the first text information; performing audio encoding processing, multi-channel mapping processing, and multi-channel accumulation processing on the first speech information to obtain the acoustic features corresponding to the first speech information, wherein the data length of the semantic features is equal to the data length of the acoustic features; generating a random number m, dividing the semantic features into a first sub-feature and a second sub-feature based on the random number m, and dividing the acoustic features into a third sub-feature and a fourth sub-feature; using the third sub-feature as the first target feature, and generating a second target feature based on the second and fourth sub-features; fusing and decoding the first target feature and the second target feature through the audio model to obtain the output result; adjusting the parameter values ​​of the audio model according to the loss between the features of the output result and the acoustic features, so that the audio model converges.

[0018] It is understandable that before using the audio model, it needs to be trained to obtain a converged network model. During training, features need to be extracted from the first text information to obtain semantic features, and features need to be extracted from the first speech information to obtain acoustic features. Then, based on randomly generated numbers m, the semantic and acoustic features are divided into two parts respectively. For example, the semantic features are divided into two parts: prompt_se (the first sub-feature) and target_se (the second sub-feature), and the acoustic features are divided into two parts: prompt_ac (the third sub-feature) and target_ac (the fourth sub-feature). Next, the audio model is used to fuse the divided prompt_se and target_se, and prompt_ac and target_ac, to obtain the target prompt (the first target feature) and the target target (the second target feature). Optionally, target prompt = prompt_ac, target target = a × target_se + b × target_ac, where a and b are weight coefficients. Finally, the electronic device can input the first and second target features into the decoder for decoding, obtain the output result, and compare the features of the output result with the acoustic features of the input speech information to calculate the loss function. Based on this loss function, the parameter values ​​of the audio model are adjusted, ultimately resulting in a converged audio model. It can be seen that when generating the first target feature (i.e., the subsequent timbre cues), it is unrelated to semantic features, that is, unrelated to text. Therefore, when using the subsequent audio features, speech with corresponding timbre can be generated based on any text information.

[0019] Optionally, the range of values ​​for the random number m can be: L is the data length of the semantic features (or acoustic features).

[0020] In conjunction with the first aspect, in some implementations of the first aspect, before inputting the text information and the user's corresponding audio features into the audio model for speech synthesis and outputting personalized speech, the above method further includes: obtaining the voice information recorded by the user when registering for the speech synthesis function; performing audio encoding processing, multi-channel mapping processing, and multi-channel accumulation processing on the recorded voice information to obtain the user's corresponding audio features.

[0021] In this application, the electronic device provides a registration interface when a user registers for the speech synthesis function. For example, in the voice assistant function scenario, the user can set the broadcast tone in the smart voice settings interface, such as customizing the tone. The custom tone settings interface displays prompts for reading phrases, which the user can then read aloud to allow the electronic device to record their voice. Optionally, if the user fails to record successfully, they can re-record. Optionally, in this application, the number of short phrases N ≤ 2 when the user inputs voice information.

[0022] After recording is complete, the electronic device can perform audio encoding, multi-channel mapping, and multi-channel accumulation processing on the user's recorded voice information to obtain the user's corresponding audio features. Optionally, the electronic device can also associate the user's corresponding audio features with the user's identifier and store them in a feature library. Then, when the electronic device performs speech synthesis, it can directly look up the user's corresponding audio features in the feature library.

[0023] In the above implementation, users only need to read a few short phrases when registering for a custom voice, without having to read multiple sentences, making the operation simpler and improving the user experience.

[0024] Secondly, this application provides an apparatus included in an electronic device, which has the function of implementing the behaviors of the electronic device in the first aspect and possible implementations thereof. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-described functions. For example, a receiving module or unit, a processing module or unit, etc.

[0025] Thirdly, this application provides an electronic device, which includes a processor, a memory, and an interface; the processor, memory, and interface cooperate with each other to enable the electronic device to execute any one of the methods in the first aspect of the technical solution.

[0026] Fourthly, this application provides a server, which includes one or more processors; one or more memories; the memories store one or more programs, which, when executed by the processors, cause the server to perform any of the methods described in the first aspect.

[0027] In one implementation, the server can be a cloud server. After receiving the personalized voice, the cloud server can return the personalized voice to the electronic device, which can then output the personalized voice or send it to other electronic devices.

[0028] Fifthly, this application provides a chip including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof.

[0029] Optionally, the chip may also include a memory, which is connected to the processor via a circuit or wire.

[0030] Alternatively, the chip may also include a communication interface.

[0031] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform any one of the methods in the first aspect of the technical solution.

[0032] In a seventh aspect, this application provides a computer program product comprising: computer program code, which, when executed on an electronic device, causes the electronic device to perform any one of the methods in the technical solution of the first aspect. Attached Figure Description

[0033] Figure 1 This is an example application scenario diagram of a voice assistant function provided in an embodiment of this application;

[0034] Figure 2 This is an example application scenario diagram of the AI ​​call function provided in the embodiments of this application;

[0035] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of an example interface for setting up a voice assistant function provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of an example of a custom timbre setting process provided in an embodiment of this application;

[0038] Figure 6This is a schematic diagram of another example of a custom timbre setting process provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of the interface for another example of a custom timbre setting process provided in the embodiments of this application;

[0040] Figure 8 This is a schematic diagram of an example audio model training process provided in an embodiment of this application;

[0041] Figure 9 This is a schematic diagram of a personalized speech synthesis method provided in an embodiment of this application;

[0042] Figure 10 This is a schematic diagram illustrating another example of a personalized speech synthesis method provided in this application embodiment;

[0043] Figure 11 This is a software structure block diagram of an electronic device provided in an embodiment of this application;

[0044] Figure 12 This is a flowchart illustrating an example of a personalized speech synthesis method provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0046] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0047] With the continuous development of electronic devices, their rich functions have greatly facilitated users' lives. For example, when the voice assistant function is enabled on an electronic device, users can have voice conversations with it. If the user inputs a voice sentence, the electronic device can respond with a voice message. For instance, if the user voice-inputs "How's the weather today?", the electronic device can reply, "Today's low is 15 degrees Celsius, and the high is 25 degrees Celsius." Alternatively, users can also control the electronic device with their voice; for example, if the user voice-inputs "Open..." The electronic device can respond with "OK" via voice and then enter... Interface. For another example, when an electronic device has its AI call function enabled, if the device receives an incoming call, the user can input text on the call interface, causing the electronic device to synthesize the text into speech and send it to the other party's electronic device.

[0048] In a voice assistant scenario, for example, when a user triggers the voice assistant activation, electronic device A can display something like this: Figure 1 The voice input interface shown allows users to input "What's the weather like today?" via voice. Electronic device A then displays the corresponding text. After recognizing the voice, electronic device A can query today's weather information online in the background and convert the corresponding text into voice for output, such as "Today's lowest temperature is 15 degrees Celsius, and the highest temperature is 25 degrees Celsius."

[0049] In AI call functionality scenarios, for example, such as Figure 2 As shown, when electronic device A receives a call from electronic device B, electronic device A displays the incoming call interface. This interface may include information such as username, caller ID, location, and carrier, and may also include an answer control 21, a reject control 22, and a first control 23. Optionally, the first control 23 can be a touch pop-up, card, control, floating ball, or other display format used for user interaction on the electronic device. Optionally, the first control 23 may include a text prompt option area for enabling the "call caption" function. If the user answers the call on electronic device A and clicks the first control 23, electronic device A can update the display of the first control 23 as the first window 200, meaning the user can directly open the call caption while answering the call. At this time, electronic device A can convert the voice information 1 transmitted by electronic device B into corresponding text information 1 and display the text information 1 in the first window 200, for example... Figure 2The text message 1 shown is "Hello, I am financial advisor Xiao Li". If the user enters the message "What can I do for you?" on electronic device A and clicks the send control 24, electronic device A can also display the user's entered text message 2 "What can I do for you?" in the first window 200, and convert the text message "What can I do for you?" into the voice message "What can I do for you?" and transmit it to electronic device B.

[0050] As described above, whether electronic device A is responding to a voice assistant or making an AI call, it involves a text-to-speech (TTS) process. TTS aims to enable electronic devices to generate human-like voices with different timbres. The difference lies in the format: in the voice assistant function, the electronic device converts retrieved text information or pre-defined text replies into speech for output; in the AI ​​call function, the electronic device converts user-inputted text information into speech and sends it to the other party's electronic device. However, both essentially require converting text information into speech.

[0051] In recent years, to improve user experience, electronic devices have offered personalized voice synthesis functions, allowing users to record their own voices so that the device outputs a voice with their own unique tone. For example, in a voice assistant scenario, users can customize their voice by recording text from a template to generate their own personalized tone. When the user then uses the voice assistant to converse with the electronic device, the output voice will be that customized voice. In an AI call scenario, users can also customize their voice to generate their own personalized tone. When the user inputs text to reply on electronic device A, the device can convert that text into a personalized voice message, effectively using the user's own voice for a better user experience.

[0052] In related technologies, when users customize their voice, they typically need to read multiple sentences of template text to record their speech. For example, most related technologies require reading more than 15 sentences, which is quite cumbersome for users. Furthermore, when generating personalized speech based on text information, these technologies rely on inference from a trained model. During model training, a general model is usually trained using a large amount of training data over a considerable period. Then, when the user completes their voice recording, the general model is fine-tuned based on the recorded speech (e.g., 15 sentences) to generate a model corresponding to the user's personalized voice. Finally, this model can be used to infer and generate personalized speech from the input text information. Therefore, this process involves a large amount of training, requiring both initial training and subsequent fine-tuning, resulting in significant data processing and low efficiency.

[0053] In view of this, embodiments of this application provide a personalized speech synthesis method. By constructing a text-independent audio model, users can complete a customized voice with fewer recorded sentences, and generate personalized speech based on the constructed audio model. This eliminates the need for pre-training and post-fine-tuning, resulting in higher data processing efficiency and better applicability. It should be noted that the personalized speech synthesis method provided in this application can be applied to electronic devices capable of personalized voice settings, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and smart home devices. Embodiments of this application do not impose any limitations on the specific type of electronic device.

[0054] For example, Figure 3 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application. Taking a mobile phone as an example, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0055] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0056] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0057] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0058] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0059] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0060] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0061] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Figure 3 The structures of antennas 1 and 2 shown are merely one example. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0062] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0063] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0064] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. Wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0065] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0066] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0067] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0068] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0069] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0070] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0071] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0072] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0073] It is understandable that the above Figure 3 The illustration shown is merely an example when the electronic device is a mobile phone. If the electronic device is a tablet, PC, PDA, wearable device, or other types of device, the structure of the electronic device may include more... Figure 3 The fewer structures shown can also include more than Figure 3 The structures shown are not limited here.

[0074] As described above, if a user wants to use the personalized voice synthesis function, they need to customize their voice (i.e., register for the voice synthesis function). This application provides a setting process for a personalized voice synthesis function, which will be illustrated below using a voice assistant function as an example. For example, as shown... Figure 4As shown, clicking the settings icon on the desktop interface of an electronic device allows the device to enter its settings interface. This interface includes various function options, such as WLAN, Bluetooth, display and brightness, and smart assistant options. Clicking the smart assistant option control 41 redirects the device to the smart assistant settings interface. The smart assistant settings interface includes various smart functions, such as YOYO suggestions, a negative one screen, smart text, smart search, and smart voice. Clicking the smart voice option control 42 redirects the device to the smart voice settings interface. The smart voice settings interface offers different wake-up methods for the voice assistant, such as voice wake-up, power button wake-up, breath wake-up, headphone in-line control wake-up, and Bluetooth device wake-up. Users can select the appropriate wake-up method to activate the voice assistant. It can be understood that after selecting a wake-up method in the smart voice settings interface, the voice assistant function is activated. Subsequently, the personalized speech synthesis method described in this application can be used to synthesize personalized speech to output voice information and engage in dialogue with the user.

[0075] Meanwhile, the broadcast tone can also be configured in the smart voice settings interface. When the user clicks the broadcast tone option control 43, the electronic device can jump to... Figure 5 The interface shown is provided. On this interface, the electronic device offers different types of voices for the user to choose from, such as YOYO (male voice), YOYO (female voice), etc. In addition, the electronic device provides a custom voice function. When the user clicks the custom voice control 51, the electronic device can jump to the custom voice settings interface. The custom voice settings interface displays already recorded voices, such as voice 1 and voice 2, as well as a new voice control 52. When the user clicks the new voice control 52, the electronic device can jump to the new voice interface. The new voice interface presents guiding terms for creating a personal voice. If the user wants to record their own personalized voice, they can click the next control 53 to trigger voice recording. When the user clicks the next control 53, the electronic device can display a prompt to read a phrase. The user can read the corresponding phrase after clicking the start control 54, and the electronic device can then record the user's voice. Optionally, in this embodiment, when registering a user's custom voice, only N phrases (N≤2) need to be read.

[0076] In some cases, if the user experiences interference or reading errors during recording, resulting in a failed recording, the electronic device may display an error message such as... Figure 6 The prompt interface shown displays the message "Entry failed, please re-record". The user can then click the start control 54 again to read the corresponding phrase.

[0077] In other cases, if the user has successfully recorded, the electronic device may display something like this. Figure 7 The interface shown displays a "Play Recording" control 71. Clicking this control allows the user to listen to the previously recorded content. If the user is not satisfied with the recorded content, they can click the "Back" control 72 to re-enter the electronic device. Figure 5 The user records using the interface shown. If the user is satisfied with the recorded content, they can click the next control 73, and the electronic device enters the synthesized voice interface. This interface displays default synthesized phrases and their corresponding synthesized voices. For example, the default synthesized phrase is "I have set your alarm for 8 AM tomorrow." The corresponding synthesized voice can be listened to using the "Listen" control 74. This synthesized voice is generated by analyzing the user's recorded voice, extracting the user's vocal characteristics, and then using these characteristics to infer the synthesized phrase. Essentially, it simulates the user's voice to read the synthesized phrase. If the user is satisfied with the synthesized voice effect, they can click the complete control 75 to finish the custom voice process. Subsequently, when using the voice assistant function, the user can experience personalized voice synthesis, and the electronic device can convert the user's text messages into personalized voice output.

[0078] As can be seen from the above scenario description, in this embodiment of the application, when a user registers a custom voice, they only need to read a few short phrases to complete the process, without having to read multiple sentences. This makes the operation simpler and improves the user experience.

[0079] It's understandable that in AI call scenarios, users can customize their voice in a similar way, allowing the electronic device to convert the user's text input into personalized speech and send it to the other party's electronic device. It's also understandable that, besides the voice assistant and AI call scenarios mentioned above, personalized speech synthesis can be used in other scenarios, such as short video dubbing. After customizing their voice, the electronic device can output the dialogue in the short video in the user's own voice, achieving a personal dubbing effect.

[0080] In the above scenario, if the user clicks the completion control 75 to complete the custom voice process, the electronic device can extract audio features from the user's recorded voice to obtain the user's corresponding audio features. If the user subsequently inputs text information, the electronic device can infer and generate personalized speech based on the constructed audio model and the user's corresponding audio features. Below, we will describe in detail the process of using a text-independent audio model for personalized speech synthesis in this application embodiment.

[0081] First, the audio model in this application embodiment needs to be trained to obtain a converged network model. In some embodiments, the training process of the audio model can be performed by a cloud server. Before training, the cloud server can acquire multiple sets of training data, where each set of training data may include a piece of text information and a piece of speech information. The speech information in different sets of training data may be recorded by different users. For example, the text information in a certain set of training data is "Today's highest temperature is 25 degrees Celsius", and the speech information is the speech of a user reading "Today's highest temperature is 25 degrees Celsius". For each set of training data, the cloud server can perform the following processing: taking training data A as an example, the cloud server extracts features from the text information and speech information in training data A to obtain semantic features and acoustic features, then constructs a text-independent prompt based on the semantic features and acoustic features, and then decodes the constructed prompt to obtain the output result. In other embodiments, if the electronic device has the ability to train the model, the training process of the audio model can also be performed by the electronic device.

[0082] like Figure 8 As shown, the process of extracting semantic features from text information by the cloud server can include front-end processing, embedding, encoding, and time stretching (LR). The front-end processing can include text regularization, prosody prediction, and phonetic annotation.

[0083] Since some words are abbreviated or shortened in daily life, the cloud server needs to standardize these words. For example, it needs to convert phone numbers, times, money, units, symbols, email addresses, and dates into standardized text, i.e., perform text regular expression processing. For instance, Sep. 11th needs to be expanded to the full spelling Sepember Eleventh. Optionally, the cloud server can use regular expressions to process the text information. Furthermore, since some words require pauses or emphasis when read aloud, inaccurate pauses can lead to disjointed and unnatural reading, even affecting the expression of the corresponding speech information. Therefore, the cloud server also needs to perform prosodic prediction processing on the text information. For instance, prosodic prediction of the text "Today's lowest temperature is 15 degrees Celsius" can result in the pause pattern "Today #lowest temperature #15# degrees Celsius". Optionally, the cloud server can use a deep network for prosodic prediction, where the deep network has been trained using prosodic text data. Next, because some characters are polyphonic and have different pronunciations in different words, the cloud server needs to perform phonetic annotation on the text information to accurately determine the pronunciation of each character. Optionally, the cloud server can also use a deep neural network to convert the text information into pinyin to solve the problem of polyphonic characters. The deep neural network used is trained using data on polyphonic characters. Optionally, after phonetic annotation, the cloud server can obtain the phoneme information corresponding to the text information.

[0084] After front-end processing of text information, the cloud server can perform dimensional mapping (embedding), encoding (encoder), and duration stretching on the text information. Embedding refers to the process of mapping high-dimensional data (such as text, images, and audio) to a low-dimensional space; encoding refers to the process of integrating features from the input text information and transforming it into high-dimensional abstract feature blocks; duration stretching refers to stretching the reading time of the text information, i.e., predicting the duration of reading each character (or phoneme) and adding the duration information to the text information to obtain the total duration frame information corresponding to the text information. For example, if the user reads a character for n seconds, then the total duration frame information corresponding to a sentence containing 10 characters is 10n seconds. Optionally, when the cloud server performs duration stretching on the text information, the corresponding stretching duration can be determined based on the speech information in training data A, because the text information and speech information in training data A correspond, and the speech duration corresponding to that speech information can be used as the duration to be stretched for the text information.

[0085] After the above processing, the cloud server can obtain the semantic features corresponding to the text information.

[0086] Simultaneously, the cloud server also extracts features from the speech information to obtain acoustic features. In other words, the cloud server can extract semantic features and acoustic features in parallel. See also... Figure 8 The process of extracting acoustic features from speech information by the cloud server can include audio encoding, multi-channel embedding, and multi-channel accumulation.

[0087] The cloud server can use an audio encoder to encode the speech information, resulting in an acoustic vector corresponding to the speech information. This acoustic vector can be a multi-channel, multi-frame feature vector, for example, an 8-channel feature vector, or a feature vector with other numbers of channels; this embodiment does not limit this. Then, the cloud server can mask the feature vector of each channel in the acoustic vector and then embed it, thus obtaining a multi-channel embedding, for example, an 8-channel embedding. Optionally, the embedding method for each channel feature vector can be acoustic_semb_i = embedding(mask(acoustic_i)). Here, the masking function masks some frames in the multi-frame feature vector for subsequent inference generation, thereby training the inference accuracy. Next, the cloud server can perform weighted summation of the multi-channel embeddings to obtain a feature vector for one channel, i.e., obtain the acoustic feature. Optionally, the cloud server can use acoustic = ∑ N i＝1 (w_i×acoustic_semb_i) calculates the acoustic features, where w_i is the weight coefficient corresponding to the channel, and N is the number of channels, for example, N=8.

[0088] After the above processing, the cloud server can obtain the acoustic features corresponding to the speech information. It can be understood that since the cloud server can determine the duration based on the speech duration corresponding to the speech information when stretching the text information, the length of the final semantic features is equal to the length of the acoustic features.

[0089] After obtaining the semantic and acoustic features, the cloud server can begin constructing a text-independent prompt (the core part of the audio model), see below. Figure 8The process may include: first generating a random number m, and then dividing the semantic features and acoustic features into two parts based on the random number m, wherein m is less than the length of the semantic feature (or acoustic feature). Preferably, `len(semantic)` represents the length of the semantic features. The cloud server then divides the semantic features into two parts, `prompt_se` and `target_se`, and the acoustic features into two parts, `prompt_ac` and `target_ac`, based on a random number `m`. Next, the cloud server fuses the divided `prompt_se` and `target_se`, and `prompt_ac` and `target_ac`, using an audio model to obtain the target `prompt` and the target `target`. Optionally, `target_prompt = prompt_ac`, and `target_ac = a × target_se + b × target_ac`, where a and b are weight coefficients. It can be seen that the target `prompt` is unrelated to the semantic features, i.e., unrelated to the text. Therefore, the target `prompt` can serve as a cue for the subsequent decoder, enabling the decoder to add timbre to the text information based on the timbre carried in the target `prompt`, thus ensuring that the timbre of the output speech is as close as possible to the timbre corresponding to the input speech information. The target `target` can then be used as the learning content for the audio model.

[0090] After obtaining the target prompt and target, the cloud server can input them into the decoder for decoding, obtaining a speech output. Then, the features of this output are compared with the acoustic features of the input speech information to calculate a loss function. Based on this loss function, the parameter values ​​of the audio model are adjusted, ultimately resulting in a converged audio model. Optionally, the decoder input method can include decoder_in = cat(target prompt, target).

[0091] After the above process, the cloud server constructs the audio model required for the embodiments of this application. If a user registers a custom voice on an electronic device, the audio model can obtain the audio features of that custom voice. For example, in a voice assistant scenario, the audio model can combine audio features and retrieved text information to infer and generate personalized speech. In an AI call scenario, the audio model can combine audio features and user-inputted text information to infer and generate personalized speech. In some embodiments, the process of the audio model inferring and generating personalized speech can still be executed on the cloud server. This requires the electronic device to send the user's custom voice information and subsequent text information to the cloud server. In other embodiments, the process of the audio model inferring and generating personalized speech can also be executed on the electronic device. This requires the cloud server to port the trained and converged audio model to the electronic device. The following description uses the cloud server as an example.

[0092] like Figure 9 As shown, after a user registers a custom voice, the electronic device can send the voice information recorded during registration to a cloud server. The cloud server can extract audio features from the voice information to obtain the audio features corresponding to the user. Optionally, the cloud server can map the user's identifier to the audio features and store it in a feature library. Here, the user's identifier can be the username used by the user when registering on the electronic device, the device identifier of the electronic device, etc., which is not limited in this embodiment. Then, if the cloud server receives text information sent by the electronic device (which can be text information entered by the user or text information queried by the electronic device in the background), the cloud server can first perform front-end processing such as text regularization, prosody prediction, and phonetic processing on the text information, and then input the front-end processed text information and the user's corresponding audio features into the audio model for inference, output personalized voice, and return it to the electronic device. Optionally, the audio model can include two parts: an acoustic model and an audio decoder. The acoustic model is used to combine audio features to convert text information into audio acoustic features (i.e., the process of constructing a prompt), and the audio decoder is used to convert the audio acoustic features into audio for output. It is understandable that, unlike the audio model training process mentioned above, where the text and speech information in the training data are paired in a corresponding manner, in the process of generating personalized speech through inference, the text information received by the cloud server changes in real time, and the audio features used are extracted when the user registers their custom timbre. The purpose is to output the real-time changing text information as personalized speech with the user's timbre based on the user's audio features.

[0093] for Figure 9 The illustrated embodiments, combined with Figure 10The execution process of the cloud server is described as follows: After receiving the voice information recorded during user-defined voice registration, the cloud server performs audio encoding, multi-channel embedding, and multi-channel accumulation processing to obtain the corresponding acoustic features, which serve as the user's audio features. These audio features are then linked to the user's identifier and stored in a feature library. When the cloud server receives text information, it performs front-end processing to obtain the corresponding phoneme information. After front-end processing, the cloud server performs dimensionality mapping, encoding, and duration stretching on the text information (phoneme information) to obtain the corresponding semantic features. Next, the cloud server uses an audio model to merge the acoustic and semantic features, using the acoustic features as prompts, and inputs them into a decoder for processing, outputting personalized voice. In essence, the cloud server can return the personalized voice to an electronic device, which can then output it to the user via a voice assistant or transmit it to another electronic device via AI call functionality.

[0094] When the cloud server stretches the text information, since there is no corresponding audio information to determine the stretching duration, the cloud server can predict the stretching duration based on a duration prediction model. This duration prediction model can be a network model trained in parallel during the training of the aforementioned audio model. Figure 8 As shown in the process, the cloud server can train a duration prediction model based on the encoder result and prompt_ac of the text information through a cross-attention mechanism. Therefore, in the process of inference to generate personalized speech, the cloud server can use the duration prediction model, inputting audio features and encoder results, to obtain the stretched duration, which can then be used to perform duration stretching processing on the text information (or phoneme information).

[0095] As can be seen from the above audio model processing process, the embodiment of this application constructs a text-independent audio model during training. The prompt is used as the prompt information, and the speech information corresponding to the timbre carried in the prompt can be inferred from any subsequent text information. Therefore, in actual use, users only need to record a short sentence during registration, extract the user's audio features as the prompt input audio model, and personalized speech can be output without the need for post-fine-tuning, which makes the data processing efficiency higher.

[0096] In other embodiments, the process of generating personalized voice through the audio model inference described above can be executed on an electronic device. The technical principle of the process executed by the electronic device is similar to that executed on a cloud server, and the specific process will not be described in detail here.

[0097] It is understood that the software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. If the electronic device uses audio model inference to generate personalized voice, it should be executed in conjunction with the software system architecture of the electronic device. This application embodiment uses the layered architecture Android system as an example to illustrate the software structure of the electronic device.

[0098] Figure 11 This is a software architecture block diagram of an electronic device according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages, such as Android application packages (APKs).

[0099] like Figure 11 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, music, video, and SMS. It may also include an APK that integrates an audio model, which can enable users to register custom voices and generate personalized voices based on the user's audio characteristics and input text information.

[0100] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0101] like Figure 11 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0102] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, and more.

[0103] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0104] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0105] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0106] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0107] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0108] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0109] When the electronic device executes the aforementioned personalized speech synthesis method Figure 12 This is a flowchart illustrating an example of a personalized speech synthesis method provided in an embodiment of this application, as shown below. Figure 12 As shown, the method may include:

[0110] S101: When the electronic device has its speech synthesis function enabled, acquire the text information in the electronic device that needs to be speech synthesized.

[0111] The text information includes text information input by the user and text information generated by the electronic device. The text information input by the user can be the information that the user enters to reply to the other party in the scenario of AI call function, and the text information generated by the electronic device can be the information that the electronic device retrieves to reply to the user in the scenario of voice assistant function.

[0112] S102 inputs text information and user-specific audio features into the audio model for speech synthesis and outputs personalized speech.

[0113] The user's corresponding audio features are generated based on the voice information recorded when the user registers for the speech synthesis function. The audio model is a text-independent model trained based on the first text information and the first voice information in the training data. The processing procedure of the audio model can be found in the description of the above embodiments, and will not be repeated here.

[0114] The foregoing has detailed examples of personalized speech synthesis methods provided in the embodiments of this application. It is understood that, in order to achieve the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0115] This application embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0116] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0117] The electronic device provided in this embodiment is used to execute the above-described personalized speech synthesis method, and therefore can achieve the same effect as the above-described implementation method.

[0118] When using integrated units, the electronic device may further include a processing module, a storage module, and a communication module. The processing module is used to control and manage the operation of the electronic device. The storage module supports the execution of stored program code and data. The communication module supports communication between the electronic device and other devices.

[0119] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a digital signal processor (DSP), and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0120] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment can be a device having... Figure 3 The device with the structure shown.

[0121] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the personalized speech synthesis method of any of the above embodiments.

[0122] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the personalized speech synthesis method described in the above embodiments.

[0123] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the personalized speech synthesis method in the above method embodiments.

[0124] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0125] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0127] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A personalized speech synthesis method, characterized in that, The method includes: When the speech synthesis function is enabled on the electronic device, the text information that needs to be synthesized in the electronic device is obtained, including text information input by the user and / or text information generated by the electronic device. The text information and the audio features corresponding to the user are input into the audio model for speech synthesis to output personalized speech. The audio features corresponding to the user are generated based on the speech information recorded when the user registers for the speech synthesis function. The audio model is a text-independent model trained based on the first text information and the first speech information in the training data. The training methods for the audio model include: The first text information is processed by front-end processing, dimension mapping processing, encoding processing, and duration stretching processing to obtain the semantic features corresponding to the first text information. The first speech information is subjected to audio encoding processing, multi-channel mapping processing, and multi-channel accumulation processing to obtain the acoustic features corresponding to the first speech information, wherein the data length of the semantic features is equal to the data length of the acoustic features; Generate a random number m, and divide the semantic features into a first sub-feature and a second sub-feature based on the random number m, and divide the acoustic features into a third sub-feature and a fourth sub-feature; The third sub-feature is used as the first target feature, and a second target feature is generated based on the second and fourth sub-features; The first target feature and the second target feature are fused and decoded using the audio model to obtain the output result. Based on the loss between the features of the output result and the acoustic features, the parameter values ​​of the audio model are adjusted until the audio model converges.

2. The method according to claim 1, characterized in that, The step of inputting the text information and the user's corresponding audio features into an audio model for speech synthesis and outputting personalized speech includes: Feature extraction is performed on the text information to obtain the semantic features corresponding to the text information; The personalized voice is output by synthesizing the semantic features and the user's corresponding audio features using the audio model.

3. The method according to claim 2, characterized in that, The step of synthesizing speech using the audio model based on the semantic features and the user-corresponding audio features, and outputting the personalized speech, includes: The user's corresponding audio features are used as timbre cues. The semantic features and the user's corresponding audio features are fused and decoded by the audio model to obtain and output the personalized voice.

4. The method according to claim 2, characterized in that, The step of extracting features from the text information to obtain the semantic features corresponding to the text information includes: The text information is processed by the front end to obtain the phoneme information corresponding to the text information. The front end processing includes at least one of the following processing procedures: text regularization processing, prosody prediction processing, and phonetic processing. The phoneme information is subjected to dimension mapping, encoding, and duration stretching to obtain the semantic features corresponding to the text information.

5. The method according to claim 4, characterized in that, Before performing duration stretching on the phoneme information, the method further includes: The duration prediction model is based on a preset duration prediction model to predict the stretch duration corresponding to the phoneme information. The duration prediction model is a model trained based on the encoding result corresponding to the first text information and the acoustic features corresponding to the first speech information in the training data.

6. The method according to any one of claims 1 to 5, characterized in that, The range of values ​​for the random number m is [ ×L, [×L], where L is the data length of the semantic feature.

7. The method according to any one of claims 1 to 5, characterized in that, Before inputting the text information and the user's corresponding audio features into the audio model for speech synthesis and outputting personalized speech, the method further includes: Obtain the voice information recorded by the user when registering for the speech synthesis function; The recorded voice information is subjected to audio encoding, multi-channel mapping, and multi-channel accumulation to obtain the audio features corresponding to the user.

8. The method according to claim 7, characterized in that, The number of short sentences corresponding to the recorded voice information is N≤2.

9. The method according to claim 7, characterized in that, After obtaining the audio features corresponding to the user, the method further includes: The audio features corresponding to the user are associated with the user's identifier and stored in the feature library.

10. The method according to any one of claims 1 to 5, characterized in that, When the electronic device is using a voice assistant, the text information is text information generated by the electronic device; when the electronic device is using artificial intelligence (AI) for communication, the text information is text information input by the user.

11. An electronic device, characterized in that, include: One or more processors; One or more memory units; The memory stores one or more programs that, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1 to 10.

12. A server, characterized in that, include: One or more processors; One or more memory units; The memory stores one or more programs that, when executed by the processor, cause the server to perform the method as described in any one of claims 1 to 10.

13. The server according to claim 12, characterized in that, When the processor executes the one or more programs, the server also executes: The personalized voice is sent to the electronic device.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 10.

Images