Personalized speech synthesis method, electronic equipment, server and storage medium

Abstract

The embodiment of the invention provides a personalized speech synthesis method, electronic equipment, a server and a storage medium, and the method comprises the steps: obtaining text information needing to be subjected to speech synthesis in the electronic equipment under the condition that the electronic equipment starts a speech synthesis function, the character information comprises character information input by a user and character information generated by the electronic equipment; the text information and the audio features corresponding to the user are input into an audio model for voice synthesis, personalized voice is output, the audio features corresponding to the user are generated according to voice information input when the user registers a voice synthesis function, and the personalized voice is output according to the voice information input when the user registers the voice synthesis function. The audio model is a text-independent model obtained by training according to the first character information and the first voice information in the training data. The method can improve the processing efficiency of speech synthesis and improve the user experience.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a personalized speech synthesis method, electronic equipment, server and storage medium. Background Art

[0002] With the continuous development of electronic devices, the rich functions of electronic devices have provided great convenience for users' lives. For example, when the voice assistant function of an electronic device is turned on, the user can have a voice conversation with the electronic device. If the user enters a voice, the electronic device can make a voice reply based on the voice. For another example, when the electronic device turns on the artificial intelligence (AI) call function, if the electronic device receives an incoming call, the user can enter text in the call interface so that the electronic device can send the text-synthesized voice to the other party's electronic device.

[0003] In the above scenario, whether the electronic device is making a voice reply or synthesizing text into speech, it involves the text to speech (TTS) process. In order to improve the user experience, the electronic device also provides a personalized speech synthesis function, which allows users to record their own voice, so that the subsequent electronic device outputs the voice with the user's own timbre. For this function, how to enable users to record voice conveniently and quickly is a problem that needs to be solved. Summary of the invention

[0004] The present application provides a personalized speech synthesis method, electronic device, server and storage medium, which can improve the processing efficiency of speech synthesis and enhance the user experience.

[0005] In a first aspect, the present application provides a personalized speech synthesis method, the method comprising: when a speech synthesis function is turned on in an electronic device, obtaining text information that needs to be speech synthesized in the electronic device, the text information including text information input by a user and text information generated by the electronic device; inputting the text information and audio features corresponding to the user into an audio model for speech synthesis, and outputting personalized speech, wherein the audio features corresponding to the user are features generated based on the voice information entered by the user when registering the speech synthesis function, and the audio model is a text-independent model obtained by training based on the first text information and the first voice information in the training data.

[0006] Among them, when the user uses the speech synthesis function of the electronic device, the user can make the electronic device output the timbre and voice (i.e., personalized voice) that the user wants to use. For example, in the scenario of the voice assistant function, after the user inputs a sentence by voice, the electronic device can reply with personalized voice. For another example, in the scenario of the AI ​​call function, the user inputs text information, and the electronic device can convert the text information into personalized voice (such as the user's own voice) and reply to the other user.

[0007] It should be noted that the personalized speech synthesis method in the present application can be executed by an electronic device or by a server (such as a cloud server). In the first aspect, the execution by an electronic device is taken as an example for explanation.

[0008] In the present application, when the user registers for the speech synthesis function (i.e., custom voice registration), the user can read a prompt phrase or sentence, and the electronic device can extract the audio features corresponding to the user based on the voice information entered by the user. Then, after the subsequent electronic device receives the text information, it can process the text information and the audio features corresponding to the user based on the trained audio model to obtain a personalized voice with the user's timbre. The audio model here is a text-independent model trained based on the first text information and the first voice information in the training data. Then, when processing the text information and the audio features corresponding to the user, the audio model can use the audio features corresponding to the user as a prompt to synthesize a voice with the user's timbre for output. That is, the audio model in the present application does not need to be fine-tuned and trained based on the audio features corresponding to the user, thereby improving the processing efficiency of speech synthesis.

[0009] For the text information that needs to be voice synthesized in the above-mentioned electronic devices, in the scenario of the voice assistant function, the text information can be the information that the electronic device queries and replies to the voice input by the user. In the scenario of the AI ​​call function, the text information can be the information input by the user to reply to the other user.

[0010] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned inputting text information and audio features corresponding to the user into the audio model for speech synthesis and outputting personalized speech includes: extracting features of the text information to obtain semantic features corresponding to the text information; performing speech synthesis on the semantic features and audio features corresponding to the user through the audio model, and outputting personalized speech.

[0011] Among them, when the electronic device uses an audio model to perform speech synthesis on text information and the audio features corresponding to the user, it can first extract features from the text information to obtain the corresponding semantic features, and then perform speech synthesis based on the semantic features and the audio features corresponding to the user.

[0012] In some implementations, the electronic device may first perform front-end processing such as text regularization, rhythm prediction, and phonetic notation on the text information to obtain the phoneme information corresponding to the text information, and then perform dimension mapping, encoding, and time stretching on the phoneme information to obtain the semantic features corresponding to the text information.

[0013] Among them, dimensional mapping processing refers to the process of mapping high-dimensional data (such as text, pictures, audio) to low-dimensional space; encoding processing refers to the process of integrating the features of the input text information and converting it into high-dimensional abstract feature blocks; duration stretching processing refers to stretching the reading time of text information, that is, predicting the duration of reading each word (or phoneme) and adding the duration information to the text information.

[0014] When the audio model is used to synthesize the semantic features and the audio features corresponding to the user, the audio features corresponding to the user can be used as a timbre prompt, and the semantic features and audio features can be fused and decoded to convert the text information into a personalized voice with the same timbre as the user. Therefore, no matter what kind of text information is, as long as the audio features of the user when registering are extracted, the text information can be converted into a personalized voice, which improves the processing efficiency of speech synthesis.

[0015] In combination with the first aspect, in some implementations of the first aspect, before the phoneme information is time-stretched, the method further includes: predicting the stretching duration corresponding to the phoneme information based on a preset duration prediction model, the duration prediction model being a model trained according to the encoding result corresponding to the first text information in the training data and the acoustic features corresponding to the first speech information.

[0016] Among them, when the electronic device performs time stretching processing on text information (or phoneme information), because there is no voice information corresponding to the text information to determine the stretching time, the stretching time can be predicted based on the time prediction model. The time prediction model here can be a network model trained in parallel during the audio model training process. Therefore, in the process of inferring and generating personalized voice, the electronic device can use the time prediction model, input the audio features (acoustic) corresponding to the user and the encoding results corresponding to the text information, and obtain the stretching time, so as to subsequently perform time stretching processing on the text information (or phoneme information).

[0017] In combination with the first aspect, in some implementation methods of the first aspect, the training method of the audio model includes: performing front-end processing, dimensional mapping processing, encoding processing, and time stretching processing on the first text information to obtain semantic features corresponding to the first text information; performing audio encoding processing, multi-channel mapping processing, and multi-channel accumulation processing on the first voice information to obtain acoustic features corresponding to the first voice information, wherein the data length of the semantic feature is equal to the data length of the acoustic feature; generating a random number m, dividing the semantic feature into a first sub-feature and a second sub-feature according to the random number m, and dividing the acoustic feature into a third sub-feature and a fourth sub-feature; using the third sub-feature as the first target feature, and generating a second target feature based on the second sub-feature and the fourth sub-feature; fusing and decoding the first target feature and the second target feature through the audio model to obtain an output result; adjusting the parameter value of the audio model according to the loss between the feature of the output result and the acoustic feature so that the audio model converges.

[0018] It can be understood that before the audio model is used as described above, the audio model needs to be trained to obtain a converged network model. During the training process, it is necessary to extract features from the first text information to obtain semantic features (semantic), and to extract features from the first voice information to obtain acoustic features (acoustic). Then, based on the randomly generated random number m, the semantic features and the acoustic features are divided into two parts, for example, the semantic features are divided into two parts, prompt_se (i.e., the first sub-feature) and target_se (i.e., the second sub-feature), and the acoustic features are divided into two parts, prompt_ac (i.e., the third sub-feature) and target_ac (i.e., the fourth sub-feature). Next, the divided prompt_se and target_se, prompt_ac and target_ac are fused through the audio model to obtain the target prompt (i.e., the first target feature) and the target target (i.e., the second target feature). Optionally, the target prompt = prompt_ac, the target target = a×target_se+b×target_ac, where a and b are weight coefficients. Finally, the electronic device can input the first target feature and the second target feature into the decoder for decoding to obtain an output result, and compare the features of the output result with the acoustic features of the above-mentioned input voice information, calculate the loss function, and adjust the parameter value of the audio model according to the loss function, and finally obtain a converged audio model. It can be seen that when generating the first target feature (i.e., the subsequent timbre prompt), it is independent of the semantic feature, i.e., it is independent of the text, so when the subsequent audio feature is used, it can generate speech of the corresponding timbre based on any text information.

[0019] Optionally, the value range of the random number m can be L is the data length of the semantic feature (or acoustic feature).

[0020] In combination with the first aspect, in some implementations of the first aspect, before inputting text information and audio features corresponding to the user into an audio model for speech synthesis and outputting personalized speech, the method further includes: obtaining the voice information entered by the user when registering for the speech synthesis function; performing audio encoding processing, multi-channel mapping processing, and multi-channel accumulation processing on the entered voice information to obtain audio features corresponding to the user.

[0021] Among them, when the user registers for the speech synthesis function, the electronic device provides a corresponding registration interface. For example, in the scenario of the voice assistant function, in the smart voice setting interface, the user can set the broadcast tone, such as customizing the tone setting. In the custom tone setting interface, a prompt for reading aloud phrases is displayed, and the user can read aloud the corresponding phrases so that the electronic device can record the user's voice. Optionally, if the user fails to record successfully, the recording can be performed again. Optionally, in this application, when the user enters the voice information, the number of short sentences N≤2 corresponding to the reading.

[0022] After the recording is completed, the electronic device can perform audio encoding processing, multi-channel mapping processing, and multi-channel accumulation processing on the voice information recorded by the user to obtain the audio features corresponding to the user. Optionally, the electronic device can also associate the audio features corresponding to the user with the user's identification and store them in a feature library, so that when the subsequent electronic device performs speech synthesis, it can directly search for the audio features corresponding to the user from the feature library.

[0023] In the above implementation, when the user registers a custom voice, he only needs to read a few phrases to complete it, without having to read multiple sentences. The operation is relatively simple, which improves the user experience.

[0024] In a second aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the electronic device behavior in the above-mentioned first aspect and the possible implementation of the above-mentioned first aspect. The function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a receiving module or unit, a processing module or unit, etc.

[0025] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0026] In a fourth aspect, the present application provides a server, comprising one or more processors; one or more memories; the memories storing one or more programs, and when the one or more programs are executed by the processors, the server executes any one of the methods in the technical solution of the first aspect.

[0027] In one implementation, the server may be a cloud server. After obtaining the personalized voice, the cloud server may return the personalized voice to the electronic device, and the electronic device may output the personalized voice or send it to other electronic devices.

[0028] In a fifth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof.

[0029] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0030] Further optionally, the chip also includes a communication interface.

[0031] In a sixth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution of the first aspect.

[0032] In a seventh aspect, the present application provides a computer program product, which includes: a computer program code, and when the computer program code runs on an electronic device, the electronic device executes any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is an application scenario diagram of a voice assistant function provided in an embodiment of the present application;

[0034] Figure 2 This is an application scenario diagram of an AI call function provided in an embodiment of the present application;

[0035] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0036] Figure 4 This is a schematic diagram of an interface for setting a voice assistant function provided in an embodiment of the present application;

[0037] Figure 5 This is a schematic diagram of an interface for a custom tone setting process provided in an embodiment of the present application;

[0038] Figure 6is a schematic diagram of another interface of a custom tone setting process provided in an embodiment of the present application;

[0039] Figure 7 is another interface schematic diagram of a custom tone setting process provided in an embodiment of the present application;

[0040] Figure 8 is a schematic diagram of an audio model training process provided in an embodiment of the present application;

[0041] Fig. 9 This is a process diagram of a personalized speech synthesis method provided in an embodiment of the present application;

[0042] Fig.10 is a process diagram of another personalized speech synthesis method provided in an embodiment of the present application;

[0043] Fig.11 is a software structure block diagram of an electronic device provided in an embodiment of the present application;

[0044] Fig.12 It is a flowchart of a personalized speech synthesis method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0046] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0047] With the continuous development of electronic devices, the rich functions of electronic devices have provided great convenience for users' lives. For example, when the voice assistant function of an electronic device is turned on, the user can have a voice conversation with the electronic device. If the user inputs a voice, the electronic device can make a voice reply based on the voice. For example, if the user inputs "What's the weather like today", the electronic device can make a voice reply "Today's minimum temperature is 15 degrees Celsius and the maximum temperature is 25 degrees Celsius"; or, the user can also control the electronic device by voice, for example, if the user inputs "Turn on the weather", the electronic device can make a voice reply based on the voice. ”, the electronic device can reply “OK” by voice and enter For another example, when the AI ​​call function is turned on in the electronic device, if the electronic device receives an incoming call, the user can enter text in the call interface, so that the electronic device can send the text-synthesized speech to the other party's electronic device.

[0048] In the scenario of the voice assistant function, for example, when the user triggers the voice assistant to be awakened, the electronic device A may display the following Figure 1 In the voice input interface shown, the user can input "how is the weather today" by voice, and the text corresponding to the voice can be displayed on the electronic device A. After the electronic device A recognizes the voice, it can query today's weather information through the network in the background, and convert the text corresponding to the weather information into voice output, for example, the voice output is "today's minimum temperature is 15 degrees Celsius, and the maximum temperature is 25 degrees Celsius".

[0049] In the scenario of the AI ​​call function, for example, Figure 2 As shown, when electronic device A receives an incoming call from electronic device B, electronic device A displays an incoming call interface, which may include information such as user name, caller number, number location, and operator, and may also include an answer control 21, a reject control 22, and a first control 23. Optionally, the first control 23 may be a display form for user interaction on electronic devices such as a touch pop-up window, a card, a control, or a floating ball. Optionally, the first control 23 may include a text prompt option area for turning on the "call subtitles" function. If the user answers the call on electronic device A and clicks on the first control 23, electronic device A may update the first control 23 to display as a first window 200, that is, the user can directly turn on the call subtitles while answering the call. At this time, electronic device A may convert the voice message 1 transmitted by electronic device B into a corresponding text message 1, and display the text message 1 in the first window 200, for example Figure 2The text message 1 shown in the figure is "Hello, I am Xiao Li, a financial advisor". If the user enters the message "What can I do for you?" on the electronic device A and clicks the send control 24, the electronic device A can also display the text message 2 "What can I do for you?" entered by the user in the first window 200, and convert the text message "What can I do for you?" into a voice message 2 "What can I do for you?" and transmit it to the electronic device B.

[0050] From the description of the above scenario, it can be seen that no matter whether electronic device A is making a voice reply of a voice assistant or a voice call of an AI call, the process of speech synthesis TTS will be involved. Speech synthesis is designed to enable electronic devices to generate human speech with different tones. The difference is that in the voice assistant function, the electronic device converts the queried text information or the fixed reply text information into speech for output, and in the AI ​​call function, the electronic device converts the text information input by the user into speech and sends it to the other electronic device, but in essence, both need to convert text information into speech information.

[0051] In recent years, in order to improve the user experience, electronic devices also provide personalized speech synthesis functions, which allow users to record their own voices so that subsequent electronic devices can output voices with the user's own timbre. For example, in the scenario of the voice assistant function, users can customize the timbre and record voices by reading template text to generate their own personalized timbre. Then, when subsequent users use the voice assistant function to communicate with electronic devices, the voice output by the electronic device is a customized personalized voice. In the AI ​​call scenario, users can also customize the timbre and generate their own personalized timbre. Then, when subsequent users enter the text message to be replied on electronic device A, the electronic device can convert the text message into a voice message corresponding to the personalized voice. In this way, it is equivalent to replying with the user's own voice, and the user experience is better.

[0052] In the related art, when users customize their voice, they usually need to read aloud multiple sentences of template text to record the voice. For example, most related technologies require reading aloud more than 15 sentences of text, which is cumbersome for users to operate. In addition, when the related art generates personalized voice based on text information, it is generated based on the model inference obtained through training. In the process of model training, it is usually necessary to first use a large amount of training data to train a general model, which takes a long time. Then, when the user completes the voice recording, the general model is trained and fine-tuned based on the recorded voice (for example, 15 sentences of voice data) to generate a model corresponding to the user's personalized voice. Finally, the input text information can be inferred based on the model to generate personalized voice. It can be seen from this that the training amount of this process in the related art is relatively large, and it needs to go through the process of first training and then fine-tuning, that is, the data processing amount is large and the efficiency is not high.

[0053] In view of this, the embodiment of the present application provides a personalized speech synthesis method, which constructs a text-independent audio model so that users can complete the customized timbre by recording fewer sentences, and generates personalized speech based on the constructed audio model reasoning, without the need for the process of first training and then fine-tuning, so that data processing efficiency is higher and applicability is better. It should be noted that the personalized speech synthesis method provided by the present application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), smart homes and other electronic devices that can perform personalized voice settings. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0054] For example, Figure 3 1 is a schematic diagram of the structure of an example of an electronic device 100 provided in an embodiment of the present application. Taking the electronic device 100 as a mobile phone as an example, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0055] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0056] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0057] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0058] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0059] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0060] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0061] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Figure 3 The structures of antenna 1 and antenna 2 in the electronic device 100 are only examples. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0062] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0063] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0064] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0065] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0066] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0067] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0068] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0069] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0070] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0071] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0072] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0073] It is also understandable that the above Figure 3 The electronic device shown is only an exemplary description when it is a mobile phone. If the electronic device is a tablet computer, PC, PDA, wearable device or other types of devices, the structure of the electronic device may include more Figure 3 The structure shown in the figure is less than Figure 3 More structures are shown in the figure, which are not limited here.

[0074] As can be seen from the above description, if the user wants to use the personalized speech synthesis function, it is necessary to customize the timbre (that is, register the speech synthesis function). The present application embodiment provides a setting process of the personalized speech synthesis function, which is described below by taking the setting in the scenario of the voice assistant function as an example. Figure 4As shown, the user clicks the setting icon on the desktop interface of the electronic device to enable the electronic device to enter the setting interface. In the setting interface, there are setting options for different functions, such as WLAN options, Bluetooth options, display and brightness options, smart assistant options, etc. When the user clicks the smart assistant option control 41, the electronic device can jump to the smart assistant setting interface. In the smart assistant setting interface, there are different smart functions, such as YOYO suggestions, negative one screen, smart text, smart search, smart voice, etc. When the user clicks the smart voice option control 42, the electronic device can jump to the smart voice setting interface. In the smart voice setting interface, there are different wake-up methods for the voice assistant, such as voice wake-up, power button wake-up, breath wake-up, headphone line control wake-up and Bluetooth device wake-up. The user can choose the corresponding wake-up method to wake up the voice assistant according to the needs. It can be understood that after the user selects the wake-up method in the smart voice setting interface, the voice assistant function is turned on, and then the personalized voice synthesis method of the present application can be used to synthesize personalized voice to output voice information and communicate with the user.

[0075] At the same time, in the smart voice setting interface, the broadcast tone can also be set. When the user clicks the broadcast tone option control 43, the electronic device can jump to Figure 5 The interface shown. On this interface, the electronic device provides different types of timbres for the user to choose, for example, YOYO (male voice), YOYO (female voice), etc. In addition, the electronic device also provides the function of customizing the timbre. When the user clicks the custom timbre control 51, the electronic device can jump to the custom timbre setting interface. In the custom timbre setting interface, the recorded timbres, such as timbre 1 and timbre 2, and the new timbre control 52 are displayed. When the user clicks the new timbre control 52, the electronic device can jump to the new timbre interface. In the new timbre interface, the guiding term for creating a personal voice is presented. If the user wants to record his own personalized voice at this time, he can click the next control 53 to trigger the voice recording. When the user clicks the next control 53, the electronic device can present a prompt to read a phrase. The user can read the corresponding phrase after clicking the start control 54, and the electronic device can record the user's voice. Optionally, when the user registers a custom timbre, the embodiment of the present application only needs to read N phrases (N≤2) to achieve it.

[0076] In some cases, if the user interferes or reads aloud abnormally during the recording process, resulting in failure of recording, the electronic device may display the following information: Figure 6 The prompt interface shown displays a prompt message of “recording failed, please re-record”, so the user can click the start control 54 again and then read the corresponding phrase.

[0077] In other cases, if the user has successfully recorded, the electronic device may display the following information: Figure 7 The interface shown in FIG. 1 shows a "play recording" control 71. The user clicks the "play recording" control 71 to listen to the content just recorded. If the user is not satisfied with the recorded content, the user can click the previous control 72 to re-enter the electronic device. Figure 5 The interface shown is used for recording. If the user is satisfied with the recorded content, the user can click the next control 73, and the electronic device enters the synthesized timbre interface, on which the electronic device can present a default synthesized phrase and a corresponding synthesized timbre. For example, the default synthesized phrase is "An alarm has been set for you at 8 o'clock tomorrow morning", and the corresponding synthesized timbre can be listened to through the "Audio" control 74. It can be understood that the synthesized timbre is the synthesized timbre obtained by the electronic device analyzing the sound of the above-mentioned user during recording, extracting the user's timbre characteristics, and then using the timbre characteristics to infer the synthesized phrase, that is, it can be understood as simulating the user's voice to read aloud the synthesized phrase. At this time, if the user is satisfied with the effect of the synthesized timbre, the user can click the completion control 75, that is, the custom timbre process is completed, so that subsequent users can experience the personalized voice synthesis function accordingly when using the voice assistant function, and the electronic device can convert the text message in reply to the user into a personalized voice for output.

[0078] It can be seen from the above scenario description that in the embodiment of the present application, when the user registers a customized tone, he only needs to read a few phrases to complete it, without having to read multiple sentences. The operation is relatively simple, which improves the user experience.

[0079] It is understandable that in the scenario of AI call function, users can also customize the timbre in a similar way as described above, so that the electronic device can convert the text information input by the user into a personalized voice and send it to the other party's electronic device. It is also understandable that in addition to the above-mentioned voice assistant function scenario and AI call function scenario, in other scenarios, such as short video dubbing scenario, the personalized voice synthesis function can also be used. After the user customizes the timbre, the electronic device can output the lines in the short video in the user's own voice to achieve the effect of personal dubbing.

[0080] In the above scenario, if the user clicks the completion control 75 to complete the custom timbre process, the electronic device can extract audio features from the sound recorded by the user to obtain the audio features corresponding to the user. If the user subsequently enters text information, the electronic device can generate personalized speech based on the constructed audio model and the audio features corresponding to the user by inferring the text information. Below we will introduce in detail the process of performing a personalized speech synthesis method based on a text-independent audio model in an embodiment of the present application.

[0081] First, the audio model of the embodiment of the present application needs to be trained to obtain a converged network model. In some embodiments, the training process of the audio model can be performed by a cloud server. Before training, the cloud server can obtain multiple sets of training data, wherein each set of training data can include a text message and a voice message, and the voice messages in different sets of training data can be recorded by different users. Exemplarily, the text message in a certain set of training data is "Today's maximum temperature is 25 degrees Celsius", and the voice message is the voice of a user reading "Today's maximum temperature is 25 degrees Celsius". For each set of training data, the cloud server can perform the following processing: Taking training data A as an example, the cloud server extracts features from the text information and voice information in training data A respectively, obtains semantic features (semantic) and acoustic features (acoustic), and then constructs a text-independent prompt based on the semantic features and acoustic features, and then decodes the constructed prompt to obtain an output result. In other embodiments, if the electronic device has the ability to train the model, the training process of the audio model can also be performed by the electronic device.

[0082] like Figure 8 As shown, the cloud server extracts features from text information, and the process of obtaining semantic features may include front-end processing, mapping processing (embedding), encoding processing (encoder) and duration stretching processing (LR). The front-end processing here may include text regularization processing, rhythm prediction processing and phonetic processing.

[0083] Among them, because some words are abbreviated or abbreviated in daily life, the cloud server needs to standardize these words, such as converting telephone, time, money, unit, symbol, mailbox, date, etc. into standardized words, that is, performing text regularization processing. For example, Sep.11th needs to be expanded to be written as September Eleventh in full spelling. Optionally, the cloud server can use regular expressions to perform text regularization processing on text information. Then, because some words need to pause or reread between words when reading aloud, if the pause is inaccurate, the reading will be incoherent and unnatural, and even affect the expression of the corresponding voice information. Therefore, the cloud server also needs to perform rhythm prediction processing on the text information. For example, after the rhythm prediction of the text "Today's minimum temperature is 15 degrees Celsius", the pause mode can be obtained as "Today#Minimum temperature#15#Celsius". Optionally, the cloud server can use a deep network to perform rhythm prediction here, and the deep network used is obtained by training with rhythmic text data. Next, because some characters are polyphonic, they have different pronunciations in different words, so the cloud server needs to perform phonetic processing on the text information to accurately determine the pronunciation of each character in the text information. Optionally, the cloud server can also use a deep network to convert the text information into pinyin to solve the problem of polyphonic characters. The deep network used is trained using polyphonic character data. Optionally, after the phonetic processing, the cloud server can obtain the phoneme information corresponding to the text information.

[0084] After the text information is processed by the front end, the cloud server can perform dimension mapping processing (embedding), encoding processing (encoder) and time stretching processing on the text information. Among them, embedding refers to the process of mapping high-dimensional data (such as text, pictures, audio) to low-dimensional space; encoder refers to the process of integrating the features of the input text information and converting the feature blocks of high-dimensional abstraction; time stretching processing refers to stretching the reading time of the text information, that is, predicting the time when each word (or phoneme) is read aloud, and adding the time information to the text information to obtain the total time frame information corresponding to the text information. For example, if the user reads a word for n seconds, then the total time frame information corresponding to the sentence containing 10 words is 10n seconds. Optionally, when the cloud server performs time stretching processing on the text information, the corresponding stretching time can be determined based on the voice information in the training data A, because the text information and voice information in the training data A correspond, then the voice time corresponding to the voice information can be used as the time to stretch the text information.

[0085] After the above processing, the cloud server can obtain the semantic features corresponding to the text information.

[0086] At the same time, the cloud server will also extract features from the voice information to obtain acoustic features. In other words, the cloud server can extract semantic features and acoustic features in parallel. Figure 8 The cloud server extracts features from the voice information, and the process of obtaining acoustic features may include audio encoding processing, multi-channel mapping processing (embedding) and multi-channel accumulation processing.

[0087] Among them, the cloud server can use an audio encoder to perform audio encoding processing on the voice information, and the obtained acoustic vector is the acoustic vector corresponding to the voice information. The acoustic vector here can be a multi-channel multi-frame feature vector, for example, it can be a feature vector of 8 channels. Of course, it can also be a feature vector of other numbers of channels. The embodiment of the present application does not limit this. Then, the cloud server can mask the feature vector of each channel in the acoustic vector and then embed it, that is, obtain a multi-channel embedding, for example, obtain an 8-channel embedding. Optionally, the embedding method of each channel feature vector can be acoustic_semb_i=embedding(mask(acoustic_i)), where the role of masking is to mask some frames in the multi-frame feature vector for subsequent reasoning generation to train the accuracy of reasoning. Next, the cloud server can perform weighted accumulation on the multi-channel embedding to obtain a feature vector of a channel, that is, to obtain the acoustic feature (acoustic). Optionally, the cloud server can be based on acoustic=∑ N i＝1 (w_i×acoustic_semb_i) calculates the acoustic features, where w_i is the weight coefficient corresponding to the channel and N is the number of channels, for example, N=8.

[0088] After the above processing, the cloud server can obtain the acoustic features corresponding to the voice information. It can be understood that since the cloud server can determine the length of the voice corresponding to the voice information when stretching the text information, the length of the semantic features finally obtained is equal to the length of the acoustic features.

[0089] After obtaining the semantic features and acoustic features, the cloud server can start to construct the text-independent prompt (the core part of the audio model). Figure 8, the process may include: firstly generating a random number m, and dividing the semantic feature and the acoustic feature into two parts according to the random number m, wherein m is less than the length of the semantic feature (or the acoustic feature), preferably, len(semantic) is the length of the semantic feature. Then the cloud server divides the semantic feature into two parts, prompt_se and target_se, and divides the acoustic feature into two parts, prompt_ac and target_ac, according to the random number m. Next, the cloud server fuses the divided prompt_se and target_se, prompt_ac and target_ac through the audio model to obtain the target prompt and target target. Optionally, target prompt = prompt_ac, target target = a×target_se+b×target_ac, where a and b are weight coefficients. It can be seen that the target prompt here is independent of the semantic feature, that is, it is independent of the text. Then the target prompt can be used as the prompt information of the subsequent decoder, so that the decoder adds timbre to the text information according to the timbre carried in the target prompt, that is, the timbre of the output voice is as close as possible to the timbre corresponding to the above-mentioned input voice information, and the target target can be used as the learning content of the audio model.

[0090] After obtaining the target prompt and the target target, the cloud server can input the target prompt and the target target into the decoder for decoding to obtain a speech output result, and then compare the features of the output result with the acoustic features of the above-mentioned input speech information, calculate the loss function, and adjust the parameter values ​​of the audio model according to the loss function, and finally obtain a converged audio model. Optionally, the input method of the decoder can include decoder_in=cat(target prompt, target target).

[0091] After the above process, the cloud server constructs the audio model required by the embodiment of the present application. If the user registers a custom timbre on the electronic device, the audio model can obtain the audio features of the custom timbre. For example, in the scenario of the voice assistant function, the audio model can combine the audio features and the queried text information to infer and generate personalized voice. In the scenario of the AI ​​call function, the audio model can combine the audio features and the text information input by the user to infer and generate personalized voice. In some embodiments, the process of generating personalized voice by audio model inference can still be executed on the cloud server, so this requires the electronic device to send the voice information and subsequent text information when the user customizes the timbre to the cloud server. In other embodiments, the process of generating personalized voice by audio model inference can also be executed on the electronic device, so this requires the cloud server to transplant the trained converged audio model to the electronic device. The following is a description of the execution of the cloud server as an example.

[0092] like Fig. 9 As shown, after the user registers for a custom timbre, the electronic device can send the voice information recorded during registration to the cloud server. The cloud server can extract audio features from the voice information to obtain the audio features corresponding to the user. Optionally, the cloud server can correspond the user's identification and audio features and store them in a feature library, where the user's identification can be the user name used by the user when registering on the electronic device, the device identification of the electronic device used, and other information, and the embodiment of the present application does not limit this. Then, if the cloud server receives the text information sent by the electronic device (which can be the text information input by the user or the text information queried by the electronic device background), the cloud server can first perform front-end processing such as text regularization processing, rhythm prediction processing, and phonetic processing on the text information, and then input the text information after the front-end processing and the audio features corresponding to the user into the audio model for reasoning, output personalized voice and return it to the electronic device. Optionally, the audio model can include an acoustic model and an audio decoder, the acoustic model is used to convert the text information into audio acoustic features (i.e., the process of constructing prompt) in combination with the audio features, and the audio decoder is used to convert the audio acoustic features into audio for output. It can be understood that, unlike the above-mentioned audio model training process, the text information and voice information in the training data are grouped in correspondence during the training process, while in the process of inference to generate personalized voice, the text information received by the cloud server changes in real time, and the audio features used are extracted when the user customizes the tone registration. The purpose is to output the real-time changing text information as personalized voice with the user's tone based on the user's audio features.

[0093] for Fig. 9 The embodiment shown, combined with Fig.10Describe the execution process of the cloud server: After receiving the voice information recorded when the user customizes the timbre registration, the cloud server can perform audio encoding processing, multi-channel mapping processing (embedding) and multi-channel accumulation processing on the voice information to obtain the acoustic features (acoustic) corresponding to the voice information as the audio features corresponding to the user, and store the audio features and the user's identification in the feature library in correspondence. In the case where the cloud server receives text information, the text information can be processed by the front end to obtain the phoneme information (phoneme) corresponding to the text information. After the text information is processed by the front end, the cloud server can perform dimension mapping processing (embedding), encoding processing (encoder) and time stretching processing on the text information (phoneme information) to obtain the semantic features (semantic) corresponding to the text information. Next, the cloud server can merge the audio features (acoustic) and semantic features (semantic) through the audio model, use the audio features (acoustic) as a prompt, input them into the decoder for processing, and output personalized voice. It can be understood that the cloud server can return the obtained personalized voice to the electronic device, which is output to the user through the voice assistant function or transmitted to the peer electronic device through the AI ​​call function.

[0094] When the cloud server performs time stretching on the text message, since there is no voice information corresponding to the text message to determine the stretching time, the cloud server can predict the stretching time based on the time prediction model. The time prediction model here can be a network model trained in parallel during the training of the audio model, that is, in the above Figure 8 In the process shown, the cloud server can obtain a duration prediction model based on the encoder result of the text information and prompt_ac through cross attention training. Therefore, in the process of inferring and generating personalized speech, the cloud server can use the duration prediction model, input audio features (acoustic) and encoder results, obtain the stretched duration, and then perform time stretching processing on the text information (or phoneme information).

[0095] It can be seen from the processing process of the above-mentioned audio model that the embodiment of the present application constructs an audio model that is independent of text during training. Prompt is used as prompt information, and any subsequent input of text information can infer the voice information corresponding to the timbre carried in the prompt. Therefore, when used in actual scenarios, the user only needs to record a shorter sentence when registering, and the user's audio features can be extracted as prompt input to the audio model to output personalized voice without the need for post-fine-tuning, which makes data processing more efficient.

[0096] In other embodiments, the above-mentioned process of generating personalized voice by audio model reasoning can be executed on an electronic device, wherein the technical principles of the process executed by the electronic device are similar to the technical principles when executed on a cloud server, and the specific process will not be repeated here.

[0097] It is understood that the software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. If the electronic device uses audio model reasoning to generate personalized voice, it should be executed in combination with the software system architecture of the electronic device. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device.

[0098] Fig.11 It is a software structure block diagram of the electronic device of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages, such as the Android application package (Android application package, APK).

[0099] like Fig.11 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, music, video, short message, etc. It may also include an APK integrated with an audio model, which can realize the functions of user-defined timbre registration and inference generation of personalized voice based on the user's audio features and input text information.

[0100] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0101] like Fig.11 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0102] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0103] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0104] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0105] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0106] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0107] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0108] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0109] When the electronic device executes the above-mentioned personalized speech synthesis method, Fig.12 is a flowchart of a personalized speech synthesis method provided in an embodiment of the present application, such as Fig.12 As shown, the method may include:

[0110] S101, when a speech synthesis function is enabled in an electronic device, obtaining text information that needs to be speech synthesized in the electronic device.

[0111] Among them, the text information includes text information input by the user and text information generated by the electronic device. The text information input by the user can be the information input by the user to reply to the other user in the scenario of the AI ​​call function, and the text information generated by the electronic device can be the information to be replied to the user retrieved by the electronic device in the scenario of the voice assistant function.

[0112] S102, input the text information and the audio features corresponding to the user into the audio model for speech synthesis, and output personalized speech.

[0113] The audio features corresponding to the user are features generated based on the voice information entered by the user when registering the speech synthesis function, and the audio model is a text-independent model obtained by training based on the first text information and the first voice information in the training data. The processing process of the audio model can be found in the description of the above embodiment, and will not be repeated here.

[0114] The above describes in detail an example of a personalized speech synthesis method provided by an embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0115] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0116] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0117] The electronic device provided in this embodiment is used to execute the above-mentioned personalized speech synthesis method, and thus can achieve the same effect as the above-mentioned implementation method.

[0118] In the case of an integrated unit, the electronic device may further include a processing module, a storage module and a communication module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device to execute stored program codes and data, etc. The communication module may be used to support the electronic device to communicate with other devices.

[0119] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0120] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 3 Device of the structure shown.

[0121] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the personalized speech synthesis method of any of the above embodiments.

[0122] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the personalized speech synthesis method in the above-mentioned embodiment.

[0123] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the personalized speech synthesis method in the above-mentioned method embodiments.

[0124] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0125] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0130] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A personalized speech synthesis method, characterized in that: The method comprises: When the speech synthesis function of the electronic device is turned on, obtaining text information that needs to be speech synthesized in the electronic device, the text information includes text information input by the user and text information generated by the electronic device; The text information and the audio features corresponding to the user are input into an audio model for speech synthesis, and personalized speech is output, wherein the audio features corresponding to the user are features generated based on the voice information entered by the user when registering for the speech synthesis function, and the audio model is a text-independent model obtained by training based on the first text information and the first voice information in the training data.

2. The method according to claim 1, characterized in that The step of inputting the text information and the audio features corresponding to the user into an audio model for speech synthesis and outputting personalized speech includes: Performing feature extraction on the text information to obtain semantic features corresponding to the text information; The audio model is used to perform speech synthesis on the semantic features and the audio features corresponding to the user, and the personalized speech is output.

3. The method according to claim 2, characterized in that The performing speech synthesis on the semantic features and the audio features corresponding to the user by using the audio model to output the personalized speech includes: The audio features corresponding to the user are used as timbre cues, and the semantic features and the audio features corresponding to the user are fused and decoded through the audio model to obtain and output the personalized voice.

4. The method according to claim 2 or 3, characterized in that: The extracting features of the text information to obtain semantic features corresponding to the text information includes: Performing front-end processing on the text information to obtain phoneme information corresponding to the text information, wherein the front-end processing includes at least one of text regularization processing, prosody prediction processing, and phonetic notation processing; The phoneme information is subjected to dimension mapping, encoding and time-stretching processing to obtain semantic features corresponding to the text information.

5. The method according to claim 4, characterized in that Before performing time-stretching processing on the phoneme information, the method further includes: The stretched duration corresponding to the phoneme information is predicted based on a preset duration prediction model, wherein the duration prediction model is a model trained according to the encoding result corresponding to the first text information and the acoustic features corresponding to the first speech information in the training data.

6. The method according to any one of claims 1 to 5, characterized in that The training method of the audio model includes: Performing front-end processing, dimension mapping processing, encoding processing, and time-stretching processing on the first text information to obtain semantic features corresponding to the first text information; Performing audio encoding processing, multi-channel mapping processing, and multi-channel accumulation processing on the first voice information to obtain acoustic features corresponding to the first voice information, wherein the data length of the semantic features is equal to the data length of the acoustic features; Generate a random number m, divide the semantic feature into a first sub-feature and a second sub-feature according to the random number m, and divide the acoustic feature into a third sub-feature and a fourth sub-feature; Using the third sub-feature as a first target feature, and generating a second target feature according to the second sub-feature and the fourth sub-feature; The first target feature and the second target feature are fused and decoded by the audio model to obtain an output result; According to the loss between the feature of the output result and the acoustic feature, the parameter value of the audio model is adjusted until the audio model converges.

7. The method according to claim 6, characterized in that The value range of the random number m is The L is the data length of the semantic feature.

8. The method according to any one of claims 1 to 7, characterized in that Before inputting the text information and the audio features corresponding to the user into the audio model for speech synthesis and outputting the personalized speech, the method further includes: Acquire the voice information entered by the user when registering for the speech synthesis function; The recorded voice information is subjected to audio encoding processing, multi-channel mapping processing and multi-channel accumulation processing to obtain audio features corresponding to the user.

9. The method according to claim 8, characterized in that The number of phrases corresponding to the input voice information is N≤2.

10. The method according to claim 8 or 9, characterized in that: After obtaining the audio feature corresponding to the user, the method further includes: The audio features corresponding to the user are associated with the user's identification and stored in a feature library.

11. The method according to any one of claims 1 to 10, characterized in that When the electronic device is in a scenario where a voice assistant is used, the text information is text information generated by the electronic device; when the electronic device is in a scenario where artificial intelligence (AI) is used for a call, the text information is text information input by the user.

12. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 11.

13. A server, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the server executes the method according to any one of claims 1 to 11.

14. The server according to claim 13, characterized in that: When the one or more programs are executed by the processor, the server further executes: The personalized voice is sent to the electronic device.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 11.

Images