Speech synthesis method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a speech synthesis method, an electronic device and a storage medium, in the speech synthesis method, multiple paragrams of a user are obtained from audio recorded by the user, a paragraph library of the user is updated based on the multiple paragrams of the user, and when speech synthesis is carried out, the speech synthesis efficiency is improved. And converting the to-be-converted text into a synthetic audio which has user sound characteristics and contains user side language characteristics based on the updated side language library. Therefore, the side language of the user can be synthesized in the synthesized audio, and the naturalness and anthropomorphic degree of the synthesized audio are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, electronic device and storage medium. Background Art

[0002] Text-to-speech (TTS) is a technology that converts text into audio with specified user voice characteristics. For example, by recording the audio of a user reading a specified text, and extracting the user voice characteristics such as loudness, pitch and timbre from the audio, when performing speech synthesis on the text to be converted, the text to be converted is converted into synthesized audio with the specified user voice characteristics based on the user voice characteristics extracted from the audio.

[0003] When users express themselves in daily life, they will involuntarily use paralanguage such as hissing, hesitation, sighing, coughing, laughing, inhaling, dragging out sounds, inversions, smacking lips, stuttering, correction of slips of the tongue, and emphasis. The speech synthesis system is unable to capture these paralanguages, resulting in the inability to synthesize these paralanguages ​​into the synthesized audio when performing speech synthesis on the converted text. This leaves room for improvement in the naturalness and anthropomorphism of the synthesized audio. Summary of the invention

[0004] In order to solve the problem that there is room for improvement in the naturalness and anthropomorphism of synthesized audio, the embodiments of the present application provide a speech synthesis method, an electronic device and a storage medium.

[0005] A first aspect of an embodiment of the present application provides a speech synthesis method, the speech synthesis method comprising: obtaining a first text to be synthesized; synthesizing the first text into first synthesized audio data through an acoustic model of an electronic device and based on a first paralanguage library of a first user, wherein the first synthesized audio data comprises: first-category audio data corresponding to text information in the first text and second-category audio data corresponding to the paralanguage of the first user.

[0006] Based on the above solution, by synthesizing multiple paralanguages ​​in the first paralanguage library of the first user into the first synthesized audio data, the naturalness and anthropomorphism of the synthesized audio can be improved.

[0007] It can be understood that the first type of audio data can be text features of the first text, and the text features do not contain features corresponding to punctuation marks of the first text. The first text is also described as the text to be converted below. The second type of audio data can be non-text features of the first text, and the non-text features include the user's paralinguistic features, such as paralinguistic features corresponding to paralinguistic features such as hissing, hesitation, sighing, coughing, laughing, inhaling, dragging, inversion, smacking, stuttering, correction of slips of the tongue, and emphasis.

[0008] In some optional instances, after the first text is synthesized into the first synthesized audio data, the synthesized audio corresponding to the first synthesized audio data can be played based on the audio module of the electronic device, and the synthesized text corresponding to the synthesized audio can be displayed on the display interface of the electronic device, and the synthesized text includes the user's paralanguage compared to the first text.

[0009] In some optional instances, the acoustic model may be a trained neural network model, described below as a neural encoding language model, such as the VALLE model (a speech synthesis model).

[0010] In a possible implementation of the first aspect, the first paralanguage library of the first user is updated in the following manner: obtaining target update frequency selection information selected by the first user; performing voice interaction with the first user based on the voice interaction system, and obtaining audio to be processed by the first user during the voice interaction process; obtaining multiple paralanguages ​​of the first user from the audio to be processed; and updating the first paralanguage library of the first user based on the target update frequency selection information and the multiple paralanguages ​​of the first user.

[0011] In some optional examples, multiple candidate update frequency selection information may be displayed on the setting interface of the electronic device, and the target update frequency selection information may be any one of the multiple candidate update frequency selection information selected by the user. The candidate update frequency selection information may include real-time update, weekly update, monthly update, and quarterly update, and the target update frequency selection information may be any one of real-time update, weekly update, monthly update, and quarterly update.

[0012] In some optional examples, a method of obtaining multiple paralanguages ​​of the first user from the audio to be processed may be: using a speech encoder to encode the audio to be processed to obtain paralanguage features, wherein the speech encoder may be any speech encoder that can convert audio of any language into a machine-recognizable language (such as binary code), for example, the speech encoder may be a speech recognition pre-trained model.

[0013] In some optional instances, if the target update frequency selection information is real-time update, the paralanguage features can be updated in real time based on the weights set for the time sequence of the audio in the audio to be processed, and the paralanguage features can be decoded using a speech decoder to obtain multiple paralanguages, which are described below as N rounds of dialogue results.

[0014] In the embodiment of the present application, by obtaining the user's unprocessed audio, that is, the recorded audio, during the voice interaction process, compared to obtaining the audio recorded when the user reads a specified text, there is no need for the user to read the specified text, and the text restriction can be eliminated. Not only can the diversity of the audio be improved, but also the freedom of user expression can be improved, thereby improving the naturalness and anthropomorphism of the synthesized audio.

[0015] In a possible implementation of the first aspect above, the target update frequency selection information includes any one of real-time update, weekly update, monthly update and quarterly update.

[0016] In some optional instances, real-time updates can be displayed on the settings interface of the electronic device in the form of real-time update controls, weekly updates can be displayed on the settings interface of the electronic device in the form of weekly update controls, monthly updates can be displayed on the settings interface of the electronic device in the form of monthly update controls, and quarterly updates can be displayed on the settings interface of the electronic device in the form of quarterly update controls.

[0017] For example, at least two of a real-time update control, a weekly update control, a monthly update control, and a quarterly update control may be displayed simultaneously on a setting interface of an electronic device.

[0018] In a possible implementation of the first aspect, the first paralanguage library of the first user is updated in the following manner: a bar graph corresponding to the first paralanguage library of the first user is obtained, where each rectangular box in the bar graph represents a paralanguage; and the height of the rectangular box represents the frequency of the paralanguage; and adjustment information for part or all of the rectangular boxes in the bar graph is obtained, and the first paralanguage library of the first user is updated based on the adjustment information.

[0019] In some optional examples, when updating the first paralanguage library of the first user, the first paralanguage library of the first user can be displayed in a visual form on the display interface of the electronic device, so that the first user or the second user can manually adjust the frequency of some or all paralanguages ​​in the first paralanguage library. For example, the first user or the second user can increase or decrease the frequency of some or all paralanguages ​​by sliding up and down.

[0020] In the embodiment of the present application, multiple paralanguages ​​of the user in the paralanguage library are displayed in the form of data visualization on the display interface of the electronic device for manual update by the user, so that the user can edit the frequency of the paralanguage independently, thereby improving the naturalness and anthropomorphism of the synthesized audio.

[0021] In a possible implementation of the first aspect, obtaining adjustment information for part or all of the rectangular boxes in the bar graph, and updating the first secondary language library of the first user based on the adjustment information includes: obtaining adjustment information for part or all of the rectangular boxes in the bar graph by the first user; and updating the first secondary language library of the first user based on the bar graph after the part or all of the rectangular boxes are adjusted.

[0022] In a possible implementation of the first aspect, the electronic device further stores a second paralanguage library for the acoustic model, wherein the second paralanguage library is applicable to a second user, and the second user is a user associated with the first user.

[0023] In a possible implementation of the first aspect above, the association relationship between the first user and the second user includes a family member relationship.

[0024] A first aspect of an embodiment of the present application provides a method for processing an acoustic model, comprising: obtaining reference synthetic audio data, sample text corresponding to the reference synthetic audio data, and a sample paralanguage library; using the sample text and the sample paralanguage library as inputs of a neural network model to obtain predicted synthetic audio data output by the neural network model; and adjusting parameters of the neural network model based on the reference synthetic audio data and the predicted synthetic audio data to obtain an acoustic model.

[0025] In some optional instances, the acoustic model may be a trained neural network model, described below as a neural encoding language model, such as the VALLE model (a speech synthesis model).

[0026] In some optional examples, the sample text may be converted to phonemes and input into a neural coding language model, and the paralinguistic features in the sample paralinguistic library may be extracted and input into the neural coding language model. The neural coding language model may encode the input phonemes and paralinguistic features, and input the encoded features (such as binary codes representing the phonemes and paralinguistic features) obtained by encoding into a speech decoder, and the speech decoder may be used to decode the encoded features to obtain predicted synthesized audio data having user voice characteristics and including user paralinguistic characteristics.

[0027] In a third aspect, the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the speech synthesis method mentioned in the present application.

[0028] In a fourth aspect, the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the speech synthesis method mentioned in the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 According to some examples of the present application, a schematic diagram of an application scenario is shown;

[0030] Figure 2 According to some examples of the present application, a block diagram of a method for performing speech synthesis based on a speech synthesis system is shown;

[0031] Figure 3 According to some examples of the present application, a schematic diagram of a framework flow of speech synthesis is shown;

[0032] Figure 4 According to some examples of the present application, another framework flow diagram of speech synthesis is shown;

[0033] Figure 5 According to some examples of the present application, a flow chart of a speech synthesis method is shown;

[0034] Figure 6 According to some examples of the present application, a block diagram of performing speech synthesis on a text to be converted is shown;

[0035] Figure 7 According to some examples of the present application, a schematic diagram of interface comparison between a text to be converted and a synthesized text is shown;

[0036] Figure 8 According to some examples of the present application, a flowchart of a method for registering a wake-up audio is shown;

[0037] Fig. 9 According to some examples of the present application, a schematic diagram of interface jump of an electronic device is shown;

[0038] Fig.10 According to some examples of the present application, a flowchart of another method for registering a wake-up audio is shown;

[0039] Fig.11 According to some examples of the present application, a flow chart of a method for obtaining and updating a user's paralanguage library is shown;

[0040] Fig.12 According to some examples of the present application, a schematic diagram of interface jump of an electronic device is shown;

[0041] Fig.13 According to some examples of the present application, a flow chart of a method for updating a user's paralanguage library based on a target update frequency is shown;

[0042] Fig.14 According to some examples of the present application, a flowchart of a method for updating a user's paralanguage library is shown;

[0043] Fig.15 According to some examples of the present application, a block diagram of a method for processing audio to be processed is shown;

[0044] Fig.16 According to some examples of the present application, a block diagram of interaction between a user and an electronic device is shown;

[0045] Fig.17 According to some examples of the present application, a schematic diagram of an interface for adjusting the frequency of a paralanguage in a paralanguage library is shown;

[0046] Fig.18 According to some examples of the present application, a block diagram of a method for updating a user's paralanguage library is shown;

[0047] Fig.19 According to some examples of the present application, a hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0048] The illustrative embodiments of the present application include, but are not limited to, a speech synthesis method, an electronic device, and a storage medium.

[0049] It can be understood that the speech synthesis method provided in the embodiments of the present application can be used in the fields of smart mobile terminals, smart homes, and smart vehicle-mounted devices. Specifically, the speech synthesis method in the embodiments of the present application can be applied to any implementable entity with an emotional speech synthesis function, such as a smart phone, a smart speaker, a smart vehicle-mounted device, and a smart robot.

[0050] It is understood that the paralanguage mentioned in the embodiments of the present application may refer to a phenomenon that has no substantial meaning and is only used to modify the user's expression, such as hissing, hesitation, sighing, coughing, laughing, inhaling, dragging, inversion, smacking lips, stuttering, correction of slips of the tongue, emphasis, etc.

[0051] like Figure 1 As shown, in some specific implementations, the first electronic device 110 can send the neural network model 111 to the second electronic device 120, so that the speech synthesis system in the second electronic device 120 can train the neural network model 111 to obtain an acoustic model that can perform speech synthesis on the input text. The first electronic device 110 can be a cloud server, and the second electronic device 120 can be any implementable terminal device such as a smart phone, a smart speaker, a smart car device, and a smart robot.

[0052] For ease of understanding, the method for performing speech synthesis based on a speech synthesis system mentioned in some embodiments is introduced below, assuming that the first electronic device is a cloud server and the second electronic device is a smart phone.

[0053] like Figure 2As shown, the smart phone 210 may include a speech synthesis system 211 , and the speech synthesis system 211 may include a collection module 212 , a model building module 213 and a synthesis module 214 .

[0054] The cloud server 220 can store historical audio and neural network models recorded by different users in different historical time periods.

[0055] Among them, the acquisition module 212 can record the audio of the user reading the specified text when the user is in a specified recording environment (such as an absolutely quiet environment) reading the specified text. For example, when the user starts the intelligent voice application of the smart phone, the audio recording information can be displayed on the display interface of the smart phone, such as "Please read the following content aloud in a quiet environment: XXX", and when the user reads "XXX", the user's audio is recorded and the audio is quality-checked. For audio segments that fail the quality inspection due to unclear pronunciation, long word breaks, etc., the sentences corresponding to the audio segments can be sent to the user, requiring the user to re-record, and the audio segments re-recorded by the user are quality-checked until the audio that passes the quality inspection is obtained and sent to the model building module 213.

[0056] When the model building module 213 builds the exclusive acoustic model of the corresponding user, the cloud server 220 can send the user's historical audio and neural network model to the model building module 213. Among them, the historical audio can be audio that has passed the quality inspection.

[0057] The model building module 213 can pre-process the audio sent by the acquisition module 212 and the audio sent by the cloud server 220, such as reducing noise, removing silent segments, increasing the volume, etc., and train the neural network model based on the pre-processed audio to obtain a user-specific acoustic model, and send the user-specific acoustic model to the synthesis module 214. Among them, the user-specific acoustic model can convert the text to be converted into a synthesized audio with the user's sound characteristics such as loudness, pitch and timbre.

[0058] The synthesis module 214 may convert the text to be converted into synthesized audio with different user voice characteristics based on the exclusive acoustic models of different users.

[0059] However, the above-mentioned speech synthesis system cannot capture the user's paralanguage, resulting in the inability to synthesize the user's paralanguage into the synthesized audio when performing speech synthesis on the text, leaving room for improvement in the naturalness and anthropomorphism of the synthesized audio.

[0060] In order to solve the above problems, some embodiments of the present application provide a speech synthesis method, which obtains multiple sub-languages ​​of the user from the user's recorded audio, and updates the user's sub-language library based on the user's multiple sub-languages. When performing speech synthesis, the text to be converted is converted into a synthesized audio having the user's voice characteristics and including the user's sub-language characteristics based on the updated sub-language library. In this way, the user's sub-language can be synthesized in the synthesized audio, improving the naturalness and anthropomorphism of the synthesized audio.

[0061] It can be understood that the method of obtaining the user's multiple paralanguages ​​from the user's recorded audio can be: encoding the recorded audio based on a speech encoder to obtain text features and non-text features in the recorded audio, wherein the non-text features include the user's paralanguage features. After obtaining the non-text features, the non-text features are decoded based on a language decoder to obtain the user's multiple paralanguages. The speech encoder can be any speech encoder that can convert audio of any language into a machine-recognizable language (such as binary code), and the language decoder can be any language decoder that converts a machine-recognizable language into text, wherein the speech encoder can be a speech recognition pre-trained model, and the speech decoder can be a HiFi-GAN model.

[0062] In some optional examples, the method of obtaining the user's recorded audio may be: when a user's wake-up operation on the speech synthesis system in the electronic device is detected, a speech interaction is performed with the user based on the speech interaction system in the electronic device, and during the speech interaction process, the user's interaction audio is recorded as the user's recorded audio. By obtaining the user's recorded audio during the speech interaction process, compared to obtaining the audio recorded when the user reads a specified text, the user does not need to read the specified text, and the text restriction can be eliminated, which can not only improve the diversity of the audio, but also improve the freedom of the user's expression, thereby improving the naturalness and anthropomorphism of the synthesized audio.

[0063] It can be understood that in some optional examples, the method of updating the user's secondary language library can be: based on the extracted multiple secondary languages ​​of the user, directly and automatically updating the user's secondary language library.

[0064] It can be understood that in some other optional examples, the method of updating the user's paralanguage library can be: displaying the user's multiple paralanguages ​​in the paralanguage library in the form of data visualization (such as a bar graph) on the display interface of the electronic device, so that the user can manually adjust the frequency of some or all of the paralanguages ​​in the paralanguage library, such as reducing the frequency of sighing, reducing the frequency of lip smacking, etc., to achieve manual update of the user's paralanguage library. By displaying the user's multiple paralanguages ​​in the paralanguage library in the form of data visualization on the display interface of the electronic device for manual update by the user, the user can edit the frequency of the paralanguage independently, thereby improving the naturalness and anthropomorphism of the synthesized audio.

[0065] In some optional examples, a paralanguage library sharing group can be created, and the paralanguage libraries of different users can be stored in the paralanguage library sharing group. When performing speech synthesis, the target user selected by the user from multiple candidate users displayed on the display interface of the electronic device can be obtained, and based on the paralanguage library corresponding to the target user in the paralanguage library sharing group, the text to be converted is converted into a synthesized audio having the voice characteristics of the target user and including the paralanguage characteristics of the target user. In this way, by creating a paralanguage library sharing group, the electronic device can perform speech synthesis based on the paralanguage library corresponding to the target user selected by the user, which can enrich the diversity of the synthesized audio.

[0066] The following is an introduction to the process of synthesizing audio mentioned in the embodiments of the present application. Figure 3 A schematic diagram of a speech synthesis framework flow is shown.

[0067] like Figure 3 As shown, the user can wake up the intelligent voice application on the electronic device based on the pre-registered wake-up audio. After the intelligent voice application on the electronic device is woken up, the content that needs to be set by the user can be displayed on the display interface, such as "whether to use personalized voice", "whether to update in real time", etc. Specifically, after the intelligent voice application is woken up, "whether to use personalized voice" can be displayed on the display interface, and when the user sets to use personalized voice, "whether to update in real time" can be displayed on the display interface.

[0068] When the user sets to use personalized voice, the background interaction system (ie, voice interaction system) of the electronic device interacts with the user through voice, and sends the user's real-time interaction audio to the personalized update module of the electronic device.

[0069] When the user sets real-time update, the personalized update module can call the background storage, that is, call the user habit library (that is, the user's paralanguage library) stored in the local memory, and obtain the user's real-time interactive audio sent by the background interaction system (that is, the voice interaction system) to update the user's paralanguage library and obtain the user habit copy (that is, the user's paralanguage library after real-time update). Then, the speech synthesis system of the electronic device can perform speech synthesis on the text to be converted in the background system based on the user habit copy to obtain synthesized audio.

[0070] When the user sets a scheduled update, the speech synthesis system of the electronic device can call the background storage, that is, call the user habit library (that is, the user's paralanguage library) stored in the local memory, and perform speech synthesis on the text to be converted in the background interaction system based on the user's paralanguage library to obtain synthesized audio.

[0071] Figure 4 Another framework flow chart of speech synthesis is shown, Figure 4 As shown, the user can wake up the intelligent voice application at different scene entrances based on the pre-registered wake-up audio, where the intelligent voice application can be "Xiaoyi Assistant", and the scene entrance can include scene 1 entrance, scene 2 entrance...scene N entrance. The scene entrance can be within the preset range of electronic devices such as smart phones, smart speakers, smart car devices and smart robots.

[0072] Then, the electronic device can record the user's wake-up audio and register the user's wake-up audio in the background. The user can then wake up the dialogue system (i.e., voice interaction system) based on the wake-up audio, and conduct voice interaction with the user based on the dialogue system (i.e., voice interaction system), and record the user's real-time interactive audio to update the paralanguage habit library in the paralanguage library sharing group (i.e., the user's paralanguage library). Among them, home users can share the paralanguage habit library in the paralanguage library sharing group (i.e., the user's paralanguage library) based on different scene entrances, and can also update some or all of the paralanguage habits in the paralanguage library sharing group by editing some or all of the paralanguage habits in the paralanguage library sharing group (i.e., the frequencies of some or all of the paralanguages ​​in the user's paralanguage library).

[0073] After the user wakes up the dialogue system (i.e., the voice interaction system) based on the wake-up audio, the dialogue system (i.e., the voice interaction system) can wake up the synthesis system (i.e., the voice synthesis system), and the synthesis system (i.e., the voice synthesis system) can perform voice synthesis on the text to be converted based on the updated paralanguage habit library (i.e., the user's paralanguage library) to obtain voice (i.e., synthesized audio).

[0074] The following is a detailed introduction to the method based on synthetic audio mentioned in the embodiments of the present application. Figure 5A flow chart of a speech synthesis method is shown. The method can be executed by electronic devices such as smart phones, smart speakers, smart car devices, and smart robots. Figure 5 As shown, the speech synthesis method may include:

[0075] 501: A user wake-up operation for an intelligent voice application is detected, and a plurality of candidate users are displayed on a display interface of the electronic device.

[0076] It can be understood that the wake-up operation can be an operation in which the user sends a registered wake-up audio within a preset range of the electronic device to wake up the intelligent voice application. The wake-up operation can wake up the voice interaction system and the voice synthesis system.

[0077] It is understood that when it is detected that the user sends a registered wake-up audio within the preset range of the electronic device to wake up the intelligent voice application, multiple family users, that is, multiple candidate users for the user to select, can be displayed on the display interface of the electronic device. In some optional instances, the candidate user can be displayed on the display interface of the electronic device with the avatar uploaded by the user based on the upload avatar control when registering the wake-up audio as a display mark. Each display mark corresponds to a candidate user control.

[0078] 502: In response to a user's determined user operation on a plurality of candidate users, speech synthesis is performed on the text to be converted based on a paralanguage library of a target user corresponding to the determined user operation to obtain synthesized audio.

[0079] It can be understood that the determined user operation can be to trigger any one of the multiple candidate user controls corresponding to the multiple display marks. When the electronic device responds to the user's determined user operation for multiple candidate users, it can obtain the target user corresponding to the determined user operation and obtain the sub-language library corresponding to the user. Then, the converted text can be speech synthesized based on the sub-language library of the target user to obtain synthesized audio.

[0080] like Figure 6 As shown, the text to be synthesized (i.e., the text to be converted) and the paralanguage library of the target user can be used as prompts respectively. For example, the text to be synthesized is used as a text prompt, and the paralanguage library of the target user is used as a user habit prompt, and they are sent to a pre-trained neural encoding language model, such as the VALLE model (a speech synthesis model).

[0081] In some optional instances, the text to be synthesized (i.e., the text to be converted) can be converted to phonemes and input into a neural coding language model. For example, the text to be synthesized (i.e., the text to be converted) "Okay, let me tell you a joke now" can be converted into phonemes "haode,xianzaijiugeinijianggexiaohuaba" and input into a neural coding language model. The text to be synthesized can be a text pre-set by an electronic device. For example, it can be pre-set that when receiving the user's interactive audio "I am a little bored now", the text to be synthesized "Okay, let me tell you a joke now" is set.

[0082] At the same time, the paralanguage library of the target user can be feature integrated, that is, the paralanguage features in the paralanguage library of the target user can be extracted and input into the neural encoding language model.

[0083] The neural coding language model can encode the input phonemes and paralinguistic features, and input the encoded features (such as binary codes representing phonemes and paralinguistic features) into the speech decoder, and use the speech decoder to decode the encoded features to obtain synthetic audio with user voice characteristics and user paralinguistic characteristics.

[0084] In some optional instances, the neural encoding language model can be trained in the following manner: obtaining reference synthetic audio data, sample text corresponding to the reference synthetic audio data, and a sample paralanguage library; using the sample text and the sample paralanguage library as inputs to the neural network model to obtain predicted synthetic audio data output by the neural network model; and adjusting the parameters of the neural network model based on the reference synthetic audio data and the predicted synthetic audio data to obtain an acoustic model.

[0085] 503: Play the synthesized audio based on the audio module of the electronic device, and display the synthesized text corresponding to the synthesized audio on the display interface of the electronic device.

[0086] In some optional examples, the synthesized audio can be played through an audio module of the electronic device, such as a speaker, also called a loudspeaker. At the same time, the synthesized text corresponding to the synthesized audio can be displayed on the display interface of the electronic device. Compared with the text to be converted, the synthesized text contains the user's paralanguage, such as pauses, dragging sounds, etc.

[0087] Figure 7 A schematic diagram showing a comparison of interfaces of a text to be converted and a synthesized text is shown, such as Figure 7 As shown, the text to be converted can be "In Haidian District, Beijing, the weather is sunny, the temperature is 23 degrees to 29 degrees, suitable for going out for fun", and the synthesized text can be "In Haidian District, Beijing, [pause] uh, the weather is sunny, the temperature [drag sound] 23 degrees to 29 degrees, suitable for going out for fun". Figure 7It can be seen that the synthesized text has the user's paralanguage characteristics compared to the text to be converted, and the synthesized audio corresponding to the synthesized text is more natural and more human.

[0088] The following is a detailed introduction to the registration method of the wake-up audio mentioned in the embodiment of the present application. Figure 8 The figure shows a flowchart of a method for registering a wake-up audio signal, which can be executed by an electronic device such as a smart phone, a smart speaker, a smart vehicle-mounted device, and a smart robot. Figure 8 As shown, the registration method of the wake-up audio may include:

[0089] 801: When a user's intelligent voice registration operation for an intelligent voice application is detected, a recording reminder message is displayed on a display interface of the electronic device.

[0090] It can be understood that the intelligent voice registration operation can be to start the intelligent voice registration function in the intelligent voice application, such as touching the intelligent voice registration control on the display interface of the electronic device, long pressing the lock screen key of the electronic device, double-clicking the display interface of the electronic device, etc. Among them, the intelligent voice application can include a speech synthesis system and a speech interaction system.

[0091] In some optional examples, recording reminder information may be displayed on the display interface of the electronic device, and recording reminder information, a confirmation registration control, and a rejection registration control may also be displayed on the display interface of the electronic device.

[0092] For example, Fig. 9 a shows a first setting interface 910 of an electronic device 900, which may include an intelligent voice registration control 911 and an exit intelligent voice registration control 912. If it is detected that the user clicks the intelligent voice registration control 911 on the first setting interface 910 of the electronic device 900, it can jump to the following Fig. 9 b, and can display a recording reminder message 921 “Hi, talking to Xiaoyi more can help Xiaoyi learn to be more like you. Your recorded audio is completely saved locally and nothing will be retained in the cloud”, and can also display a confirmation registration control 922 and a rejection registration control 923 on the second setting interface 920 of the electronic device 900.

[0093] In an embodiment of the present application, by displaying recording reminder information on the display interface of the electronic device so that the user can confirm or refuse to register, it is possible to avoid the user's audio being mistakenly collected due to the user accidentally starting the intelligent voice registration function.

[0094] 802: In response to the user's confirmation registration operation for recording reminder information, multiple candidate update frequency selection information is displayed on a display interface of the electronic device.

[0095] In some optional examples, when a user's intelligent voice registration operation for an intelligent voice application is detected and a recording reminder message is displayed on the display interface of the electronic device, the confirmed registration operation may be an operation of not exiting the intelligent voice registration within a preset time period. Fig. 9 As shown, if the user is not detected to click the exit intelligent voice registration control 912 on the first setting interface 910 of the electronic device 900 within the preset time period, the user can respond to the confirmation registration operation of the user for the recording reminder information, and jump to the following Fig. 9 c, and multiple candidate update frequency selection information can be displayed on the third setting interface 930 of the electronic device 900.

[0096] In some optional examples, when a user's intelligent voice registration operation for an intelligent voice application is detected, a recording reminder message, a confirmation registration control, and a rejection registration control are displayed on the display interface of the electronic device, and the confirmation registration operation may be to touch the confirmation registration control on the display interface of the electronic device. Fig. 9 As shown, if it is detected that the user clicks the confirmation registration control 922 on the second setting interface 920 of the electronic device 900 within a preset time period, the user's confirmation registration operation for the recording reminder information can be responded to, and multiple candidate update frequency selection information can be displayed on the third setting interface 930 of the electronic device 900.

[0097] It can be understood that the candidate update frequency selection information may include at least two of a real-time update control, a weekly update control, a monthly update control, and a quarterly update control. Fig. 9 As shown, when the user is not detected touching the exit intelligent voice registration control 912 on the first setting interface 910 of the electronic device 900 within the preset time period, or when the user is detected touching the confirmation registration control 922 on the second setting interface 920 of the electronic device 900 within the preset time period, it can jump to the Fig. 9 c, and candidate update frequency selection information such as a real-time update control 931, a weekly update control 932, a monthly update control 933, and a quarterly update control 934 can be displayed on the third setting interface 930 of the electronic device 900.

[0098] In an embodiment of the present application, by displaying multiple candidate update frequency selection information on the display interface of the electronic device for the user to choose, the timeliness of the registered wake-up audio can be improved to avoid subsequent changes in the user's voice characteristics. For example, for adolescents, their voice characteristics will change significantly in a short period of time, resulting in the user's wake-up audio being unable to wake up the intelligent voice application.

[0099] 803: In response to the user's operation of determining the update frequency for a plurality of candidate update frequency selection information, displaying a user registration information control and a wake-up audio recording control on a display interface of the electronic device.

[0100] It can be understood that the operation of determining the update frequency can be to trigger any one of the multiple candidate update frequency controls. When the electronic device responds to the user's operation of determining the update frequency for multiple candidate update frequency selection information, it can obtain the target update frequency corresponding to the operation of determining the update frequency, that is, the update frequency corresponding to any one of the triggered candidate update frequency controls. The electronic device can update the registered wake-up audio based on the target update frequency to avoid the subsequent change in the user's voice characteristics, resulting in the user's wake-up audio failing to wake up the intelligent voice application.

[0101] It can be understood that the user registration information control may include a control for uploading an avatar and / or a control for setting a user identity (IDentity, ID).

[0102] In some optional embodiments, such as Fig. 9 As shown, when it is detected that the user clicks on any one of the multiple candidate update frequency controls on the third setting interface 930 of the electronic device 900, it can jump to the following Fig. 9 d, and can display an upload avatar control 941, a set user ID control 942, a wake-up audio recording control 943, and a wake-up text 944 on the fourth setting interface 940 of the electronic device 900.

[0103] In some optional instances, the user ID set by the user ID control can represent the user's identity information. The wake-up audio based on the user ID can wake up the intelligent voice applications of multiple electronic devices in multiple scenarios, such as in-car navigation broadcasts, smartphone voice interactions, etc., as long as the electronic devices in each scenario are logged in to the user ID.

[0104] 804: In response to a wake-up audio recording operation of a user on a wake-up audio recording control, storing the user's recorded audio.

[0105] It can be understood that the wake-up audio recording operation can be a long press to wake up the audio recording control.

[0106] In some optional embodiments, such as Fig. 9As shown, when it is detected that the user long presses the wake-up audio recording control 943 on the third setting interface 930 of the electronic device 900, the user's interactive audio can be recorded as the user's recorded audio, and the audio of the user reading the wake-up text can also be recorded as the user's recording, and the recorded user's recorded audio is stored in the local storage. Specifically, after the recorded user's recorded audio is stored in the local storage, it can jump to the following Fig. 9 The fifth setting interface 950 shown in FIG. 5 may display a display mark 951 on the fifth setting interface 950 of the electronic device 900 . The display mark 951 may specifically be an avatar uploaded by the user based on the upload avatar control.

[0107] In an embodiment of the present application, by storing the user's recorded audio in a local storage device instead of uploading it to a cloud server for storage, leakage of the recorded audio can be avoided and the security of the user's voice characteristic information can be protected.

[0108] The following is an introduction to the registration method of the wake-up audio mentioned above. Fig.10 A flowchart of another method for registering a wake-up audio is shown. Fig.10 As shown, the registration method of the wake-up audio may include:

[0109] 1001: Confirm to use personalized voice.

[0110] As mentioned above, the intelligent voice registration control can be displayed on the display interface of the electronic device for the user to determine to use the personalized voice. The user can determine to use the personalized voice by touching the intelligent voice registration control on the display interface of the electronic device. The personalized voice can also be determined by long pressing the lock screen key of the electronic device, double-clicking the display interface of the electronic device, and other operations.

[0111] 1002: Determine whether to agree to the agreement. If yes, go to step 1003, otherwise end.

[0112] As mentioned above, the recording reminder message "Hi, talking to Xiaoyi more can help Xiaoyi learn to be more like you. Your recorded audio is completely saved locally and nothing will be retained in the cloud", the confirmation registration control, the rejection registration control and the exit smart voice registration control can be displayed on the display interface of the electronic device. If the user does not click the exit smart voice registration control on the display interface of the electronic device within the preset time period, or the user clicks the confirmation registration control on the display interface of the electronic device within the preset time period, it can indicate that the user agrees to the agreement, that is, agrees to record the wake-up audio.

[0113] 1003: Set the update method.

[0114] As mentioned above, multiple candidate update frequency selection information such as a real-time update control, a weekly update control, a monthly update control, and a quarterly update control can be displayed on the display interface of the electronic device for the user to set the update frequency of the wake-up audio. The user can touch any one of the multiple candidate update frequency controls to set the update frequency of the wake-up audio to the update frequency corresponding to the any one of the candidate update frequency controls.

[0115] 1004: Determine whether to register. If so, go to step 1005, otherwise, end.

[0116] In some optional instances, the user can agree to register by touching any one of the multiple candidate update frequency controls, that is, agree to upload the avatar, set the user ID, and further confirm that they agree to record the wake-up audio.

[0117] 1005: Enter the voiceprint registration page.

[0118] It can be understood that when entering the voiceprint registration page, the user can read aloud the wake-up text corresponding to the wake-up audio and register the user's voiceprint, so that subsequent electronic devices can distinguish the user's voice through the voiceprint corresponding to the wake-up audio.

[0119] As mentioned above, the upload avatar control, the setting user ID control, the wake-up audio recording control and the wake-up text can be displayed on the display interface of the electronic device. The user long presses the wake-up audio recording control and reads the wake-up text.

[0120] The following is a detailed description of the method for obtaining the user's secondary language and updating the user's secondary language library mentioned in the embodiment of the present application. Fig.11 The present invention shows a flow chart of a method for updating a user's paralanguage library, which can be executed by electronic devices such as smart phones, smart speakers, smart car devices, and smart robots. Fig.11 As shown, the method for updating the user's secondary language library may include:

[0121] 1101: If a wake-up operation of the user on the intelligent voice application is detected, a recording reminder message is displayed on the display interface of the electronic device.

[0122] It can be understood that the wake-up operation can be an operation in which the user sends a registered wake-up audio within a preset range of the electronic device to wake up the intelligent voice application, and the wake-up operation can wake up the voice interaction system of the intelligent voice application.

[0123] In some optional instances, recording reminder information may be displayed on the display interface of the electronic device, and recording reminder information, a confirmation recording control, and a rejection recording control may also be displayed on the display interface of the electronic device.

[0124] For example, if it is detected that the user is within a preset range of the electronic device 1200 (such as a range represented by a circle with the electronic device 1200 as the center and a radius of 1 meter), a signal is sent through Figure 8 and Fig. 9 The wake-up audio registered by the method shown (such as Fig. 9 The corresponding audio of the wake-up text 943 in the text field can be displayed as follows: Fig.12 a, and a recording reminder message 1211 “Hi, talking to Xiaoyi more can help Xiaoyi learn to be more like you. Your recorded audio is completely saved locally and nothing will be retained in the cloud” can be displayed on the first setting interface 1210 of the electronic device 1200. At the same time, a confirmation recording control 1212 and a rejection recording control 1213 can be displayed on the first setting interface 1210 of the electronic device 900.

[0125] In an embodiment of the present application, by displaying recording reminder information on the display interface of the electronic device so that the user can confirm or refuse recording, it is possible to avoid erroneous collection of the user's audio due to the user accidentally waking up the intelligent voice application.

[0126] 1102: In response to the user confirming the recording operation for the recording reminder information, displaying a plurality of candidate update frequency selection information on a display interface of the electronic device.

[0127] It can be understood that the recording confirmation operation is to touch the recording confirmation control on the display interface of the electronic device. Fig.12 As shown, if it is detected that the user clicks the confirmation recording control 1212 on the first setting interface 1210 of the electronic device 1200 within the preset time period, in response to the user's confirmation recording operation for the recording reminder information, it can jump to the Fig.12 b, and multiple candidate update frequency selection information can be displayed on the second setting interface 1220 of the electronic device 1200.

[0128] It can be understood that the candidate update frequency selection information may include at least two of a real-time update control, a weekly update control, a monthly update control, and a quarterly update control. Fig.12 As shown, when it is detected that the user touches the confirmation recording control 1212 on the first setting interface 1210 of the electronic device 1200, it can jump to the Fig.12 b, and candidate update frequency selection information such as a real-time update control 1221, a weekly update control 1222, a monthly update control 1223, and a quarterly update control 1224 can be displayed on the second setting interface 1220 of the electronic device 1200.

[0129] In the embodiment of the present application, by displaying a plurality of candidate update frequency selection information on the display interface of the electronic device for the user to select, the timeliness of the user's paralanguage subsequently obtained from the recorded audio can be ensured, and the naturalness and anthropomorphism of the subsequent synthesized audio can be improved.

[0130] 1103: In response to the user's operation of determining an update frequency for a plurality of candidate update frequency selection information, a voice interaction is performed with the user based on the voice interaction system.

[0131] It can be understood that the operation of determining the update frequency can be to trigger any one of the multiple candidate update frequency controls. When the electronic device responds to the user's operation of determining the update frequency for multiple candidate update frequency selection information, it can obtain the target update frequency corresponding to the operation of determining the update frequency, that is, the update frequency corresponding to any one of the triggered candidate update frequency controls.

[0132] In some optional embodiments, such as Fig.12 As shown, when it is detected that the user clicks on any one of the multiple candidate update frequency controls on the second setting interface 1220 of the electronic device 1200, it can jump to the Fig.12 c, and can display an interactive text 1231 corresponding to the interactive voice on the display interface 1230 of the electronic device 1200, and emit an interactive audio corresponding to the interactive text 1231 based on the voice interaction system to achieve voice interaction with the user, that is, to achieve voice dialogue with the user.

[0133] 1104: During voice interaction with the user, audio to be processed is obtained.

[0134] In some optional examples, during the process of voice interaction with the user, multiple interactive audios emitted by the user may be obtained as audios to be processed. Fig.12 As shown, in one round of voice interaction, after the electronic device 120 issues the interactive audio corresponding to the interactive text 1231-1 based on the voice interaction system, the interactive audio issued by the user can be obtained, and the interactive audio issued by the user can be displayed on the display interface 1230 of the electronic device 1200 in the form of interactive text 1231-2, and the interactive audio issued by the user can be stored in the storage device of the electronic device 1200. In this way, through multiple rounds of voice interaction, the interactive audio issued by the voice interaction system and the interactive text corresponding to the interactive audio issued by the user (such as Fig.12 1231-1 to 1231-N), and multiple interactive audios emitted by the user can be obtained as audios to be processed.

[0135] In an embodiment of the present application, by obtaining the user's recorded audio during the voice interaction process, compared to the audio recorded when the user reads aloud the specified text in a quiet environment, not only can the uniformity of the rhythm of the audio obtained by the user reading the specified text be avoided, the diversity of the audio can be improved, and the restrictions on the user reading the specified text can be avoided, the user's freedom of expression can be improved, and the naturalness and anthropomorphism of the subsequent synthesized audio can be improved.

[0136] 1105: Acquire the user's sub-language from the audio to be processed, and update the user's sub-language library based on the target update frequency and the user's sub-language corresponding to the update frequency operation.

[0137] It is understandable that the user's paralanguage, such as hissing, hesitation, sighing, coughing, laughing, inhaling, dragging, inversion, smacking, stuttering, correction of slips of the tongue, emphasis, etc., which have no substantial meaning and are only a phenomenon of voice modification of the user's expression, can be obtained from the audio to be processed, and the user's paralanguage library can be updated based on the target update frequency corresponding to the update frequency operation. The specific implementation method of updating the user's paralanguage library based on the target update frequency is described in Fig.13 Described in detail in.

[0138] In some optional instances, if the user triggers the real-time update frequency control from multiple candidate update frequency controls, the time weight can be set according to the time sequence of each audio in the audio to be processed, and N audios are retained from the audio to be processed, where N is a positive number greater than 0, so as to update the user's paralanguage library based on the N audios. For example, when N=10, the user's paralanguage library can be updated based on the 11th to 20th audios out of 20 audios. The time weight corresponding to the audio that comes earlier in time can be less than the time weight corresponding to the audio that comes later in time. For example, the time weight corresponding to the 11th audio out of 20 audios is less than the time weight corresponding to the 20th audio.

[0139] In some other optional examples, if the user triggers a weekly update control, a monthly update control, or a quarterly update control from multiple candidate update frequency controls, the user's sub-language can be counted regularly (such as weekly update, monthly update, or quarterly update) based on the target update frequency corresponding to the update frequency operation, and the user's sub-language library can be updated regularly.

[0140] The following is an introduction to the method of updating the user's paralanguage library based on the target update frequency. Fig.13 A flow chart of a method for updating a user's paralanguage library based on a target update frequency is shown. The method can be executed by electronic devices such as smart phones, smart speakers, smart car devices, and smart robots.

[0141] like Fig.13As shown, the user voice 1301 (i.e., the audio to be processed) can be input into the voice encoder 1302, and the voice encoder 1302 can perform encoding processing on the user voice 1301 based on the paralanguage library 1303 to obtain the paralanguage feature 1304. Among them, the paralanguage feature 1304 can represent converting the user voice 1301 into a machine-recognizable language, such as binary code. Among them, the processing processes of the voice encoder and the voice decoder for the audio to be processed are described in detail in Fig.15 which will be elaborated below.

[0142] In some optional examples, the paralanguage library 1303 can include the mapping relationship between the audio signal and the machine-recognizable language. For example, the audio segment corresponding to the paralanguage "hissing sound" can be mapped to the binary code "000", and the audio segment corresponding to the paralanguage "pause" can be mapped to the binary code "001", etc. This is only an example listed in the embodiments of the present application and does not represent the only implementation method.

[0143] It can be understood that the voice encoder 1302 can be any voice encoder capable of converting the audio of any language into a machine-recognizable language (such as binary code). Among them, the voice encoder can be a pre-trained model for speech recognition or a language model, such as an N-gram model.

[0144] In some optional examples, when the user's paralanguage library is updated in real time, the paralanguage feature 1304 can be updated in real time based on the time weight set according to the time sequence of the user voice 1301 to obtain the N-round dialogue result 1305.

[0145] In some other optional examples, when the user's paralanguage library is updated regularly, the paralanguage feature 1304 can be updated regularly based on a preset time interval to obtain the paralanguage segment 1306. For example, the frequency of each paralanguage can be counted at a specified frequency (i.e., the frequency corresponding to the preset time interval) to update the N-gram model. Among them, the frequency of each paralanguage can refer to the frequency of the paralanguage appearing in each scenario, such as the frequency of the paralanguage "pause" appearing after the modal particle "ne".

[0146] It can be understood that the N-round dialogue result 1305 can be used as the user habit library 1307, that is, the user habit copy described above, which is also the updated user's paralanguage library described above. The paralanguage segment 1306 can also be used as the user habit library 1307, that is, the user habit library described above, which is also the user's paralanguage library described above. Among them, the paralanguage segment 1306 can be displayed on the display interface of the electronic device in the form of data visualization (such as a bar chart).

[0147] In some specific implementations, the user can adjust the frequencies of some or all of the paralanguages ​​in the paralanguage library, and can save the adjusted paralanguage library. The user who adjusts the frequencies of some or all of the paralanguages ​​in the user's paralanguage library can be the user himself, or other users in the paralanguage library sharing group except the user himself. The specific method of adjusting the frequencies of some or all of the paralanguages ​​in the paralanguage library will be described in Fig.16 Described in detail in.

[0148] In some optional examples, when a user in a paralanguage library sharing group adjusts part or all of the paralanguages ​​in the user's own paralanguage library, the adjusted paralanguage library can be saved as the user's own paralanguage library, and the original paralanguage library can be deleted, or the adjusted paralanguage library can be saved as a new paralanguage library, i.e., another paralanguage library of the user. When a user in a paralanguage library sharing group adjusts part or all of the paralanguages ​​in the paralanguage library of another user, the adjusted paralanguage library can be saved as a new paralanguage library, i.e., another paralanguage library of another user.

[0149] For example, the paralanguage library sharing group may include family members, and family member A may adjust the frequencies of some or all paralanguages ​​in his own paralanguage library, and save the adjusted paralanguage library as the paralanguage library of A. Family member B may adjust the frequencies of some or all paralanguages ​​in the paralanguage library of family member A, and save the adjusted paralanguage library as another paralanguage library of B.

[0150] Fig.14 A schematic diagram of a method for updating a user's paralinguistic library is shown. Fig.14 As shown in the figure, when a user interacts with an electronic device through voice based on a voice interaction system, a large amount of user speech recognition (Automatic Speech Recognition, ASR) data (i.e., the user's interactive audio) will pass through the voice interaction system. If the user agrees and registers the wake-up audio, i.e., registers the user's voiceprint, the speech encoder can be used to extract paralinguistic features from the user speech recognition data (i.e., the user's interactive audio), and the discrete acoustic representations near the paralinguistic features are retained as user habits in the user habit library (i.e., the user's paralinguistic library) based on the user habit retention scheme.

[0151] Furthermore, discrete acoustic representations near paralinguistic features can be used to make subtle adjustments, so that the timbre can be better preserved.

[0152] The following introduces the specific training methods of the speech encoder and speech decoder for processing audio. Fig.15A block diagram of a method for processing audio to be processed is shown, and the method can be executed in a cloud server or in an electronic device.

[0153] like Fig.15 As shown, the speech encoder and speech decoder can be pre-trained based on a large-scale speech library in the cloud, where the speech encoder can be a speech recognition pre-training model and the speech decoder can be a HiFi-GAN model. And the text unit mapping (text-to-unit, T2U) can be updated through the paralanguage library and text library in the cloud-side data. Among them, the text unit mapping can represent the mapping relationship between the audio signal and the machine-recognizable language.

[0154] The trained speech encoder can encode the paralanguage and text in the input audio to be processed based on the text unit mapping, for example, encoding the paralanguage and text in the audio to be processed into discrete acoustic representations. The trained speech decoder can decode the output encoded features based on the text unit mapping, for example, decoding the discrete acoustic representations into paralanguage and text.

[0155] The following is an introduction to the specific implementation method of the user interacting with the electronic device. Fig.16 A block diagram of interaction between a user and an electronic device is shown.

[0156] like Fig.16 The user 1601 shown can interact with the electronic device based on the visual interface 1602 (i.e., display interface) of the electronic device. For example, the user can touch any timbre control on the visual interface of the electronic device to select the timbre of the target user with whom the user interacts. The user can also touch the edit timbre control on the visual interface of the electronic device to update the paralanguage library of the target user corresponding to the selected timbre control.

[0157] The paralanguage analysis module 1603 may read the paralanguage library of the target user based on the timbre control touched by the user, so as to interact with the user using the voice characteristics and paralanguage characteristics of the target user during the interaction with the user.

[0158] The paralanguage analysis module 1603 may also obtain multiple paralanguages ​​of the user after adjustment based on the user's adjustment of part or all of the paralanguages ​​in the paralanguage library of the target user, and update the user paralanguage library 1604 based on the multiple paralanguages ​​of the user after adjustment.

[0159] The following is an introduction to the specific implementation of adjusting part or all of the paralanguages ​​in the paralanguage library of the target user described above. Fig.17 A schematic diagram of an interface for adjusting the frequency of a paralanguage in a paralanguage library is shown.

[0160] like Fig.17 As shown, users in the paralanguage library sharing group can select the timbre of a target user to interact with based on the visual interface (ie, display interface) of the electronic device. Fig.17 a shows a first display interface 1710 of an electronic device 1700, the first display interface 1710 may include a tone selection box 1711 and a tone editing control 1712, the tone selection box 1711 includes a plurality of tone controls, such as a tone control 1711-1 corresponding to the user, a tone control 1711-2 corresponding to family member A, a tone control 1711-3 corresponding to family member B, or a tone control 1711-4 corresponding to family member C.

[0161] When it is detected that the user clicks on any one of the timbre controls corresponding to the user in the timbre selection box 1711 on the first display interface 1710 of the electronic device 1700, such as detecting that the user clicks on any one of the timbre controls 1711-1 corresponding to the user himself, the timbre control 1711-2 corresponding to family member A, the timbre control 1711-3 corresponding to family member B, or the timbre control 1711-4 corresponding to family member C, and detecting that a subsequent user clicks on the edit timbre control 1712 on the first display interface 1710 of the electronic device 1700, the program can jump to the following screen: Fig.17 b, and the paralanguage library of the target user corresponding to the timbre control can be displayed on the second display interface 1720 of the electronic device 1700, such as the paralanguage segments of the target user displayed in the form of a bar graph, that is, the frequency of each paralanguage.

[0162] In some optional examples, when it is detected that the user clicks the timbre control 1711-1 corresponding to the user in the timbre selection box 1711, the page can be jumped to the following example: Fig.17 b, and the user's own paralanguage segments, that is, the frequencies of each paralanguage, can be displayed in the form of a bar graph on the second display interface 1720 of the electronic device 1700. Then, the user can press and hold the rectangular boxes corresponding to some or all of the paralanguages ​​and slide them up and down to increase or decrease the frequencies of the part or all of the paralanguages.

[0163] After the user has adjusted the frequency of part or all of the paralanguages, for example, if the user is not detected touching the image display interface of the electronic device within a preset time period, the interface may be jumped to the following Fig.17c, can display saving controls on the third display interface 1730 of the electronic device 1700, such as a save as original timbre control 1731, a save as new timbre control 1732, etc., for the user to choose to save the adjusted para-language library as the user's own para-language library, that is, save as the original timbre, or save the adjusted para-language library as the user's own new para-language library, that is, another para-language library of the user, that is, save as a new timbre.

[0164] The following describes the above-mentioned method of obtaining the user's secondary language and updating the user's secondary language library using a specific scenario. Fig.18 A block diagram of a method for updating a user's paralanguage library is shown, which can be executed by an electronic device, such as a smart phone, a smart speaker, a smart car device, a smart robot, etc. The electronic device may include a wake-up module, a voiceprint module, a recording module, a feature extraction module, and a paralanguage analysis module.

[0165] like Fig.18 As shown, the method for updating the user's secondary language library may include:

[0166] 1801: The wake-up module determines whether the intelligent voice application is awakened. If the judgment result is yes, go to step 1802, otherwise end.

[0167] It can be understood that the wake-up module can receive the user's wake-up audio and then determine whether the intelligent voice application is woken up.

[0168] It can be understood that the wake-up module can receive the wake-up audio of any one of the multiple users, wherein the multiple users may include users who have registered the wake-up audio and users who have not registered the wake-up audio. When the wake-up module receives the wake-up audio of the user who has registered the wake-up audio, the intelligent voice application can be woken up immediately. When the wake-up module receives the wake-up audio of the user who has not registered the wake-up audio, the intelligent voice application may not be woken up.

[0169] It can be understood that if the user's wake-up audio is the wake-up audio pre-registered by the user in the electronic device, such as the wake-up audio issued by the user is "Xiaoyi Xiaoyi", the intelligent voice application will be awakened, so that the wake-up module can detect that the intelligent voice application is awakened. If the user's wake-up audio is not the wake-up audio pre-registered by the user in the electronic device, such as the wake-up audio issued by the user is "Xiaoyou Xiaoyou", the intelligent voice application will not be awakened, so that the wake-up module can detect that the intelligent voice application is not awakened.

[0170] 1802: The voiceprint module determines whether the wake-up audio is a registered voiceprint. If the judgment result is yes, go to step 1803, otherwise end.

[0171] It can be understood that if user A has registered the wake-up audio in the electronic device in advance, then the wake-up audio of user A is the registered voiceprint, and the voiceprint module can detect that the wake-up audio is the registered voiceprint. If user B has not registered the wake-up audio in the electronic device in advance, then the wake-up audio of user B is an unregistered voiceprint, and the voiceprint module can detect that the wake-up audio is an unregistered voiceprint. At this time, even if the intelligent voice application is mistakenly woken up, the intelligent voice application will be closed.

[0172] In the embodiment of the present application, by verifying that the wake-up audio is a registered voiceprint through the voiceprint module, it can be determined that the intelligent voice application is awakened by the user who has registered the wake-up audio, which can reduce the false awakening of the intelligent voice application and further reduce the occurrence of subsequent false updates to the user's paralanguage library.

[0173] 1803: The recording module records the user's interactive audio to obtain the audio to be processed.

[0174] As mentioned above, during the process of the electronic device performing voice interaction with the user based on the voice interaction system, the recording module can obtain multiple interactive audios emitted by the user as audios to be processed.

[0175] 1804: The feature extraction module extracts the user's paralinguistic features from the audio to be processed.

[0176] 1805: The paralanguage analysis module performs paralanguage analysis based on the user's paralanguage features, obtains multiple paralanguages ​​of the user, and updates the user paralanguage library based on the multiple paralanguages ​​of the user.

[0177] In some optional examples, the user's paralanguage may include the N rounds of dialogue results, paralanguage segments, etc. described above.

[0178] The following is an introduction to the hardware structure of electronic equipment. Fig.19As shown, the electronic device 1900 may include a processor 1910, an external memory interface 1920, an internal memory 1921, a universal serial bus (USB) interface 1930, a charging management module 1940, a power management module 1941, a battery 1942, an antenna 1, an antenna 2, a mobile communication module 1950, a wireless communication module 1960, an audio module 1970, a speaker 1970A, a receiver 1970B, a microphone 1970C, an earphone interface 1970D, a sensor module 1980, a button 1990, a motor 1991, an indicator 1992, a camera 1993, a display screen 1994, and a subscriber identification module (SIM) card interface 1995, etc. The sensor module 1980 may include a pressure sensor 1980A, a gyroscope sensor 1980B, an air pressure sensor 1980C, a magnetic sensor 1980D, an acceleration sensor 1980E, a distance sensor 1980F, a proximity light sensor 1980G, a fingerprint sensor 1980H, a temperature sensor 1980J, a touch sensor 1980K, an ambient light sensor 1980L, a bone conduction sensor 1980M, and the like.

[0179] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device 1900 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0180] The electronic device 1900 implements the display function through a GPU, a display screen 1994, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1994 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1910 may include one or more GPUs that execute program instructions to generate or change display information.

[0181] The display screen 1994 is used to display images, videos, etc. The display screen 1994 includes a display panel.

[0182] The electronic device 1900 can implement audio functions such as music playing and recording through the audio module 1970, the speaker 1970A, the receiver 1970B, the microphone 1970C, the headphone jack 1970D, and the application processor.

[0183] The audio module 1970 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 1970 can also be used to encode and decode audio signals. In some embodiments, the audio module 1970 can be arranged in the processor 1910, or some functional modules of the audio module 1970 can be arranged in the processor 1910.

[0184] The speaker 1970A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 1900 can listen to music or listen to a hands-free call through the speaker 1970A.

[0185] The receiver 1970B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 1900 receives a call or voice message, the voice can be received by placing the receiver 1970B close to the human ear.

[0186] Microphone 1970C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 1970C to input the sound signal into the microphone 1970C. The electronic device 1900 can be provided with at least one microphone 1970C. In other embodiments, the electronic device 1900 can be provided with two microphones 1970C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 1900 can also be provided with three, four or more microphones 1970C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0187] The pressure sensor 1980A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 1980A can be set on the display screen 1994. There are many types of pressure sensors 1980A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 1980A, the capacitance between the electrodes changes. The electronic device 1900 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 1994, the electronic device 1900 detects the touch operation intensity according to the pressure sensor 1980A. The electronic device 1900 can also calculate the touch position according to the detection signal of the pressure sensor 1980A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to a first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0188] The touch sensor 1980K is also called a "touch control device". The touch sensor 1980K can be set on the display screen 1994, and the touch sensor 1980K and the display screen 1994 form a touch screen, also called a "touch control screen". The touch sensor 1980K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 1994. In other embodiments, the touch sensor 1980K can also be set on the surface of the electronic device 1900, which is different from the position of the display screen 1994.

[0189] The key 1990 includes a power key, a volume key, etc. The key 1990 may be a mechanical key or a touch key. The electronic device 1900 may receive key input and generate key signal input related to user settings and function control of the electronic device 1900.

[0190] The above describes the hardware structure that the electronic device may have. It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0191] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0192] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0193] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.

[0194] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0195] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0196] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0197] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0198] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A speech synthesis method, used in electronic equipment, characterized in that: include: Obtaining a first text to be synthesized; The first text is synthesized into first synthesized audio data through the electronic device acoustic model and based on a first paralanguage library of a first user, wherein the first synthesized audio data includes: first-category audio data corresponding to text information in the first text and second-category audio data corresponding to the paralanguage of the first user.

2. The method according to claim 1, characterized in that Update the first secondary language library of the first user by: Acquire target update frequency selection information selected by the first user; Performing voice interaction with the first user based on the voice interaction system, obtaining the first user's audio to be processed during the voice interaction process; Acquire multiple sub-languages ​​of the first user from the audio to be processed; Based on the target update frequency selection information and the plurality of sub-languages ​​of the first user, a first sub-language library of the first user is updated.

3. The method according to claim 2, characterized in that The target update frequency selection information includes any one of real-time update, weekly update, monthly update and quarterly update.

4. The method according to claim 1, characterized in that: The first secondary language library of the first user is updated by: Obtaining a bar graph corresponding to the first paralanguage library of the first user, wherein each rectangular box in the bar graph represents a paralanguage, and the height of the rectangular box represents the frequency of the paralanguage; Acquire adjustment information for part or all of the rectangular boxes in the bar graph, and update the first secondary language library of the first user based on the adjustment information.

5. The method according to claim 4, characterized in that The acquiring adjustment information of part or all of the rectangular boxes in the bar graph, and updating the first secondary language library of the first user based on the adjustment information, includes: Acquire adjustment information of part or all of the rectangular boxes in the bar chart by the first user; The first secondary language library of the first user is updated based on the bar graph after adjusting part or all of the rectangular boxes.

6. The method according to claim 1, characterized in that The electronic device further stores a second paralanguage library for the acoustic model, wherein the second paralanguage library is applicable to a second user, and the second user is a user associated with the first user.

7. The method according to claim 6, characterized in that The association relationship between the first user and the second user includes a family member relationship.

8. A method for processing an acoustic model, characterized in that: include: Acquire reference synthesized audio data, sample text corresponding to the reference synthesized audio data, and a sample paralanguage library; Using the sample text and the sample paralanguage library as inputs of a neural network model, and obtaining predicted synthesized audio data output by the neural network model; Based on the reference synthesized audio data and the predicted synthesized audio data, the parameters of the neural network model are adjusted to obtain an acoustic model.

9. An electronic device, characterized in that: include: A memory, used to store instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, used to execute the speech synthesis method described in any one of claims 1-7.

10. A readable storage medium, characterized in that: The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the speech synthesis method according to any one of claims 1 to 7.

Images