Voice preview method, device and electronic device

Abstract

The present disclosure relates to a method, device and electronic device for voice preview, wherein the method comprises: receiving text input; caching voice data synthesized from the text input by a voice synthesis service in real time; and decoding and playing the cached voice data when the synthesized voice data is cached to a playable length.

Description

Technical Field

[0001] The present disclosure relates to the field of signal processing, and in particular to a method, device and electronic device for voice preview. Background Art

[0002] In the related art, the scenario in which a terminal device (such as a mobile phone) uses a text-to-speech (TTS) service is: input text, generate a voice file by calling a network or offline software development kit (SDK), and then return it to the terminal device through the network or file. After that, the terminal device plays it by calling the voice file. In the video editing scenario, the user uses the terminal device to edit the shot video and edit the text, and then uses the edited text to generate audio files with different tones, and then synthesizes them into the video to complete the dubbing process. However, in similar video editing scenarios, using files for transmission will cause a relatively large delay. For example, if the user needs to switch tones and preview in real time, the traditional TTSSDK needs to have a process of generating files and transmitting them back through the network. If the input text is long and the entire file is large, it will cause the waiting time to be too long, and the real-time preview effect cannot be achieved, which will seriously affect the user experience. Summary of the invention

[0003] The present disclosure provides a method, device and electronic device for voice preview, which at least solves the problem of poor real-time preview effect in the related art, and may not solve any of the above problems. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for voice preview is provided, characterized in that it includes: receiving text input; caching voice data synthesized from the text input through a TTS service in real time; and when the synthesized voice data is cached to a playable length, decoding and playing the cached voice data.

[0005] Optionally, the method further includes: sending the text input to a server; and receiving, from the server in real time, voice data synthesized from the text input via a TTS service.

[0006] Optionally, when a tone switching operation is received, the server is notified to stop using the TTS service to perform speech synthesis on the text input according to the tone before the switching, and the server is notified to use the TTS service to re-synthesize the text input according to the switched tone, and the speech data synthesized from the text input by the TTS service according to the switched tone received in real time from the server is re-cached locally.

[0007] Optionally, the method further includes: locally performing speech synthesis on the text input through a TTS service to obtain speech data.

[0008] Optionally, when the receiving unit receives a tone switching operation, it stops using the TTS service to perform speech synthesis on the text input according to the tone before switching, uses the TTS service to perform speech synthesis on the text input again according to the switched tone, and re-caches the speech data synthesized from the text input through the TTS service according to the switched tone.

[0009] According to a second aspect of an embodiment of the present disclosure, a device for voice preview is provided, the device comprising: a receiving unit configured to receive text input; a caching unit configured to cache in real time voice data synthesized from the text input through a TTS service; a decoding unit configured to decode the cached voice data when the synthesized voice data is cached to a playable length; and a playing unit configured to play the decoded audio data.

[0010] Optionally, the device further comprises: a sending unit configured to send the text input to a server, wherein the receiving unit is further configured to receive, from the server in real time, voice data synthesized from the text input via a TTS service.

[0011] Optionally, when the receiving unit receives a tone switching operation, the sending unit is configured to notify the server to stop using the TTS service to perform speech synthesis on the text input according to the tone before the switching, and to notify the server to use the TTS service to perform speech synthesis on the text input again according to the switched tone, and the cache unit is configured to re-cache the speech data synthesized from the text input through the TTS service according to the switched tone, which is received in real time by the receiving unit from the server.

[0012] Optionally, the device further includes: a speech synthesis unit configured to perform speech synthesis on the text input through a TTS service to obtain speech data.

[0013] Optionally, when the receiving unit receives a tone switching operation, the speech synthesis unit is configured to stop using the TTS service to perform speech synthesis on the text input according to the tone before the switching, and to re-speech synthesize the text input according to the switched tone, and the cache unit is configured to re-cache the speech data synthesized from the text input through the TTS service according to the switched tone.

[0014] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the voice preview method as described above.

[0015] According to a fourth aspect of an embodiment of the present disclosure, a speech processing system is provided, characterized in that it includes: a terminal device, configured to receive text input, send the text input to a server, cache in real time the speech data synthesized from the text input via a TTS service received from the server, and decode and play the cached speech data when the received speech data is cached to a playable length; and a server, configured to perform speech synthesis on the text input received from the terminal device via a TTS service to obtain speech data, and transmit the obtained speech data to the terminal device in real time.

[0016] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, characterized in that when the instructions are executed by at least one processor, the at least one processor is prompted to execute the voice preview method as described above.

[0017] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, characterized in that instructions in the computer program product are executed by at least one processor in an electronic device to perform the voice preview method as described above.

[0018] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: the delay is greatly reduced through real-time transmission, and a very small amount of voice data is cached before starting the real-time preview, which is almost no waiting. At the same time, when switching the timbre, the local terminal device itself or the notification server will no longer perform TTS on the remaining text that has not yet been TTSed, thereby reducing the cost of TTS services, thereby increasing the speed of TTS preview for users in video editing and optimizing the user experience.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0021] Figure 1 is an exemplary system architecture diagram in which exemplary embodiments of the present disclosure may be applied;

[0022] Figure 2 is a flowchart of a method for voice preview according to an exemplary embodiment of the present disclosure;

[0023] Figure 3 is a detailed flow chart of a method for voice preview when a TTS service is executed on a server side according to an exemplary embodiment of the present disclosure;

[0024] Figure 4 A detailed flowchart of a method for voice preview when a TTS service is executed locally on a terminal device according to an exemplary embodiment of the present disclosure;

[0025] Figure 5 is a block diagram of a device for voice preview according to an exemplary embodiment of the present disclosure;

[0026] Figure 6 is a detailed block diagram of a device for voice preview according to an exemplary embodiment of the present disclosure;

[0027] Figure 7 is a detailed block diagram of a device for voice preview according to another exemplary embodiment of the present disclosure;

[0028] Figure 8 is a block diagram of a system for voice preview according to an exemplary embodiment of the present disclosure;

[0029] Fig. 9 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0031] It should be understood that the data used in this way can be interchanged where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0033] As mentioned in the background technology of the present disclosure, in the related art, when using text input to generate an audio file and preview the audio file in the scenario of video editing, the text file input by the user needs to go through the processes of text input and / or uploading, speech synthesis, speech encoding, and downloading. In other words, the traditional file usage method is executed serially, and the input text file needs to wait for the above steps to be completed before the downloaded audio file can be previewed. In response to this, the present disclosure proposes that after receiving the text input, the audio data synthesized by the TTS service can be cached in real time, and the real-time preview of the audio file can be started when very little data is cached, with almost no waiting. At the same time, when the tone is switched, the text that has not yet been speech synthesized can no longer be speech synthesized, thereby reducing the cost of the TTS service.

[0034] Figure 1 An exemplary system architecture 100 is shown in which exemplary embodiments of the present disclosure may be applied.

[0035] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. Users can use terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (such as TTS service requests, audio and video data upload requests, audio and video data acquisition requests), etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as video recording applications, audio playback applications, video and audio editing applications, instant messaging tools, mailbox clients, social platform software, etc. The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display screens and capable of playing, recording, and editing audio and video, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, and can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0036] The terminal devices 101, 102, and 103 may be equipped with an image acquisition device (e.g., a camera) to acquire video data. In addition, the terminal devices 101, 102, and 103 may also be equipped with a component for converting an electrical signal into sound (e.g., a speaker) to play sound, and may also be equipped with a device for converting an analog audio signal into a digital audio signal (e.g., a microphone) to acquire sound.

[0037] The terminal devices 101, 102, and 103 can use the image acquisition device installed thereon to acquire video data, and use the audio acquisition device installed thereon to acquire audio data. In addition, the terminal devices 101, 102, and 103 can perform TTS service on the received text input to synthesize audio data from the text input, and can use the audio processing component installed thereon that supports audio playback to play the audio data.

[0038] The server 105 may be a server that provides various services, such as a background server that provides support for audio and video recording applications, audio and video editing applications, etc. installed on the terminal devices 101, 102, and 103. The background server may parse, provide TTS services, store, and process the uploaded text input data, and may also receive TTS service requests sent by the terminal devices 101, 102, and 103, and feed back the audio data of speech synthesis to the terminal devices 101, 102, and 103.

[0039] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0040] It should be noted that the voice preview method provided in the embodiment of the present application is generally executed by the terminal devices 101 , 102 , 103 , and accordingly, the voice preview apparatus is generally disposed in the terminal devices 101 , 102 , 103 .

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers, and the present disclosure has no limitation on this.

[0042] Figure 2 The present invention is a flowchart of a method for voice preview according to an exemplary embodiment of the present invention.

[0043] In step S201, text input is received. Here, the text input can be text input or edited by the user on the terminal device in any way. Specifically, when the user starts the editing preview function in the audio and video editing software on the terminal device, the user can directly input text in the audio and video editing software, or can directly load a text file received from other devices or downloaded from the server into the audio and video editing software. The above is only an example, and the present disclosure does not specifically limit the method of receiving text input. Any method that can perform text input is included in the scope of the present disclosure.

[0044] In step S202, the voice data synthesized from the text input by the TTS service is cached in real time. Then, in step S203, when the synthesized voice data is cached to a playable length, the cached voice data is decoded and played. Specifically, the TTS service can be a local TTS service called by the terminal device through the audio and video editing software, or it can be a server-side TTS service called by the audio and video editing software. In other words, the TTS service can be performed locally on the terminal device to synthesize the text input into voice data, or the terminal device can request the server side to perform the TTS service to synthesize the uploaded text input into voice data. The following will refer to the attached Figure 3 and attached Figure 4 The detailed processes of the method for voice processing when the TTS service is executed on the server side and when the TTS service is executed locally on the terminal device are described respectively.

[0045] Figure 3 is a detailed flowchart of a method for voice preview when a TTS service is executed on a server side according to an exemplary embodiment of the present disclosure.

[0046] In step S301, a text input is received. Since this step is the same as the operation in step S201, it will not be described again here.

[0047] In step S302, the text input is sent to the server. Specifically, after the user starts the editing preview function in the audio and video editing software on the terminal device, the terminal device can call the background TTS service, which is located on the server side, that is, the terminal device can call the TTS service on the server side and upload the text input to the server, and then the server synthesizes the text input received from the terminal device into voice data through the TTS service.

[0048] In step S303, voice data synthesized from the text input by the TTS service is received from the server in real time. In other words, the terminal device can receive the voice data synthesized from the text input by the TTS service from the server in real time by streaming. Specifically, when the server receives the text input uploaded from the terminal device, the server performs speech synthesis on the text input by the TTS service according to the request of the terminal device, thereby generating audio encoding data in a specific format, such as generating audio encoding data in the Advanced Audio Coding (AAC) format. Among them, the request of the terminal device may include various voice settings related to the TTS service, such as the timbre, pitch, speech speed, tone, background music, etc. of the synthesized voice that the user expects. In addition, the present disclosure does not impose special restrictions on the format of the generated audio encoding data, as long as the generated audio encoding data can be subsequently streamed and each audio frame can be independently decoded and the audio format with a shorter duration of the audio frame is included in the scope of the present disclosure. Thereafter, the generated audio encoding data in a specific format is packaged by the server and transmitted to the terminal device in real time in a streaming manner. For example, the generated audio encoding data can be packaged by the server in the AAC streaming transmission format (ADTS) format and transmitted back to the terminal device in real time in a streaming manner.

[0049] In step S304, the voice data synthesized from the text input by the TTS service is cached in real time. Specifically, when the terminal device receives the voice data synthesized by the TTS service from the server by streaming, the terminal device caches the received voice data in a buffer in real time. For example, the terminal device can cache the data packets in the ADTS format received from the server in a buffer in real time.

[0050] In step S305, when the synthesized voice data is cached to a playable length, the cached voice data is decoded and played. Specifically, when the voice data cached in the buffer of the terminal device reaches the playable length (i.e., a certain size), for example, when the terminal device caches voice data of a playable length of 1 second in the buffer, the terminal device can perform a decoding operation on the currently cached voice data and play the decoded PCM data. In addition, during the above decoding process, the terminal device always receives the just synthesized voice data from the server in a streaming mode and caches the received voice data in the buffer to ensure the continuity of decoding and playing. The above-described process of receiving the synthesized voice data from the server in real time in a streaming mode, caching the received voice data in real time, and starting to decode and play after caching a certain amount of data can realize real-time preview, greatly reducing preview delay.

[0051] In addition, in actual applications, users may be dissatisfied with the voice synthesized according to the current voice settings related to the TTS service, and then change the voice settings related to the TTS service during voice preview. For example, users may change the timbre, pitch, speaking speed, tone or background music of the voice. In related technologies, voice preview can only be performed after the TTS service completes the synthesis of voice data corresponding to the entire text input according to the voice settings related to the TTS service set by the user and receives all the synthesized voice data from the server. Therefore, if the user does not like the voice synthesized according to the current voice settings, it will inevitably cause a waste of resources and affect the user experience.

[0052] However, in the present disclosure, when the voice setting related to the TTS service is changed, the terminal device may notify the server to stop the speech synthesis operation of the text input according to the voice setting before the change by using the TTS service, and notify the server to re-speech synthesize the text input according to the changed voice setting by using the TTS service, and re-cache the voice data synthesized from the text input by the TTS service according to the changed voice setting received from the server in real time in a streaming manner locally on the terminal device. Among them, when the voice setting related to the TTS service is changed, the terminal device may delete the previously cached voice data synthesized according to the previous voice setting, and decode and preview the cached voice data when the re-cached voice data synthesized according to the changed voice setting reaches the playable length. In this way, when the voice setting related to the TTS service is changed, the server no longer performs speech synthesis on the remaining text input that has not yet completed speech synthesis, thereby reducing the cost of the TTS service.

[0053] In addition, when the user completes editing according to the finalized voice settings, the terminal device can convert the cached ADTS data corresponding to the current voice settings into audio data in other formats such as M4A, and generate audio metadata, such as duration, sampling rate, number of channels and other information, which will eventually be provided to the audio and video editing SDK for use.

[0054] The above describes the detailed process of the method for implementing speech preview of speech synthesis by the terminal device calling the TTS service on the server side. The following describes the detailed process of the method for implementing speech preview of speech synthesis by the terminal device calling the local TTS service.

[0055] Figure 4 The figure is a detailed flowchart of a method for voice preview when it is executed locally on a terminal device according to an exemplary embodiment of the present disclosure.

[0056] like Figure 4As shown in FIG. 4 , in step S401 , a text input is received. Since this step is the same as the operation of step S201 , it will not be described again here.

[0057] In step S402, the text input is synthesized into speech data locally through the TTS service. Specifically, after the user starts the editing preview function in the audio and video editing software on the terminal device, the terminal device can call the background TTS service, which is located locally on the terminal device, that is, the terminal device can call the local TTS service, for example, the terminal device calls the local TTS service through the API of the audio and video editing software, and transmits the text input to the local TTS service, and then synthesizes the text input into speech data through the TTS service according to the user's voice settings related to the TTS service. Among them, the terminal device can synthesize the text input into audio encoding data in a specific format through the local TTS service, for example, generating audio encoding data in AAC format. The voice settings related to the TTS service may include, for example, the timbre, background music, pitch, speech speed, tone, etc. of the voice that the user expects to be synthesized.

[0058] In step S403, the voice data synthesized from the text input by the TTS service is cached in real time. Specifically, the terminal device can cache the voice data synthesized by calling the local TTS service in real time. For example, the terminal device can cache the audio encoding data in AAC format synthesized by the local TTS service in a buffer.

[0059] In step S404, when the synthesized voice data is cached to a playable length, the cached voice data is decoded and played. Specifically, when the voice data cached in the buffer of the terminal device reaches the playable length (i.e., a certain size), for example, when the terminal device caches voice data of a playable length of 1 second in the buffer, the terminal device can perform a decoding operation on the currently cached voice data and play the decoded PCM data. In addition, when performing the above decoding, the terminal device always caches the voice data just synthesized by the local TTS service in real time to ensure the continuity of audio decoding and playback. The terminal device described above directly implements voice synthesis and caches the synthesized voice data by calling the local TTS service, and starts decoding and playing after a certain amount of data is cached, thereby achieving real-time preview and greatly reducing preview delay.

[0060] In addition, Figure 3Similar to the situation when the TTS service described is executed on the server side, in the present disclosure, when the voice setting related to the TTS service is changed, the terminal device can stop using the local TTS service to perform speech synthesis on the text input according to the voice setting before the change, and use the TTS service to re-speech synthesize the text input according to the changed voice setting, and re-cache the voice data synthesized from the text input by the TTS service according to the changed voice setting. Among them, when the voice setting related to the TTS service is changed, the terminal device can delete the previously cached voice data synthesized according to the previous voice setting, and decode and preview the cached voice data when the re-cached voice data synthesized according to the changed voice setting reaches the playable length. In this way, when the voice setting related to the TTS service is changed, the terminal device no longer performs speech synthesis on the remaining text input that has not yet completed speech synthesis, thereby reducing the cost of the TTS service.

[0061] In addition, when the user completes editing according to the finalized voice settings, the terminal device can convert the cached voice data corresponding to the current voice settings into audio data in other formats such as M4A, and generate audio metadata, such as duration, sampling rate, number of channels and other information, and finally provide it to the audio and video editing SDK for use.

[0062] Figure 5 It is a block diagram of a voice preview device 500 according to an exemplary embodiment of the present disclosure.

[0063] Reference Figure 5 The device 500 for voice preview may include a receiving unit 510, a buffer unit 520, a decoding unit 530 and a playing unit 540. The receiving unit 510 may receive text input. Figure 2 The contents related to text input have been described in detail, so they will not be repeated here.

[0064] The cache unit 520 can cache the voice data synthesized from the text input through the TTS service in real time. Then, the decoding unit 530 can decode the cached voice data when the synthesized voice data is cached to a playable length, and the playback unit 540 can play the decoded audio data. Specifically, the TTS service can be a local TTS service called by the device 500, or it can be a TTS service called on the server side. In other words, the TTS service can be performed locally on the device 500 to synthesize the text input into voice data, or the device 500 can request the server side to perform the TTS service to synthesize the uploaded text input into voice data. The following will refer to the attached Figure 6 and attached Figure 7Detailed processes of the method for voice processing when the TTS service is executed on the server side and when the TTS service is executed locally on the device 500 are described respectively.

[0065] Figure 6 It is a detailed block diagram of a voice preview device 600 according to an exemplary embodiment of the present disclosure.

[0066] Reference Figure 6 The voice preview device 600 may include a receiving unit 610, a cache unit 620, a decoding unit 630, a playing unit 640 and a sending unit 650.

[0067] The receiving unit 610 can receive text input, that is, the receiving unit 610 can perform the same as the above reference Figure 2 The corresponding operations of step S210 are described, so they will not be repeated here.

[0068] The sending unit 650 can send the text input to the server. Specifically, after the user starts the editing preview function in the audio and video editing software on the device 600, the device 600 can call the background TTS service, which is located on the server side. That is, the device 600 can call the TTS service on the server side and upload the text input to the server, and then the server synthesizes the text input received from the device 600 into voice data through the TTS service.

[0069] In addition, the receiving unit 610 can receive the voice data synthesized from the text input through the TTS service from the server in real time. In other words, the device 600 can receive the voice data synthesized from the text input through the TTS service from the server in real time through streaming. Specifically, when the server receives the text input uploaded from the device 600, the server performs speech synthesis on the text input through the TTS service according to the request of the device 600, thereby generating audio encoding data in a specific format, such as generating audio encoding data in AAC format. Thereafter, the generated audio encoding data in a specific format is packaged by the server and transmitted to the device 600 in real time through streaming.

[0070] The cache unit 620 can cache the voice data synthesized from the text input through the TTS service in real time. Specifically, when the device 600 receives the voice data synthesized through the TTS service from the server, the cache unit 620 caches the received voice data, for example, the cache unit 620 can cache the data packets in the ADTS format received from the server in real time.

[0071] The decoding unit 630 can decode the cached voice data when the synthesized voice data is cached to a playable length. Specifically, when the voice data cached by the cache unit 620 reaches a playable length (i.e., a certain size), the decoding unit 630 can decode the currently cached voice data, and the playback unit 640 plays the decoded PCM data. In addition, when performing the above decoding, the cache unit 620 always caches the voice data received from the server in real time in a streaming manner to ensure the continuity of the decoding operation of the decoding unit 630 and the playback operation of the playback unit 640.

[0072] In addition, when the voice setting related to the TTS service is changed, the sending unit 650 may notify the server to stop the speech synthesis operation of the text input according to the voice setting before the change by the TTS service, and notify the server to re-speech synthesize the text input according to the changed voice setting by the TTS service, and the cache unit 620 may re-cache the voice data synthesized from the text input by the TTS service according to the changed voice setting received in real time from the server by the receiving unit 610. When the voice setting related to the TTS service is changed, the cache unit 620 may delete the previously cached voice data synthesized according to the previous voice setting, and when the voice data synthesized according to the changed voice setting re-cached by the cache unit 620 reaches the playable length, the decoding unit 620 decodes the cached voice data, and the playback unit 640 previews and plays the decoded voice data.

[0073] In addition, when the user completes editing according to the final voice settings, the device 600 can convert the cached ADTS data corresponding to the current voice settings into audio data in other formats such as M4A, and generate audio metadata, such as duration, sampling rate, number of channels and other information, and finally provide it to the audio and video editing SDK for use.

[0074] In addition, due to Figure 3 The method of voice preview shown can be Figure 6 The voice preview device 600 shown is used to perform, therefore, Figure 6 Any details on the operations performed by the units in Figure 3 The corresponding description will not be repeated here.

[0075] Figure 7 is a detailed block diagram of a voice preview device 700 according to another exemplary embodiment of the present disclosure.

[0076] Reference Figure 7The voice preview device 700 may include a receiving unit 710, a cache unit 720, a decoding unit 730, a playing unit 740 and a voice synthesis unit 750.

[0077] The receiving unit 710 can receive text input, that is, the receiving unit 710 can perform the same as the above reference Figure 2 The corresponding operations of step S210 are described, so they will not be repeated here.

[0078] The speech synthesis unit 750 can perform speech synthesis on the text input locally through the TTS service to obtain speech data. Specifically, after the user starts the editing preview function in the audio and video editing software on the device 700, the speech synthesis unit 750 can call the background TTS service, which is located locally in the device 700. That is, the speech synthesis unit 750 can call the local TTS service. For example, the speech synthesis unit 750 can call the local TTS service through the API of the audio and video editing software, and transmit the text input to the local TTS service, and then synthesize the text input into speech data through the TTS service according to the user's voice settings related to the TTS service.

[0079] The cache unit 720 can cache the voice data synthesized from the text input by the TTS service in real time. Specifically, the cache unit 720 can cache the voice data synthesized by calling the local TTS service in real time.

[0080] The decoding unit 730 can decode the cached voice data when the synthesized voice data is cached by the cache unit 720 to a playable length, and the playback unit 740 can preview and play it. Specifically, when the voice data cached by the cache unit 720 reaches a playable length (i.e., a certain size), the decoding unit 730 can decode the currently cached voice data, and the playback unit 740 can play the decoded PCM data. In addition, when performing the above decoding, the cache unit 720 always caches the voice data just synthesized by the local TTS service in real time to ensure the continuity of the decoding operation of the decoding unit 730 and the playback operation of the playback unit 740.

[0081] In addition, Figure 6Similar to the situation when the TTS service described is executed on the server side, in the present disclosure, when the voice setting related to the TTS service is changed, the speech synthesis unit 750 may stop the speech synthesis operation of the text input according to the voice setting before the change using the TTS service, and re-speech synthesize the text input according to the changed voice setting, and the cache unit 720 may re-cache the voice data synthesized from the text input by the TTS service according to the changed voice setting. Among them, when the voice setting related to the TTS service is changed, the cache unit 720 may delete the previously cached voice data synthesized according to the previous voice setting, and when the voice data synthesized according to the changed voice setting re-cached by the cache unit 720 reaches the playable length, the decoding unit 730 decodes the cached voice data, and the playback unit 740 performs preview playback.

[0082] In addition, when the user completes editing according to the final voice settings, the device 700 can convert the cached voice data corresponding to the current voice settings into audio data in other formats such as M4A, and generate audio metadata, such as duration, sampling rate, number of channels and other information, and finally provide it to the audio and video editing SDK for use.

[0083] In addition, due to Figure 4 The method of voice preview shown can be Figure 7 The voice preview device 700 shown is used to perform, therefore, Figure 7 Any details on the operations performed by the units in Figure 4 The corresponding description will not be repeated here.

[0084] Figure 8 FIG. 8 is a block diagram of a voice preview system 800 according to an exemplary embodiment of the present disclosure.

[0085] Reference Figure 8 , the system 800 includes a terminal device 810 and a server 820. The terminal device 810 may be Figure 1 Any one of 101, 102 and 103 shown in FIG. 8 , the server 820 may be Figure 1 The server 105 is shown in FIG.

[0086] The terminal device 810 can receive text input, send the text input to the server 820, cache the voice data synthesized from the text input received from the server 820 through the TTS service in real time, and decode and play the cached voice data when the received voice data is cached to a playable length.

[0087] Since the operations performed by the terminal device 810 are respectively the same as those in the above reference Figure 6 The operation of the apparatus 600 described is the same, therefore, with respect to Figure 8 Any relevant details involved in the operations performed by the terminal device 810 in Figure 6 The corresponding description of the device 600 will not be repeated here.

[0088] The server 820 can perform speech synthesis on the text input received from the terminal device 810 through the TTS service to obtain speech data, and transmit the obtained speech data to the terminal device 810 in real time. Figure 6 The operation of the described servers is identical and therefore will not be described again here.

[0089] Fig. 9 It is a block diagram of an electronic device 900 according to an embodiment of the present disclosure. The electronic device 900 may include at least one memory 910 and at least one processor 920. A set of computer executable instructions is stored in the at least one memory 910. When the computer executable instruction set is executed by the at least one processor, the method of voice preview according to the embodiment of the present disclosure is executed.

[0090] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0091] In electronic devices, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0092] The processor can execute instructions or codes stored in the memory, wherein the memory can also store data. Instructions and data can also be sent and received through the network via the network interface device, wherein the network interface device can adopt any known transmission protocol.

[0093] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0094] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0095] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to perform a method for voice preview according to an exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0096] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by at least one processor in an electronic device to implement the voice preview method according to the exemplary embodiment of the present disclosure.

[0097] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0098] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for voice preview, characterized in that: include: Receive text input; caching in real time the speech data synthesized from the text input by the speech synthesis service; When the synthesized voice data is cached to a playable length, the cached voice data is decoded and played. wherein when the voice setting related to the speech synthesis service is changed, the speech synthesis operation of the text input according to the voice setting before the change is stopped, and the speech synthesis of the text input is restarted according to the voice setting after the change, The speech settings related to the speech synthesis service include setting at least one of the timbre, pitch, speaking speed, tone, and background music of the desired synthesized speech.

2. The method according to claim 1, characterized in that Also includes: Sending the text input to a server; The voice data synthesized from the text input by the voice synthesis service is received from the server in real time.

3. The method according to claim 2, characterized in that When the voice setting related to the speech synthesis service is changed, the notification server stops using the speech synthesis service to perform speech synthesis on the text input according to the voice setting before the change, and the notification server uses the speech synthesis service to perform speech synthesis on the text input again according to the changed voice setting, and locally re-caches the speech data synthesized from the text input by the speech synthesis service according to the changed voice setting and received in real time from the server.

4. The method according to claim 1, characterized in that Also includes: The text input is subjected to speech synthesis locally through a speech synthesis service to obtain speech data.

5. The method according to claim 4, characterized in that When the voice setting related to the speech synthesis service is changed, the speech synthesis operation of the text input using the speech synthesis service according to the voice setting before the change is stopped, the text input is re-speech synthesized using the speech synthesis service according to the changed voice setting, and the speech data synthesized from the text input by the speech synthesis service according to the changed voice setting is re-cached.

6. A device for voice preview, comprising: A receiving unit, configured to receive text input; a cache unit configured to cache in real time the speech data synthesized from the text input by the speech synthesis service; A decoding unit, configured to decode the buffered voice data when the synthesized voice data is buffered to a length that can be played; A playback unit is configured to play the decoded audio data. wherein when the voice setting related to the speech synthesis service is changed, the speech synthesis operation of the text input according to the voice setting before the change is stopped, and the speech synthesis of the text input is restarted according to the voice setting after the change, The speech settings related to the speech synthesis service include setting at least one of the timbre, pitch, speaking speed, tone, and background music of the desired synthesized speech.

7. The device according to claim 6, characterized in that Also includes: a sending unit, configured to send the text input to a server, The receiving unit is further configured to receive, from the server in real time, voice data synthesized from the text input via a voice synthesis service.

8. The device according to claim 7, characterized in that When the voice setting related to the speech synthesis service is changed, the sending unit is configured to notify the server to stop using the speech synthesis service to perform speech synthesis on the text input according to the voice setting before the change, and to notify the server to use the speech synthesis service to perform speech synthesis on the text input again according to the changed voice setting, and the caching unit is configured to re-cache the speech data synthesized from the text input by the speech synthesis service according to the changed voice setting and received in real time by the receiving unit from the server.

9. The device according to claim 6, characterized in that Also includes: The speech synthesis unit is configured to perform speech synthesis on the text input through a speech synthesis service to obtain speech data.

10. The device according to claim 9, characterized in that When the voice setting related to the speech synthesis service is changed, the speech synthesis unit is configured to stop the speech synthesis operation of the text input using the speech synthesis service according to the voice setting before the change, and re-speech synthesize the text input according to the changed voice setting, and the cache unit is configured to re-cache the speech data synthesized from the text input by the speech synthesis service according to the changed voice setting.

11. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method of any one of claims 1 to 5.

12. A speech processing system, characterized in that: include: The terminal device is configured to receive text input, send the text input to a server, cache voice data synthesized from the text input by a speech synthesis service received from the server in real time, and decode and play the cached voice data when the received voice data is cached to a playable length; as well as The server is configured to perform speech synthesis on the text input received from the terminal device through a speech synthesis service to obtain speech data, and transmit the obtained speech data to the terminal device in real time, wherein when the voice setting related to the speech synthesis service is changed, the speech synthesis operation of the text input according to the voice setting before the change is stopped, and the speech synthesis of the text input is restarted according to the voice setting after the change, The speech settings related to the speech synthesis service include setting at least one of the timbre, pitch, speaking speed, tone, and background music of the desired synthesized speech.

13. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 5.

14. A computer program product, characterized in that The instructions in the computer program product are executed by at least one processor in an electronic device to perform the method according to any one of claims 1 to 5.

Images