Voice synthesis broadcasting method and device, related equipment and computer program product
By simultaneously starting the local synthesis process and requesting server synthesis in the voice synthesis broadcast, and using real-time throughput to judge network interruption, using local audio data to continue broadcasting, the problem of audio quality degradation in poor network environment is solved, and stable audio is achieved Output and smooth user experience.
Patent Information
- Application Number
- CN202411582070.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2024-12-31
AI Technical Summary
Maintaining the stability and clarity of audio quality in various network environments is an urgent problem, especially when the network is in poor condition, traditional voice synthesis technology can easily cause interruption or reduce the smoothness of audio playback.
By starting the local synthesis process when receiving the voice synthesis broadcast instruction and requesting the server to perform voice synthesis, the throughput rate is calculated in real time to judge network interruption, and using local audio data to continue broadcasting when the throughput rate is less than the threshold, ensuring the continuity of the audio data .
It realizes a coherent user experience during network fluctuations, avoids user perception of service interruptions, ensures sound quality and improves tolerance for network latency.
Smart Images
Figure CN119229841A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech synthesis technology, and more specifically, to a speech synthesis broadcasting method, apparatus, related equipment and computer program product. Background Art
[0002] With the continuous advancement of technology, various mobile devices with speech synthesis technology have sprung up. These devices are deeply loved by customers for their convenient broadcasting functions. However, with the increase in market demand, customers are increasingly demanding on the audio quality of broadcasts. They not only expect to get a high-quality audio experience when the network is good, but also hope to enjoy high-quality audio effects in occasional poor network scenarios.
[0003] Therefore, how to maintain the stability and clarity of audio quality in various network environments has become an urgent problem for R&D personnel to solve. Summary of the invention
[0004] In view of the above problems, this application is proposed to provide a speech synthesis broadcasting method, apparatus, related equipment and computer program product to improve the quality of synthesized speech broadcasted in various network environments. The specific scheme is as follows:
[0005] In a first aspect, a speech synthesis broadcasting method is provided, comprising:
[0006] When receiving the speech synthesis broadcast instruction, starting the local synthesis process of the target text, saving the synthesized local audio data, and sending the target text to the server to request the server to perform speech synthesis on the target text;
[0007] Continuously receive the synthesized audio data returned by the server, and play the audio data returned by the server according to the set throwing rules;
[0008] Calculate the throughput rate in real time, where the throughput rate is the ratio of the duration of the audio that has been thrown out to the duration required to generate the audio;
[0009] If it is determined that the currently calculated throughput rate is less than the set threshold, the local audio data is used for continued broadcasting.
[0010] In a possible design, in another implementation of the first aspect of the embodiment of the present application, while first receiving the synthesized audio data returned by the server, first duration information returned by the server is also received, wherein the first duration information includes the audio duration corresponding to each word in the target text calculated by the server;
[0011] The process of throwing out the audio data returned by the server for playback according to the set throwing rules includes:
[0012] After receiving the audio data returned by the server each time, temporarily store the received audio data and calculate the total number of bytes of the remaining audio data after deducting the audio data that has been thrown out from all the audio data returned by the server so far;
[0013] Determine whether the total number of bytes is not less than the number of bytes of the audio data corresponding to the i+1th word, wherein i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text, and the number of bytes of the audio data corresponding to the i+1th word is the product of the audio duration corresponding to the i+1th word in the first duration information and the audio sampling rate of the server;
[0014] If so, select the audio data to be thrown out this time from the audio data returned by the server that is temporarily stored locally, and throw it out for playback.
[0015] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of selecting the audio data to be thrown out this time from the audio data returned by the locally temporarily stored server and throwing it out for playback includes:
[0016] The audio data of the i+1th word and the words before it in the audio data returned by the server and temporarily stored locally are thrown out for playback.
[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of selecting the audio data to be thrown out this time from the audio data returned by the locally temporarily stored server and throwing it out for playback includes:
[0018] The audio data of each word before the i+1th word in the audio data returned by the server and stored locally, as well as the audio data of the i+1th word excluding the audio data of the first set time length at the tail are thrown out for playback.
[0019] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of using the local audio data for continued broadcasting includes:
[0020] Take the audio data of the first set time length at the end of the i-th word in the audio data returned by the server, and take the audio data of the second set time length at the beginning of the i+1-th word in the local audio data, perform smoothing, throw out the smoothed audio data for playback, and then continue to play the audio data starting from the second set time length of the i+1-th word in the local audio data;
[0021] Wherein, i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text.
[0022] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the storing of the synthesized local audio data includes:
[0023] The local audio data of a set number m of words after the synthesized i-th word are saved, and the local audio data of the i-th word and the words before the i-th word are deleted;
[0024] Wherein, i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text.
[0025] In a possible design, in another implementation of the first aspect of the embodiment of the present application, after the local synthesis process is started, second duration information is calculated, and the second duration information includes the audio duration corresponding to each word in the target text calculated locally;
[0026] The process of saving the local audio data of a set number m of words after the synthesized i-th word includes:
[0027] According to the audio duration of each word in the second duration information, local audio data of a set number m of words after the ith word are selected in the synthesized local audio data stream for storage.
[0028] In a possible design, in another implementation of the first aspect of the embodiment of the present application, after starting the local synthesis process of the target text, the method further includes:
[0029] After each time the audio data synthesized by the server is thrown out, the last target word corresponding to the local audio data to be saved is determined based on the latest i value, and the local synthesis process is blocked after the local synthesis process completes the synthesis of the audio data corresponding to the target word, until the local synthesis process is started again when the start condition is met, and the start condition includes: determining that the local synthesis process needs to be continued based on the latest i value, or determining that the currently calculated throughput is less than a set threshold.
[0030] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, when it is determined that the currently calculated throughput is less than a set threshold, the method further includes:
[0031] Notify the server to terminate speech synthesis.
[0032] In a second aspect, a speech synthesis and broadcasting device is provided, comprising:
[0033] The instruction response unit is used to start the local synthesis process of the target text when receiving the speech synthesis broadcast instruction, save the synthesized local audio data, and send the target text to the server to request the server to perform speech synthesis on the target text;
[0034] The audio data receiving and throwing unit is used to continuously receive the synthesized audio data returned by the server, and throw the audio data returned by the server for playback according to the set throwing rules;
[0035] A throughput calculation unit, used for calculating the throughput in real time, wherein the throughput is the ratio of the duration of the audio that has been thrown out to the duration required to generate this audio;
[0036] The replay unit is used to use the local audio data to replay if it is determined that the currently calculated throughput rate is less than a set threshold.
[0037] In a third aspect, an electronic device is provided, comprising: a memory and a processor;
[0038] The memory is used to store programs;
[0039] The processor is used to execute the program to implement the speech synthesis and broadcasting method described in any one of the first aspects of the present application.
[0040] In a fourth aspect, a speech synthesis and broadcasting system is provided, comprising: a terminal and a server;
[0041] The terminal is used to start the local synthesis process of the target text when receiving the speech synthesis broadcast instruction, save the synthesized local audio data, and send the target text to the server to request the server to perform speech synthesis on the target text; continuously receive the synthesized audio data returned by the server, and according to the set throwing rule, throw the audio data returned by the server for playback; calculate the throughput rate in real time, and the throughput rate is the ratio of the audio duration that has been thrown to the duration required to generate this audio; if it is determined that the currently calculated throughput rate is less than the set threshold, then use the local audio data for continued playback;
[0042] The server is used to perform speech synthesis on the target text and return the latest synthesized audio data to the terminal according to a set strategy.
[0043] In a fifth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis and broadcasting method described in any one of the first aspects of the present application is implemented.
[0044] In a sixth aspect, a computer program product is provided, including a computer program, which, when executed by a processor, implements the speech synthesis and broadcasting method described in any one of the first aspects of the present application.
[0045] By means of the above technical solution, when receiving the speech synthesis broadcast instruction, the present application starts the local synthesis process and requests the server to perform speech synthesis at the same time, and the server transmits the synthesized audio data to the terminal through the network, and the terminal throws out the audio data returned by the server according to the set throwing rules for playback. With the help of the powerful computing power of the server, the quality of the synthesized audio data can be guaranteed. In addition, in order to more timely and accurately judge whether the network transmission is interrupted, the present application can calculate the throughput rate in real time, and use the throughput rate as the judgment indicator of whether the network is interrupted. The throughput rate can timely judge whether the network transmission has or is about to be interrupted from the user's auditory perspective. When it is determined that the currently calculated throughput rate is less than the set threshold, it means that the network transmission has or is about to be interrupted. At this time, in order to continuously provide relatively stable audio data, the present application can use the locally synthesized audio data to replace the server-synthesized audio data for continued broadcasting, so that a coherent user experience can be maintained when the network fluctuates, thereby improving the quality of voice broadcasting. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0047] Figure 1 A schematic diagram of an implementation system architecture of the speech synthesis and broadcasting method provided in an embodiment of the present application;
[0048] Figure 2 A schematic diagram of a terminal structure provided in an embodiment of the present application;
[0049] Figure 3 A schematic diagram of a server structure provided in an embodiment of the present application;
[0050] Figure 4 A flowchart of a speech synthesis and broadcasting method provided in an embodiment of the present application;
[0051] Figure 5 Another flowchart of the speech synthesis and broadcasting method provided in the embodiment of the present application;
[0052] Figure 6 An example diagram of a breakpoint connection scenario provided in an embodiment of the present application;
[0053] Figure 7Another flowchart of the speech synthesis and broadcasting method provided in the embodiment of the present application;
[0054] Figure 8 A schematic diagram of the structure of a speech synthesis and broadcasting device provided in an embodiment of the present application;
[0055] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0057] This application provides a speech synthesis broadcasting method, which can be applied to Figure 1 The system architecture shown in FIG. 1 may include a terminal 100 and a server 200. The server 200 may be a physical server or a cloud server. The server 200 may include one or more servers ( Figure 1 A server is included as an example for explanation).
[0058] The terminal 100 and the server 200 can be used together to execute the speech synthesis and broadcasting method provided in the embodiment of the present application.
[0059] Next describe Figure 1 The product form of the mid-terminal 100;
[0060] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a learning machine, a wearable device, a vehicle-mounted device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.
[0061] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.
[0062] refer to Figure 2As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160, a speaker 161, a microphone 162, an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.
[0063] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.
[0064] Among them, other input devices 132 can receive input data and so on.
[0065] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display various interactive interfaces, processing results, etc. in the speech synthesis broadcasting method.
[0066] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.
[0067] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.
[0068] Among them, the memory 120 can be used to store software codes related to the speech synthesis and broadcasting method, the processor 170 can execute the steps of the speech synthesis and broadcasting method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.
[0069] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.
[0070] In the embodiment of the present application, the RF unit 110 can send data to the server 200 and receive the processing result sent by the server 200. Exemplarily, the RF unit 110 sends the target text to be synthesized to the server 200, and the server 200 performs speech synthesis on the target text and returns the synthesized audio data to the terminal 100 for voice broadcast.
[0071] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.
[0072] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.
[0073] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .
[0074] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.
[0075] Next describe Figure 1 The product form of the server 200;
[0076] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.
[0077] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0078] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0079] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0080] The memory 204 may be used to store software codes related to the speech synthesis and announcement method, and the processor 202 may execute the steps of the speech synthesis and announcement method, and may also schedule other units to implement corresponding functions.
[0081] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.
[0082] High-quality audio synthesis on mobile devices faces computing power limitations and usually relies on server processing. However, when the network is unstable, the traditional approach is to either interrupt speech synthesis or switch to lower-quality local synthesis. An existing speech synthesis process includes the following steps:
[0083] 1. The user sends a data synthesis request to the cloud server through the terminal.
[0084] 2. After receiving the request, the cloud server starts to perform speech synthesis on the text to be synthesized and returns the synthesis result to the client in real time.
[0085] 3. The client sets a timeout. If it does not receive the synthesis result from the cloud server within the timeout, it is considered that there is a problem in the communication with the cloud server.
[0086] 4. Once a timeout occurs, the client immediately stops waiting for the result from the cloud server and starts the local backup engine instead, handing over the text to be synthesized to the local engine for processing.
[0087] 5. The local engine processes and synthesizes data according to established logic and algorithms.
[0088] 6. After the synthesis is completed, the client outputs the broadcast.
[0089] Analysis of the existing solution shows that it has the following defects: the terminal uses a fixed timeout period to judge network interruption, which has low timeliness and makes the network interruption obvious. In addition, the local synthesis process is started only after the timeout period, which further reduces the smoothness of audio playback.
[0090] To solve this problem, this application proposes an innovative solution: using the limited computing power of mobile devices for basic synthesis, and using intelligent caching and network interruption optimization judgment methods to ensure timely detection of network anomalies, and to continue to provide relatively stable audio output even in a network-disconnected environment. This can not only ensure sound quality, but also maintain a consistent user experience when the network fluctuates, avoiding users from obviously perceiving service interruptions.
[0091] The present invention provides a method for synthesizing speech. Figure 1 Taking the computer device in the example as an example, the computer device can be Figure 1 The terminal 100 in FIG. Figure 4 The speech synthesis broadcasting method specifically comprises the following steps:
[0092] Step S100: receiving a speech synthesis broadcast instruction.
[0093] Specifically, the speech synthesis and broadcast instruction is used to instruct to perform speech synthesis and broadcast on the target text. After receiving the speech synthesis and broadcast instruction, in response to the instruction, steps S110a, S110b and S110c are executed.
[0094] Among them, step S110a, step S110b and step S110c can be executed synchronously.
[0095] Step S110a: start the local synthesis process for the target text.
[0096] Specifically, a speech synthesis engine may be started locally on the terminal to execute a local speech synthesis process of the target text to synthesize local audio data in a streaming manner, and then step S120a is executed.
[0097] Step S120a: Save the synthesized local audio data.
[0098] Specifically, the local audio data synthesized by the local speech synthesis engine can be cached for backup, and when the network between the terminal and the server is interrupted, the local audio data can be used for continued broadcasting.
[0099] Step S110b: request the server to perform speech synthesis.
[0100] Specifically, considering that the computing power of the server is generally stronger than that of the terminal, the powerful computing power of the server can be used to synthesize higher quality audio data. When the terminal responds to the speech synthesis broadcast instruction, the terminal can send the target text to be synthesized to the server to request the server to perform speech synthesis. After that, steps S120b-S130b are executed.
[0101] Step S120b: Continue to receive the synthesized audio data returned by the server.
[0102] Specifically, the server side can start a speech synthesis engine to perform speech synthesis on the target text. The speech synthesis can be streaming, and the synthesized audio segments can be continuously sent to the terminal device according to the set strategy, for example, the latest synthesized audio data is sent to the terminal at a set time interval, or the audio data of a fixed length is sent to the terminal after being synthesized, etc. Then the terminal continuously receives the synthesized audio data returned by the server.
[0103] Step S130b: according to the set throwing rules, the audio data returned by the server is thrown out for playing.
[0104] Specifically, when the network between the terminal and the server is not interrupted, the terminal can continuously receive the synthesized audio data returned by the server, and then can prioritize the audio data returned by the server for playback according to the set throwing rules, so as to play high-quality audio data externally.
[0105] The term "throwing" in this application refers to outputting audio data externally, such as sending audio data to a player. Taking the speech synthesis broadcasting scenario of a vehicle terminal as an example, the audio data is sent to the audio track provided by the system.
[0106] In this embodiment, when the terminal throws the audio data returned by the server, the terminal can execute the throwing of the audio data according to the pre-configured throwing rules. Among them, the throwing rules can specify the throwing time and throwing quantity of the audio data. An example of an optional throwing rule is as follows: the audio data is thrown only when it is determined that a complete word of audio data has been received, so as to ensure the integrity of the thrown audio data.
[0107] Step S110c: Calculate the throughput rate in real time.
[0108] In this embodiment, when responding to the speech synthesis broadcast instruction, the throughput rate can be calculated in real time through a separate thread. The "real-time calculation" referred to in this application can be understood as calculation according to a very small time period. Therefore, when an abnormality occurs in the network, the throughput rate calculated in real time can be used to quickly detect network abnormalities.
[0109] The throughput rate in this step can be expressed as the ratio of the duration of the audio that has been thrown out to the duration required to generate this audio. The formula is as follows:
[0110] f=T1 / T2
[0111] T1=getAudioSampleCount / audioRate
[0112] Where f represents the throughput rate, T1 represents the duration of the audio that has been thrown, getAudioSampleCount represents the number of audio sampling points corresponding to the audio data sent by the server that has been thrown by the terminal, audioRate represents the sampling rate of the audio data sent by the server, and T2 represents the duration required to generate this audio, which can also be expressed as the time interval from the first frame of audio thrown to the current moment.
[0113] Compared with the prior art that uses a fixed timeout as a criterion for determining whether a network is interrupted, the present application uses throughput as a criterion, which can predict whether a network interruption is about to occur or has occurred based on the audio data that has been sent out from the user's hearing perspective, and can more timely and accurately predict the result of whether a network interruption occurs. This makes it easier to use local audio data to resume playback in a timely manner after determining that the network is interrupted, thereby ensuring the continuity of the played audio data.
[0114] Step S120c, determine whether the throughput rate is less than the set threshold, if so, execute step S130c, if not, return to execute step S110c.
[0115] Specifically, in order to measure whether a network interruption occurs, the present application may pre-configure a threshold of the throughput rate, and the threshold may be a value not less than 1 to avoid unnatural sensations such as plosive sounds.
[0116] In a possible implementation, it has been verified experimentally that setting the threshold value to 1.1 has a better effect, which can not only ensure that the server will not be easily interrupted, but also ensure the smooth connection of audio data without unnatural sounds such as plosives.
[0117] Step S130c: continue broadcasting using local audio data.
[0118] When the throughput rate is less than the set threshold, it indicates that the network is interrupted. In order to ensure the continuity of the output audio, the synthesized local audio data can be used for continued playback.
[0119] In one possible implementation, when the throughput rate is less than a set threshold, considering that the network between the terminal and the server has been interrupted and the local audio data can be used for continued broadcasting, the server can be further notified in this embodiment to terminate speech synthesis, thereby reducing the server's invalid resource overhead.
[0120] The speech synthesis and broadcasting method provided in the embodiment of the present application, when receiving the speech synthesis and broadcasting instruction, simultaneously starts the local synthesis process and requests the server to perform speech synthesis, and the server transmits the synthesized audio data to the terminal through the network, and the terminal throws out the audio data returned by the server according to the set throwing rules for playback. With the help of the powerful computing power of the server, the quality of the synthesized audio data can be guaranteed. In addition, in order to more timely and accurately judge whether the network transmission is interrupted, the present application can calculate the throughput rate in real time, and use the throughput rate as the judgment index of whether the network is interrupted. The throughput rate can timely judge whether the network transmission has or is about to be interrupted from the user's hearing perspective. When it is determined that the currently calculated throughput rate is less than the set threshold, it means that the network transmission has or is about to be interrupted, and the audio will not be able to be connected in time. There will be unnatural problems such as jamming and blasting sounds in the sense of hearing. At this time, in order to continuously provide relatively stable audio data, the present application can use the locally synthesized audio data to replace the server synthesized audio data for continued broadcasting, while maximizing the tolerance of network delay, it can also ensure that a coherent user experience can be maintained when the network fluctuates, thereby improving the quality of speech broadcasting.
[0121] In some embodiments of the present application, Figure 5 As shown, another speech synthesis broadcasting method is provided, wherein step S200 is the same as step S100, steps S210a-S220a are the same as steps S110a-S120a in the above embodiment, and steps S210c-S230c are the same as steps S110c-S130c in the above embodiment.
[0122] Compared with the previous embodiment, the process branch for requesting the server to perform speech synthesis is improved in this embodiment. In this embodiment, only the improved branch steps are introduced, and the other same steps can refer to the relevant introduction of the previous embodiment.
[0123] Step S210b: request the server to perform speech synthesis.
[0124] Step S220b: receiving the synthesized audio data and the first duration information returned by the server.
[0125] Specifically, compared to the previous embodiment, in this embodiment, when the server receives the target text to be synthesized, it first processes the target text, calculates the audio duration corresponding to each word therein, and uses it as the first duration information. When the synthesized audio data is returned to the terminal for the first time, the first duration information is also returned to the terminal.
[0126] Step S230b: temporarily store the audio data received this time and calculate the total number of bytes of the remaining audio data after subtracting the audio data that has been thrown out from all the audio data returned by the server received so far.
[0127] Specifically, each time after receiving audio data returned by the server, the terminal temporarily stores the received audio data and calculates the total number of bytes of remaining audio data after deducting the discarded audio data from all the audio data returned by the server received so far.
[0128] Step S240b, determine whether the total number of bytes is not less than the number of bytes of the audio data corresponding to the i+1th word.
[0129] Among them, i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text. Taking the target text as "Thank you for your support for our work" as an example, assuming that the last word currently thrown out is "I", the current value of i is 5. The number of bytes of the audio data corresponding to the i+1th word is the product of the audio duration corresponding to the i+1th word in the first duration information and the audio sampling rate of the server.
[0130] In this step, it is determined whether the total number of bytes is not less than the number of bytes of the audio data corresponding to the i+1th word, that is, it is determined whether the local has received the complete audio data of the i+1th word. When the judgment result in step S240b is yes, it means that the local has received the complete audio data of the i+1th word, and step S250b is executed. Otherwise, the process returns to step S220b to continue receiving the audio data returned by the server.
[0131] Step S250b: Select the audio data to be thrown out this time from the audio data returned by the server and temporarily stored locally, throw it out for playing, and update the value of i.
[0132] Specifically, in this embodiment, when the terminal throws out the audio data returned by the server, it determines that the local has received the complete audio data of the (i+1)th word, thereby determining the integrity of the thrown audio data.
[0133] After throwing out the selected audio data, the i value can be updated to set i=i+1.
[0134] For the above step S250b, this embodiment provides two optional implementation methods.
[0135] The first optional implementation method is to play the audio data of the i+1th word and the words before it in the audio data returned by the server and temporarily stored locally.
[0136] A second optional implementation method is to play the audio data of each word before the i+1th word and the audio data of the i+1th word excluding the audio data of the first set time length at the end of the audio data returned by the server that is temporarily stored locally.
[0137] In the second implementation method mentioned above, the terminal does not throw out the complete audio data of the i+1th word returned by the server, but retains the audio data of the first set duration at the tail, so that when a network interruption occurs when the server transmits the audio data of the next word to the terminal, the locally temporarily stored audio data of the first set duration at the tail of the i+1th word returned by the server and the audio data of the second set duration at the head of the audio data of the next word synthesized by the local engine can be used for smoothing, thereby improving the listening experience of the connecting part of the audio and avoiding unnatural listening problems such as explosive sounds.
[0138] In some embodiments of the present application, the process of using local audio data to resume broadcasting in step S130c in the aforementioned embodiment is described.
[0139] In a possible implementation, the sorting number of the last word corresponding to the audio data synthesized by the server that has been currently thrown out in the target text is defined as i. When the server sends a network interruption when transmitting the audio data of the i+1th word to the terminal, the local audio data starting from the i+1th word synthesized by the local engine can be used for continued playback.
[0140] In another possible implementation, in order to improve the listening experience at the connection between the audio data synthesized by the server and the audio data synthesized locally, the audio data synthesized by the server and the audio data synthesized locally may be smoothed at the connection.
[0141] Specifically, the audio data of the first set length at the end of the i-th word in the audio data returned by the server and the audio data of the second set length at the beginning of the i+1-th word in the local audio data are taken, smoothed, the smoothed audio data are played out, and then the audio data starting from the second set length of the i+1-th word in the local audio data are used to continue playing.
[0142] The first set time length and the second set time length may be the same or different. In a possible implementation, the first set time length and the second set time length are the same, and both are 20 ms.
[0143] The resumed broadcast method provided in this embodiment can avoid unnatural listening problems such as plosive sounds by smoothing the connection points, thereby improving the listening experience of the resumed broadcast audio at the connection points.
[0144] Combination Figure 6 As shown, it provides an example diagram of a breakpoint connection scenario.
[0145] The target text is "Thank you for your support of our work."
[0146] The audio data synthesized by the cloud server Cloud is sent to the terminal Local, and a network interruption occurs when the audio data corresponding to the "party" is transmitted.
[0147] Then, the last 20ms of the audio data corresponding to the word "I" synthesized by the cloud server Cloud can be taken, and the first 20ms of the audio data corresponding to the word "Fang" synthesized by the terminal Local can be taken, and the two audio segments can be smoothed.
[0148] Figure 6 The example below shows how to smooth the audio at the joint.
[0149] Among them, after smoothing the two selected 20 ms audio segments, 20 ms of audio data after smoothing is obtained.
[0150] The process of smoothing the audio may use existing audio processing tools or algorithms, which will not be described in detail here.
[0151] In some embodiments of the present application, the process of saving the synthesized local audio data in step S120a in the aforementioned embodiment is described.
[0152] In a possible implementation, after starting the local synthesis process, the local speech synthesis engine can generate local audio data in a streaming manner, and the generated local audio data can be saved.
[0153] In another possible implementation, in order to reduce memory usage, only the local audio data of a set number m of words after the synthesized ith word may be saved, and the local audio data of the ith word and the words before the ith word may be deleted.
[0154] Wherein, i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text.
[0155] Since the audio data of the i-th word and the previous words have been thrown out for playback, there is no need to play the audio of the i-th word and the previous words even if a network interruption occurs. Therefore, the storage of the local audio data of the i-th word and the previous words can be abandoned, which can also reduce memory usage.
[0156] Reference Figure 7 , Figure 7 Another flowchart of the speech synthesis and broadcasting method of the present application is illustrated.
[0157] Among them, step S300 is the same as step S100, steps S310c-S330c are the same as steps S110c-S130c in the above embodiment, and steps S310b-S340b are the same as steps S210b-S240b in the above embodiment.
[0158] Compared with the above-mentioned embodiment, the process branch of the terminal local synthesis is improved in this embodiment. In this embodiment, only the improved branch steps are introduced, and the other same steps can refer to the relevant introduction of the above-mentioned embodiment.
[0159] Step S310a, start the local synthesis process.
[0160] Step S320a: Process the text to obtain first duration information.
[0161] Specifically, after starting the local synthesis process, the target text is first processed, including but not limited to: word segmentation, prosody processing, phoneme processing, etc. The phoneme processing process is to calculate the audio duration corresponding to each word in the target text, and then synthesize the audio data corresponding to each word according to the audio duration.
[0162] In this step, during the text processing, second duration information is obtained, and the second duration information includes the audio duration corresponding to each word in the target text obtained by local calculation.
[0163] Before synthesizing the audio, the present application pre-calculates the audio duration corresponding to each word, so that the number of audio data bytes of each word can be accurately determined based on the information, thereby better controlling the storage of local audio data.
[0164] Step S330a, save local audio data of i+m words.
[0165] Specifically, according to the audio duration of each word in the second duration information, local audio data of a set number m of words after the ith word may be selected in the synthesized local audio data stream for storage.
[0166] Here, m is a preset value. For example, m can be 5 or other values.
[0167] Still taking the target text "Thank you for your support for our work" as an example, m=5. Assuming that the i-th word currently thrown out is "I", it is necessary to save the local audio data corresponding to "the support of our work".
[0168] like Figure 7 As shown, after the i value is updated in step S340b, the execution of step S330a can be triggered, that is, based on the latest i value, the local audio data to be saved is updated.
[0169] According to the method of this embodiment, the terminal does not need to keep all the audio data corresponding to the target text locally, and the local audio data corresponding to the discarded words can be deleted in time to reduce memory usage. In addition, in order to ensure that the local audio data can be played back in time after the network is interrupted, the size of m can be reasonably set, that is, while ensuring the smoothness of the local audio data, the value of m can be minimized to reduce memory usage.
[0170] In some possible implementations, after starting the local synthesis process for the target text, the following processing steps may be further included:
[0171] After each time the audio data synthesized by the server is thrown out, the last target word corresponding to the local audio data to be saved is determined based on the latest i value, and the local synthesis process is blocked after the local synthesis process completes synthesizing the audio data corresponding to the target word until the local synthesis process is started again when the start conditions are met.
[0172] The start condition includes: determining that the local synthesis process needs to be continued based on the latest i value, or determining that the currently calculated throughput is less than a set threshold.
[0173] Specifically, the above embodiment introduces that only the local audio data of i+m words need to be saved locally. In this embodiment, the start and stop of the local synthesis process can be controlled. After determining that the local audio data of i+m words have been synthesized locally, the local synthesis process can be blocked and speech synthesis can be stopped until it is determined that the local synthesis process needs to be continued to synthesize new audio data according to the updated i value, or, according to the throughput calculated in real time, it is determined that the throughput is less than the set threshold, that is, it is determined that the network is interrupted, and the local synthesis process can be started to continue to synthesize speech backward.
[0174] In this embodiment, the start and stop of the local speech synthesis process can be controlled to ensure that the terminal stores enough local audio data (local audio data of i+m words) to block the local synthesis process, thereby further reducing the local computing resource overhead of the terminal.
[0175] From the above embodiments, it can be seen that the speech synthesis and broadcasting method provided by the present application includes but is not limited to the following advantages:
[0176] 1. Whether the network is interrupted is determined by real-time throughput detection rather than by a fixed timeout period. This ensures that when the audio is about to be unable to be connected in time and there is about to be a freeze or popping sound, the subsequent audio is connected in time. This maximizes the tolerance for network delays while allowing audio to be connected seamlessly.
[0177] 2. The server and the terminal perform speech synthesis simultaneously. After the network is interrupted, the audio at the connection point is smoothed by using local audio and server-synthesized audio, which improves the listening experience of the continued audio and avoids problems such as plosive sounds.
[0178] 3. The server can send the calculated first duration information (the audio duration corresponding to each word calculated by the server) to the terminal in advance, and the terminal can accurately determine the number of audio data bytes for each word based on the first duration information, so as to better control the audio output. After starting the local synthesis process, the terminal calculates the second duration information (the audio duration corresponding to each word calculated locally) in advance, and the terminal can accurately control the length of the local audio data to be saved based on the second duration information.
[0179] 4. The terminal will execute the audio throwing action only after ensuring that a complete word of audio data has been received. This can ensure the integrity of the thrown audio data and ensure that the words are connected when the network is disconnected to avoid unnatural listening.
[0180] The speech synthesis and announcement device provided in the embodiment of the present application is described below. The speech synthesis and announcement device described below and the speech synthesis and announcement method described above can be referred to each other.
[0181] See also Figure 8 , Figure 8 A schematic diagram of the structure of a speech synthesis and broadcasting device disclosed in an embodiment of the present application.
[0182] like Figure 8 As shown, the device may include:
[0183] The instruction response unit 11 is used to start the local synthesis process of the target text when receiving the speech synthesis broadcast instruction, save the synthesized local audio data, and send the target text to the server to request the server to perform speech synthesis on the target text;
[0184] The audio data receiving and throwing unit 12 is used to continuously receive the synthesized audio data returned by the server, and throw the audio data returned by the server for playback according to the set throwing rules;
[0185] A throughput calculation unit 13, used for calculating the throughput in real time, wherein the throughput is the ratio of the duration of the audio that has been thrown out to the duration required to generate the audio;
[0186] The replay unit 14 is configured to continue playing the local audio data if it is determined that the currently calculated throughput rate is less than a set threshold.
[0187] In a possible implementation, the audio data receiving and throwing unit receives the synthesized audio data returned by the server for the first time and also receives the first duration information returned by the server, wherein the first duration information includes the audio duration corresponding to each word in the target text calculated by the server, and on this basis, according to the set throwing rule, throws the audio data returned by the server for playback, including:
[0188] After receiving the audio data returned by the server each time, temporarily store the received audio data and calculate the total number of bytes of the remaining audio data after deducting the audio data that has been thrown out from all the audio data returned by the server so far;
[0189] Determine whether the total number of bytes is not less than the number of bytes of the audio data corresponding to the i+1th word, wherein i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text, and the number of bytes of the audio data corresponding to the i+1th word is the product of the audio duration corresponding to the i+1th word in the first duration information and the audio sampling rate of the server;
[0190] If so, select the audio data to be thrown out this time from the audio data returned by the server that is temporarily stored locally, and throw it out for playback.
[0191] In a possible implementation, the audio data receiving and throwing unit selects the audio data to be thrown out this time from the audio data returned by the server and temporarily stored locally, and throws out the process of playing, including:
[0192] The audio data of the i+1th word and the words before it in the audio data returned by the server and temporarily stored locally are thrown out for playback.
[0193] In another possible implementation, the audio data receiving and throwing unit selects the audio data to be thrown out this time from the audio data returned by the server and temporarily stored locally, and throws out the process of playing, including:
[0194] The audio data of each word before the i+1th word in the audio data returned by the server and stored locally, as well as the audio data of the i+1th word excluding the audio data of the first set time length at the tail are thrown out for playback.
[0195] In a possible implementation, the process of the replay unit replaying the local audio data includes:
[0196] Take the audio data of the first set time length at the end of the i-th word in the audio data returned by the server, and take the audio data of the second set time length at the beginning of the i+1-th word in the local audio data, perform smoothing, throw out the smoothed audio data for playback, and then continue to play the audio data starting from the second set time length of the i+1-th word in the local audio data;
[0197] Wherein, i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text.
[0198] In a possible implementation, the process of the instruction response unit saving the synthesized local audio data includes:
[0199] The local audio data of a set number m of words after the synthesized i-th word are saved, and the local audio data of the i-th word and the words before the i-th word are deleted;
[0200] Wherein, i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text.
[0201] In a possible implementation, after the instruction response unit starts the local synthesis process, the second duration information is calculated, and the second duration information includes the audio duration corresponding to each word in the target text calculated locally. On this basis, the process of saving the local audio data of a set number m of words after the synthesized i-th word includes:
[0202] According to the audio duration of each word in the second duration information, local audio data of a set number m of words after the ith word are selected in the synthesized local audio data stream for storage.
[0203] In a possible implementation, the device of the present application may further include:
[0204] A local synthesis process control unit is used to determine the last target word corresponding to the local audio data to be saved based on the latest i value after the instruction response unit starts the local synthesis process for the target text and after each time the audio data synthesized by the server is thrown out, and to block the local synthesis process after the local synthesis process completes the synthesis of the audio data corresponding to the target word until the local synthesis process is started again when the start condition is met, and the start condition includes: determining the need to continue the local synthesis process based on the latest i value, or determining that the currently calculated throughput is less than a set threshold.
[0205] In a possible implementation, the device of the present application may further include:
[0206] The synthesis termination notification unit is used to notify the server to terminate the speech synthesis when it is determined that the currently calculated throughput rate is less than a set threshold.
[0207] The present application also provides a speech synthesis and announcement system in an embodiment, including: a terminal and a server. The terminal is used to execute each step of the speech synthesis and announcement method described in any of the above embodiments, such as:
[0208] When the terminal receives the speech synthesis broadcast instruction, it starts the local synthesis process for the target text, saves the synthesized local audio data, and sends the target text to the server to request the server to perform speech synthesis on the target text; continuously receives the synthesized audio data returned by the server, and plays the audio data returned by the server according to the set throwing rules; calculates the throughput in real time; if it is determined that the currently calculated throughput is less than the set threshold, the local audio data is used to continue playing.
[0209] The server is used to perform speech synthesis on the target text and return the latest synthesized audio data to the terminal according to a set strategy.
[0210] Furthermore, the server can also be used to cooperate with the speech synthesis broadcasting method of the terminal to provide corresponding data. For example, the server calculates the first duration information according to the target text and sends it to the terminal, etc.
[0211] The present application also provides an electronic device in an embodiment. Fig. 9 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, learning machines, teaching large screens, wearable devices, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0212] like Fig. 9 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 to the random access memory (RAM) 603 to implement the speech synthesis broadcasting method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.
[0213] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0214] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the speech synthesis and broadcasting methods provided in the embodiments of the present application.
[0215] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the speech synthesis and broadcasting methods provided in the embodiments of the present application.
[0216] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0217] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0218] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0219] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
[0220] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.
Claims
1. A speech synthesis and broadcasting method, characterized in that: include: When receiving the speech synthesis broadcast instruction, starting the local synthesis process of the target text, saving the synthesized local audio data, and sending the target text to the server to request the server to perform speech synthesis on the target text; Continuously receive the synthesized audio data returned by the server, and play the audio data returned by the server according to the set throwing rules; Calculate the throughput rate in real time, where the throughput rate is the ratio of the duration of the audio that has been thrown out to the duration required to generate the audio; If it is determined that the currently calculated throughput rate is less than the set threshold, the local audio data is used for continued broadcasting.
2. The method according to claim 1, characterized in that When the synthesized audio data returned by the server is first received, first duration information returned by the server is also received, wherein the first duration information includes the audio duration corresponding to each word in the target text calculated by the server; The process of throwing out the audio data returned by the server for playback according to the set throwing rules includes: After receiving the audio data returned by the server each time, temporarily store the received audio data and calculate the total number of bytes of the remaining audio data after deducting the audio data that has been thrown out from all the audio data returned by the server so far; Determine whether the total number of bytes is not less than the number of bytes of the audio data corresponding to the i+1th word, wherein i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text, and the number of bytes of the audio data corresponding to the i+1th word is the product of the audio duration corresponding to the i+1th word in the first duration information and the audio sampling rate of the server; If so, select the audio data to be thrown out this time from the audio data returned by the server that is temporarily stored locally, and throw it out for playback.
3. The method according to claim 2, characterized in that The process of selecting the audio data to be thrown out this time from the audio data returned by the server stored locally and throwing it out for playing includes: The audio data of the i+1th word and the words before it in the audio data returned by the server and temporarily stored locally are thrown out for playback.
4. The method according to claim 2, characterized in that: The process of selecting the audio data to be thrown out this time from the audio data returned by the server stored locally and throwing it out for playing includes: The audio data of each word before the i+1th word in the audio data returned by the server and stored locally, as well as the audio data of the i+1th word excluding the audio data of the first set time length at the tail are thrown out for playback.
5. The method according to claim 1, characterized in that The process of using the local audio data to resume playing includes: Take the audio data of the first set time length at the end of the i-th word in the audio data returned by the server, and take the audio data of the second set time length at the beginning of the i+1-th word in the local audio data, perform smoothing, throw out the smoothed audio data for playback, and then continue to play the audio data starting from the second set time length of the i+1-th word in the local audio data; Wherein, i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text.
6. The method according to claim 1, characterized in that The storing of the synthesized local audio data comprises: The local audio data of a set number m of words after the synthesized i-th word are saved, and the local audio data of the i-th word and the words before the i-th word are deleted; Wherein, i is the sorting number of the last word corresponding to the audio data synthesized by the server that has been thrown out in the target text.
7. The method according to claim 6, characterized in that After the local synthesis process is started, second duration information is calculated, and the second duration information includes the audio duration corresponding to each word in the target text calculated locally; The process of saving the local audio data of a set number m of words after the synthesized i-th word includes: According to the audio duration of each word in the second duration information, local audio data of a set number m of words after the ith word are selected in the synthesized local audio data stream for storage.
8. The method according to claim 6, characterized in that After starting the local synthesis process for the target text, it also includes: After each time the audio data synthesized by the server is thrown out, the last target word corresponding to the local audio data to be saved is determined based on the latest i value, and the local synthesis process is blocked after the local synthesis process completes the synthesis of the audio data corresponding to the target word, until the local synthesis process is started again when the start condition is met, and the start condition includes: determining that the local synthesis process needs to be continued based on the latest i value, or determining that the currently calculated throughput is less than the set threshold.
9. The method according to any one of claims 1 to 8, characterized in that: When it is determined that the currently calculated throughput rate is less than the set threshold, the following further includes: Notify the server to terminate speech synthesis.
10. A speech synthesis broadcasting device, characterized in that: include: The instruction response unit is used to start the local synthesis process of the target text when receiving the speech synthesis broadcast instruction, save the synthesized local audio data, and send the target text to the server to request the server to perform speech synthesis on the target text; The audio data receiving and throwing unit is used to continuously receive the synthesized audio data returned by the server, and throw the audio data returned by the server for playback according to the set throwing rules; A throughput calculation unit, used for calculating the throughput in real time, wherein the throughput is the ratio of the duration of the audio that has been thrown out to the duration required to generate this audio; The replay unit is used to use the local audio data to replay if it is determined that the currently calculated throughput rate is less than a set threshold.
11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the speech synthesis and broadcasting method as described in any one of claims 1 to 9.
12. A speech synthesis broadcasting system, characterized in that: include: Terminals and servers; The terminal is used to start a local synthesis process for the target text when receiving the speech synthesis broadcast instruction, save the synthesized local audio data, and send the target text to the server to request the server to perform speech synthesis on the target text; Continuously receive the synthesized audio data returned by the server, and play the audio data returned by the server according to the set throwing rules; Calculate the throughput rate in real time, where the throughput rate is the ratio of the duration of the audio that has been thrown out to the duration required to generate the audio; If it is determined that the currently calculated throughput rate is less than the set threshold, the local audio data is used to continue playing; The server is used to perform speech synthesis on the target text and return the latest synthesized audio data to the terminal according to a set strategy.
13. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the speech synthesis and broadcasting method as described in any one of claims 1 to 9 is implemented.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the speech synthesis and broadcasting method as described in any one of claims 1 to 9 is implemented.