Speech synthesis method, apparatus, device, storage medium, and program product

Abstract

Embodiments of the present disclosure relate to a speech synthesis method, device, equipment, storage medium and program product. The method comprises: obtaining acoustic features according to speech text; inputting the acoustic features into a vocoder for speech conversion processing to obtain a speech signal corresponding to the speech text; wherein the speech conversion processing comprises up-sampling processing, the up-sampling processing comprises a plurality of operation processes, and a first precision of a data format adopted by a target operation process in the plurality of operation processes is less than a second precision of a data format adopted by other operation processes. The method can save the time of the operation processes involved in the process of converting acoustic features into a speech signal, and improve the operation efficiency of the entire speech synthesis system.

Description

Technical Field

[0001] This disclosure relates to the field of speech synthesis technology, and in particular to a speech synthesis method, apparatus, device, storage medium, and program product. Background Technology

[0002] Text-to-Speech (TTS) technology refers to the technology that can convert any text into speech. Currently, TTS technology is widely used in various scenarios such as voice assistants and navigation announcements. In these scenarios, TTS technology is generally used to construct a TTS system, which is then run on a specific device to achieve voice output in various scenarios such as voice assistants and navigation announcements.

[0003] A speech synthesis system generally consists of three modules: a text analysis module, an acoustic model, and a vocoder. The text analysis module converts externally transmitted text into linguistic features and inputs them into the acoustic model. The acoustic model converts the linguistic features into acoustic features (such as Mel spectrum) and inputs them into the vocoder. The vocoder finally converts the acoustic features into a speech signal and outputs it.

[0004] However, the vocoder in current speech synthesis systems involves a time-consuming computation process in converting acoustic features into speech signals, resulting in low computational efficiency for the entire speech synthesis system. Summary of the Invention

[0005] This disclosure provides a speech synthesis method, apparatus, device, storage medium, and program product, which can save the time involved in the computation process of converting acoustic features into speech signals and improve the computational efficiency of the entire speech synthesis system.

[0006] In a first aspect, embodiments of this disclosure provide a speech synthesis method applied to a terminal, the method comprising:

[0007] Acoustic features are obtained from speech text;

[0008] The acoustic features are input into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text.

[0009] The speech conversion process includes upsampling, which involves multiple computational processes. In these processes, the first precision of the data format used by the target computational process is less than the second precision of the data format used by other computational processes.

[0010] Secondly, embodiments of this disclosure provide a speech synthesis method applied to a server, the method comprising:

[0011] The voice text sent by the receiving terminal;

[0012] Acoustic features are obtained from speech text;

[0013] Acoustic features are input into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text; wherein, the speech conversion processing includes upsampling processing, which includes multiple calculation processes, and the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes;

[0014] The voice signal is sent to the terminal.

[0015] Thirdly, embodiments of this disclosure provide a speech synthesis device applied to a terminal, the device comprising:

[0016] The first determining module is used to obtain acoustic features based on the speech text;

[0017] The first conversion module is used to input acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text; wherein, the speech conversion processing includes upsampling processing, which includes multiple calculation processes, and the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes.

[0018] Fourthly, embodiments of this disclosure provide a speech synthesis apparatus applied to a server, the apparatus comprising:

[0019] The receiving module is used to receive voice and text messages sent by the terminal.

[0020] The second determining module is used to obtain acoustic features based on the speech text;

[0021] The second conversion module is used to input acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text; wherein, the speech conversion processing includes upsampling processing, which includes multiple calculation processes, and the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes;

[0022] The transmitting module is used to send voice signals to the terminal.

[0023] Fifthly, embodiments of this disclosure provide a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0024] In a sixth aspect, embodiments of this disclosure provide a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the second aspect described above.

[0025] In a seventh aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods of the first and second aspects described above.

[0026] Eighthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the methods of the first and second aspects described above.

[0027] The speech synthesis method, apparatus, device, storage medium, and program product provided in this disclosure obtain acoustic features from speech text and input these features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text. The speech conversion processing includes upsampling processing, which comprises multiple computational processes. The target computational process uses a data format with a first precision lower than the data format used by other computational processes with a second precision. In this method, because multiple data formats with different precisions can be used to execute the multiple computational processes involved in the upsampling processing during speech conversion, compared to simply using a high-precision data format to execute all computational processes involved in the entire upsampling processing, the method reduces the complexity of the upsampling process and saves time during the upsampling process, thereby shortening the overall speech conversion processing time and improving the computational efficiency of the entire speech synthesis system, while ensuring the accuracy of the final result. Attached Figure Description

[0028] Figure 1 This is a diagram illustrating the application environment of a speech synthesis method in one embodiment;

[0029] Figure 2 This is a flowchart illustrating a speech synthesis method in one embodiment;

[0030] Figure 3 This is a flowchart illustrating the speech synthesis method in another embodiment;

[0031] Figure 4 This is an example diagram illustrating the use of single-precision floating-point numbers to process acoustic features in another embodiment.

[0032] Figure 5 This is an example diagram illustrating the use of mixed-precision floating-point numbers to process acoustic features in another embodiment;

[0033] Figure 6This is a flowchart illustrating the speech synthesis method in another embodiment;

[0034] Figure 7 This is a flowchart illustrating the speech synthesis method in another embodiment;

[0035] Figure 8 Example diagram of actual test results for multiple strategies in another embodiment;

[0036] Figure 9 This is a structural block diagram of a speech synthesis device in one embodiment;

[0037] Figure 10 This is a structural block diagram of a speech synthesis device in another embodiment;

[0038] Figure 11 This is a diagram of the internal structure of a terminal in one embodiment;

[0039] Figure 12 This is a diagram of the internal structure of a server in one embodiment. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this disclosure and are not intended to limit the embodiments of this disclosure.

[0041] The technical solutions involved in the embodiments of this disclosure will be described below in conjunction with the scenarios in which they are applied.

[0042] The speech synthesis method provided in this disclosure can be applied to, for example... Figure 1 In the application environment shown, terminal 102 can communicate with server 104 via a network. Terminal 102 can send the acquired voice data to server 104 for processing, and server 104 will then send the voice signal back to terminal 102 after processing. Alternatively, terminal 102 can directly process the acquired voice data to obtain the voice signal. Terminal 102 can be a processing terminal installed in a vehicle (e.g., an in-vehicle terminal), or it can be a personal computer, laptop, smartphone, tablet, portable wearable device, or IoT device. The vehicle can be a shared vehicle or other private vehicle; the portable wearable device can be a smart bracelet, smartwatch, etc.; the IoT device can be a smart refrigerator, smart TV, smart speaker, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0043] The following embodiments will describe the solutions of the present disclosure embodiments using the terminal 102 and the server 104 as the execution entities, respectively.

[0044] In one embodiment, such as Figure 2 As shown, a speech synthesis method is provided, which can be applied to... Figure 1 Taking a terminal as an example, the method may include the following steps:

[0045] S202, Obtain acoustic features from speech text.

[0046] The voice text can be transmitted to the terminal from an upstream application, such as a map component. This voice text can include the tone, timbre, and specific content of the voice. It can be used for navigation announcements, or for voice assistants, etc.

[0047] Specifically, after obtaining the speech text from the upstream application, the terminal can perform text analysis on the speech text to obtain the corresponding speech features. These speech features can then be input into an acoustic model for conversion to obtain the acoustic features corresponding to the language features. Here, a text analysis model can be used for the speech text analysis. This text analysis module can be a network model composed of neural networks, which can perform text analysis on the speech text quickly and accurately. Alternatively, the acoustic model can also be a model composed of neural networks, such as the Fast-speech acoustic model. Through this acoustic model composed of neural networks, acoustic features can be obtained quickly and accurately.

[0048] S204, input the acoustic features into the vocoder for speech conversion processing, and obtain the speech signal corresponding to the speech text.

[0049] In this step, acoustic features are mostly represented using Mel spectra, which are generally used to represent the spectrum of acoustic signals and can indicate the relationship between the frequency and amplitude or phase of sound. After obtaining the acoustic features through the acoustic model, a vocoder can be used to perform speech conversion processing on the acoustic features; the vocoder can be a vocoder composed of neural networks, such as a HifiGAN vocoder.

[0050] When a vocoder is used to perform speech conversion processing on acoustic features, the speech conversion processing includes upsampling processing, which includes multiple computational processes. In these multiple computational processes, the first precision of the data format used by the target computational process is less than the second precision of the data format used by other computational processes.

[0051] In other words, the process of processing acoustic features using a vocoder typically involves multiple computational steps. Current technologies generally employ the same high-precision data format for each step to obtain high-precision results. This high-precision data format is, for example, the second-precision data format mentioned above. However, the high-precision results obtained using existing technologies are meaningless and, in some scenarios, may even negatively impact the vocoder's computational efficiency.

[0052] Based on this, this embodiment employs a hybrid approach of high-precision and low-precision data formats to perform data operations on multiple computational processes, obtaining computational results that meet the accuracy requirements while also improving the computational efficiency of the vocoder. The computational result can be the speech signal corresponding to the speech text. This speech signal can be a continuous speech waveform (e.g., a waveform composed of time and amplitude) or discrete multiple sampling points (where each sampling point can represent the correspondence between time and amplitude). Furthermore, the computational processes using the first-precision data format are denoted as target computational processes, which can be one or more computational processes. Conversely, the computational processes using the second-precision data format are denoted as other computational processes, which can also be one or more computational processes. These other computational processes and the target computational processes together constitute the multiple computational processes in the vocoder's upsampling processing.

[0053] Specifically, here the first precision is less than the second precision; that is, the first precision can be considered low precision, and the second precision can be considered high precision. By mixing high and low precision data formats in multiple calculation processes, compared to the existing technology that uses only high precision data formats in multiple calculation processes, the computational load of the vocoder is relatively reduced, and the calculation time is correspondingly shortened. This reduces the time spent in the vocoder's calculation process, improves the vocoder's computational efficiency, and consequently improves the computational efficiency of the entire speech synthesis system, including the vocoder.

[0054] It should be noted that in this embodiment, the voice signal synthesis process is performed on the terminal side only. This is mainly for scenarios with high requirements for voice signal latency, such as navigation broadcasts. This can ensure the latency requirements of the voice signal while also meeting the actual needs of the application scenarios.

[0055] In the aforementioned speech synthesis method, the terminal obtains acoustic features based on the speech text and inputs these features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text. The speech conversion processing includes upsampling, which comprises multiple computational steps. The target computational step uses a data format with a first precision lower than the data format used in other computational steps with a second precision. In this method, because multiple data formats with different precisions can be used to execute the multiple computational steps involved in the upsampling process during speech conversion, compared to simply using a high-precision data format, this method reduces the complexity of the upsampling process and saves time while ensuring the accuracy of the final result. This shortens the overall speech conversion process time and improves the computational efficiency of the entire speech synthesis system.

[0056] In another embodiment, the data format of the first precision is half-precision floating-point number, and the data format of the second precision is single-precision floating-point number.

[0057] Single-precision floating-point numbers are used to represent real numbers with a decimal part. They occupy 4 bytes (32 bits) of storage space, have a value range of -3.4E38 to 3.4E38, and have very high precision, typically reaching 2^32. -149 .

[0058] In the upsampling processing of existing vocoders, many calculations are performed using single-precision floating-point numbers, resulting in redundant calculations for the vocoder. This is because: 1. HifiGAN vocoders have relatively few intermediate calculation results, and their values ​​are concentrated within a narrow range; 2. The final speech signal obtained by the vocoder is assumed to be composed of sample points, with each sample point having a data length of 16 bits. Excessive data precision is meaningless and only increases the computational load and time.

[0059] Therefore, this embodiment incorporates half-precision floating-point numbers into multiple operations during the vocoder's upsampling process. Half-precision floating-point numbers refer to binary floating-point data types, occupying 2 bytes (16 bits) of storage space. The data size of a half-precision floating-point number is half that of a single-precision floating-point number. This reduces the computational load during vocoder operations, shortens the vocoder's processing time, and improves its efficiency.

[0060] In this embodiment, the data format for the first precision is half-precision floating-point number and the data format for the second precision is single-precision floating-point number. In this way, the vocoder uses a mixed method of half-precision floating-point number and single-precision floating-point number for data operation, which can reduce the amount of calculation in the vocoder operation process, shorten the operation time of the vocoder, and improve the operation efficiency of the vocoder.

[0061] In another embodiment, the above upsampling process uses Single Instruction Multiple Data (SIMD) for data computation.

[0062] SIMD stands for Single Instruction Multiple Data. An SMID instruction set is a set of instructions that can process multiple data streams with a single instruction. Using SIMD instruction sets for simple mathematical operations is generally significantly more efficient than using ordinary CPU instructions.

[0063] Furthermore, speech synthesis systems are generally loaded and executed by hardware chips. In this embodiment, the hardware chip primarily uses an Arm 8.2 or later chip architecture. The SIMD instruction set typically corresponds to the neon instruction set on Arm chips.

[0064] On Arm 8.2 chips, when using 128-bit vector registers and employing the NEON instruction set to accelerate the computation process, the data processed typically needs to be an integer multiple of 16 bits (i.e., half-precision floating-point numbers). For single-precision floating-point numbers, one NEON instruction can process a maximum of four single-precision floating-point numbers; for half-precision floating-point numbers, one NEON instruction can process a maximum of eight single-precision floating-point numbers. Therefore, using half-precision floating-point numbers for data operations can double the computation speed / efficiency.

[0065] In this embodiment, by using Single Instruction Multiple Data (SIMD) for data processing in the upsampling process, each instruction in the instruction set processes more data per operation, thereby further improving the data processing speed and thus further improving the vocoder's processing efficiency.

[0066] In another embodiment, another speech synthesis method is provided, based on the above embodiments, such as... Figure 3 As shown, the method may include the following steps:

[0067] S302, input the acoustic features into the vocoder, perform pre-convolution processing on the acoustic features according to the second precision data format, and determine the pre-convolution result.

[0068] In this step, see Figure 4The diagram illustrates a specific process for processing acoustic features in a vocoder. The vocoder includes three main computational processes: presampling (e.g., pre conv1d in the diagram), upsampling (e.g., upsampling in the diagram), and postsampling (e.g., last conv1d in the diagram). See also... Figure 4 As shown, in the existing technology, all three major calculation processes use 32-bit single-precision floating-point numbers for calculation, which makes the entire calculation process very time-consuming and the calculation efficiency low.

[0069] Therefore, this embodiment uses mixed-precision floating-point numbers to process acoustic features, see [link to documentation]. Figure 5 The diagram shown is an example of the specific process of processing acoustic features in the vocoder of this embodiment, specifically an example of processing acoustic features using mixed-precision floating-point numbers. The architecture of the vocoder for processing acoustic features remains unchanged in this embodiment; only the data format used in each computational process has been altered.

[0070] In this embodiment, after obtaining acoustic features from the acoustic model, the computational cost of the acoustic features in the pre-convolution part is relatively small, and the data computation dimension is not an integer multiple of 8 (the 8 comes from the fact that the neon instruction can retrieve eight 16-bit half-precision floating-point numbers from a 128-bit register for each calculation). If the neon instruction were used for half-precision floating-point calculations, it would inevitably introduce extra padding data, leading to a decrease in computation speed. In other words, using half-precision floating-point numbers in the pre-convolution process neither reduces the computational cost nor increases the computation speed, and it also increases the data handling overhead caused by the neon instruction. Therefore, in this embodiment, single-precision floating-point numbers (i.e., according to the second-precision data format) are still used for pre-convolution processing of the acoustic features to obtain the pre-convolution result. This pre-convolution processing can be a 1D convolution of the acoustic features, or it can be a convolution processing of other dimensions.

[0071] S304. According to the data format of the second precision and the data format of the first precision, the preconvolution result is upsampled cyclically for a first preset number of times to determine the upsampled result.

[0072] In this step, the upsampling process is the most computationally intensive part of the vocoder. This process involves multiple iterative upsampling operations, meaning multiple upsampling operations need to be performed. Each upsampling operation includes multiple computational processes such as activation function processing, deconvolution operation processing, and residual operation processing. Optionally, each upsampling operation can be performed as follows: The upsampling operation includes the following steps A1-A3:

[0073] A1. Apply activation function operation to the preconvolution result using the second-precision data format to determine the first activation result; A2. Apply deconvolution operation to the first activation result using the second-precision data format to determine the deconvolution result; A3. Apply residual operation to the deconvolution result using the first-precision data format to determine the initial upsampling result.

[0074] Here, the activation function can be a leaky ReLU (linear unit with leakage correction) activation function. Since the computational cost and time consumption of the activation function operation in this step are relatively low, single-precision floating-point numbers (i.e., second-precision data format) are used for the operation to obtain the first activation result. The data format corresponding to the first activation result obtained here is the second-precision data format.

[0075] Since the deconvolution and residual operations are the most time-consuming parts of the upsampling process, the data format used in these two operations can be changed. Common methods for deconvolution include zero-padding, interleaved addition, and small convolution kernels, with zero-padding being the most frequently used. This embodiment also preferably uses zero-padding, and when performing deconvolution on the first activation result, single-precision floating-point numbers (i.e., second-precision data format) are still used to obtain the deconvolution result. This balances computational efficiency with other aspects (such as memory overhead). The data format corresponding to the deconvolution result obtained here is the second-precision data format.

[0076] For residual operation processing (e.g.) Figure 4 and 5 In this embodiment, when processing the deconvolution result (using the residual stack), unlike the single-precision floating-point numbers (i.e., the second-precision data format) used in the activation function and deconvolution processes described above, half-precision floating-point numbers (i.e., the first-precision data format) are used to perform the calculations, obtaining the upsampling result. Since this upsampling result is not the final upsampling result, it is denoted as the initial upsampling result. The data format corresponding to the upsampling result obtained here is the first-precision data format.

[0077] Then, the initial upsampling result can be used as the new pre-convolution result, and the above upsampling processing operation can be performed again until the first preset number of times is reached to obtain the upsampling result.

[0078] In other words, after obtaining the initial upsampling result in the first loop of the upsampling process, this initial upsampling result can be used as a new pre-convolution result. The activation function is then applied to this initial upsampling result using a second-precision data format to obtain a new first activation result. A deconvolution operation is then performed on this new first activation result using a second-precision data format to obtain a new deconvolution result. Finally, a residual operation is performed on this new deconvolution result using a first-precision data format to obtain a new initial upsampling result. Afterward, it is determined whether the number of loops has reached a first preset number. If the number of loops has reached the first preset number, the loop stops, and the initial upsampling result obtained in this loop is used as the final upsampling result. If the number of loops has not reached the first preset number, the loop iteration continues until the number of loops reaches the first preset number, ultimately obtaining the final upsampling result.

[0079] The first preset number of times mentioned here mainly refers to the number of loops in the upsampling process. In each upsampling process, the sampling multiple of the upsampling is usually involved. Optionally, the sampling multiple of the upsampling is different in each upsampling process in the first preset number of upsampling processes.

[0080] For example, the first preset number of iterations can be 3, with the sampling multipliers corresponding to each upsampling process being 8, 4, and 2 respectively. By setting the number of upsampling loops to 3 and the sampling multipliers for each iteration to be 8, 4, and 2 respectively, the complexity of the vocoder training process can be simplified while ensuring the accuracy of the vocoder model.

[0081] S306 uses a second-precision data format to perform post-convolution processing on the upsampled results to determine the speech signal.

[0082] In this step, similar to the pre-convolution in S302 above, the computational load of the post-convolution part is relatively small, and the data computation dimension is not an integer multiple of 8. If the neon instruction is used for half-precision floating-point calculation, it will inevitably introduce extra padding data, leading to a decrease in computation speed. That is, using half-precision floating-point numbers in the post-convolution process will not shorten the computation load or increase the computation speed, and will also increase the data handling overhead caused by the neon instruction. Therefore, in this embodiment, when performing post-convolution processing on the upsampled result, single-precision floating-point numbers (i.e., processing according to the second-precision data format) are still used to obtain the speech signal. The post-convolution processing here can be a 1D convolution processing on the upsampled result, or it can be a convolution processing of other dimensions.

[0083] In this embodiment, the acoustic features are pre-convolved in the vocoder using a second-precision data format, then upsampled using a mixed second-precision and first-precision data format, and finally post-convolved using the second-precision data format to obtain the speech signal. The upsampling process is the most time-consuming; using a mixed high- and low-precision data format reduces the computational load and increases the processing speed, thereby improving the vocoder's efficiency. Furthermore, each iteration of the upsampling process includes activation function operations, deconvolution operations, and residual operations. The most time-consuming residual operation is performed using a low-precision data format, minimizing the computational load and effectively increasing the processing speed, thus improving the vocoder's efficiency. Moreover, by using different sampling factors in each iteration for the most time-consuming upsampling process, the complexity of the vocoder training process is simplified, while maintaining the model accuracy.

[0084] In another embodiment, before step A3 is executed, since the data format corresponding to the deconvolution result obtained in step A2 is a second-precision data format, while the residual operation in step A3 uses a first-precision data format, the data format of the deconvolution result can be converted from the second-precision data format to the first-precision data format. That is, before performing the residual operation, the data format of the deconvolution result can be converted from single-precision floating-point numbers to half-precision floating-point numbers to facilitate fast and accurate residual operation.

[0085] Accordingly, since the initial upsampling result obtained in step A3 corresponds to a first-precision data format, while the activation function operation in the loop return step A1 uses a second-precision data format, the data format of the initial upsampling result can be converted from the first-precision data format to the second-precision data format. In other words, before performing the activation function operation, the data format of the initial upsampling result can be converted from half-precision floating-point numbers to single-precision floating-point numbers to facilitate fast and accurate activation function operation later.

[0086] In another embodiment, another speech synthesis method is provided, based on the above embodiments, such as... Figure 6 As shown, step A3 above may include the following steps:

[0087] S402, perform residual operation, which includes: performing activation function operation on the deconvolution result using a first-precision data format to determine a second activation result; performing symmetric filling on the second activation result using a first-precision data format to determine a filling result; and performing one-dimensional convolution on the filling result using a first-precision data format to determine an initial convolution result.

[0088] In this step, the residual calculation process typically involves multiple iterative residual calculation operations, meaning that multiple residual calculation operations need to be performed. Each residual calculation operation includes an activation function operation (e.g., Figure 4 and 5 In the example of Leaky Relu), symmetrical filling processing (e.g.) Figure 4 and 5 (reflection pad in the middle), one-dimensional convolution processing (e.g.) Figure 4 and 5 The process involves multiple operations such as conv1d, symmetric padding, and one-dimensional convolution.

[0089] When using data formats of different precisions in multiple computational processes, data format conversion is inevitably required, and the time consumed by multiple data format conversions is generally not negligible. In the upsampling process of the HifiGAN vocoder, the array storing intermediate variables has a maximum dimension exceeding 200,000. Frequent data format conversions would negate the speedup provided by using half-precision floating-point operations. Therefore, for multiple computational processes within each residual calculation, the same first-precision data format (i.e., half-precision floating-point) is used for data operations. This eliminates the need for data format conversion during each residual calculation, significantly improving the computational speed of residual and upsampling processes. Furthermore, by using half-precision floating-point operations without data format conversion during each residual calculation, the number / weights of arrays stored in the entire vocoder model can be reduced to half that of existing technologies, thus saving memory space.

[0090] Specifically, in each residual operation, half-precision floating-point numbers can be used to sequentially process the deconvolution result with activation function operations, symmetric padding, one-dimensional convolution, symmetric padding, and another one-dimensional convolution, ultimately obtaining the initial convolution result. The data format corresponding to the initial convolution result obtained here is the first-precision data format.

[0091] S404: Take the initial convolution result as the new deconvolution result and return to perform residual operation until the second preset number of times is reached to obtain the initial upsampling result.

[0092] In this step, after obtaining the initial convolution result in the first iteration of the residual operation, this initial convolution result can be used as the new deconvolution result. The new deconvolution result is then subjected to activation function operations, symmetric padding, one-dimensional convolution, and symmetric padding again, using the first-precision data format, to obtain a new initial convolution result. Next, it is determined whether the number of iterations has reached a second preset number. If the number of iterations has reached the second preset number, the loop stops, and the initial convolution result obtained in this loop is used as the initial upsampling result. If the number of iterations has not reached the second preset number, the loop iteration continues until the second preset number of iterations is reached, finally obtaining the initial upsampling result.

[0093] The second preset number of iterations refers to the number of loops in the residual calculation process during each upsampling operation. The first preset number of iterations refers to the number of loops in the upsampling process. Optionally, the second preset number of iterations can be the same as the first, for example, it could be 3. Setting the number of upsampling loops to be equal to the number of residual calculation loops in each upsampling operation reduces the computational load of the vocoder model while maintaining its accuracy, thus improving its computational efficiency.

[0094] In this embodiment, by using the same first-precision data format to perform activation function operations, symmetric padding, and one-dimensional convolution during residual operations, data format conversion is eliminated during each residual operation, significantly improving the speed of residual and upsampling operations. Furthermore, obtaining the initial upsampling result through multiple iterations of residual operations enhances the accuracy of the initial upsampling result. Moreover, by setting the number of upsampling iterations to be equal to the number of residual operation iterations in each upsampling iteration, the computational load of the vocoder model can be reduced while maintaining its accuracy, thus improving its computational efficiency.

[0095] In another embodiment, such as Figure 7 As shown, another speech synthesis method is provided, which can be applied to... Figure 1 Taking a server as an example, this method may include the following steps:

[0096] S502, receiving voice and text messages sent by the receiving terminal.

[0097] In this step, the terminal can receive voice text sent by the upstream application and then send the voice text to the server for processing. For a detailed explanation of the voice text, please refer to the explanation in S202 above; it will not be repeated here.

[0098] S504, Acquire acoustic features based on speech text.

[0099] In this step, after obtaining the speech text, the server can perform text analysis on the speech text to obtain the corresponding speech features. These speech features can then be input into an acoustic model for conversion to obtain the acoustic features corresponding to the language features. For details, please refer to the explanation in S202 above; it will not be repeated here.

[0100] S506 inputs acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text.

[0101] The speech conversion process includes upsampling, which involves multiple computational processes. In these processes, the first precision of the data format used by the target computational process is less than the second precision of the data format used by other computational processes.

[0102] In this step, after obtaining the acoustic features, the server can perform speech conversion processing to obtain a speech signal. The specific processing procedure can be found in the explanation of S204 above, and will not be repeated here.

[0103] S508 sends voice signals to the terminal.

[0104] In this step, after the server obtains the voice signal, it can send the voice signal back to the terminal so that the terminal can output voice based on the obtained voice signal to assist the user in operation.

[0105] It should be noted that in this embodiment, the server side performs the voice signal synthesis process on one side, which is mainly for scenarios where the voice signal latency requirement is not very high, such as voice assistants. This can reduce the processing power consumption of the terminal while ensuring the actual needs of the application voice signal scenario.

[0106] In this embodiment, the server receives voice text sent by the terminal, obtains acoustic features from the voice text, inputs the acoustic features into a vocoder for speech conversion processing, obtains the corresponding voice signal, and sends the voice signal to the terminal. The speech conversion processing includes upsampling, which comprises multiple computational processes. The target computational process uses a data format with a first precision lower than the data format used by other computational processes with a second precision. In this method, because multiple data formats with different precisions can be used to execute the multiple computational processes involved in the upsampling process during speech conversion, compared to simply using a high-precision data format, this method reduces the complexity of the upsampling process and saves time while ensuring the accuracy of the final result. This shortens the overall speech conversion process time and improves the computational efficiency of the entire speech synthesis system. Furthermore, using a server for speech signal synthesis reduces the terminal's processing power consumption while meeting the actual needs of the application scenario.

[0107] To facilitate a better understanding of the embodiments of this disclosure, a specific embodiment is given below using the terminal side as an example to illustrate the technical solution. Based on the above embodiment, the above method may include the following steps:

[0108] S1, Obtain acoustic features from speech text.

[0109] S2, input the acoustic features into the vocoder, perform pre-convolution processing on the acoustic features according to single-precision floating-point numbers, and determine the pre-convolution result.

[0110] S3 uses single-precision floating-point numbers to perform activation function operations on the pre-convolution results to determine the first activation result.

[0111] S4 uses single-precision floating-point numbers to perform deconvolution operation on the first activation result to determine the deconvolution result.

[0112] S5 uses half-precision floating-point numbers to perform activation function operations on the deconvolution result to determine the second activation result.

[0113] S6. Use half-precision floating-point numbers to perform symmetrical filling on the second activation result to determine the filling result.

[0114] S7 uses half-precision floating-point numbers to perform one-dimensional convolution on the filling result to determine the initial convolution result.

[0115] S8 takes the initial convolution result as the new deconvolution result and returns to execute S5-S7 until the preset number of residual loops is reached, and takes the initial convolution result that has reached the preset number of residual loops as the initial upsampling result.

[0116] S9 takes the initial upsampling result as the new preconvolution result and returns to execute S3-S8 until the preset number of upsampling loops is reached, and takes the initial upsampling result that has reached the preset number of upsampling loops as the final upsampling result.

[0117] S10 uses single-precision floating-point numbers to perform post-convolution processing on the upsampling results to determine the speech signal.

[0118] For example, see Figure 8 As shown, assuming the Qualcomm 8155P chip (a chip with Arm version 8.2 or higher) is used, the same neural network parameters are used in the vocoder. For different calculation strategies, the time taken for speech synthesis every 40 frames is calculated, and the sound quality under each calculation strategy is given.

[0119] exist Figure 8 The calculation strategies include: 1. Using ordinary CPU instructions for calculation; 2. Using only the neon instructions corresponding to single-precision floating-point numbers for calculation; 3. Using only half-precision floating-point numbers for calculation, with floating-point length conversion performed before and after each neon instruction; 4. Performing floating-point length conversion only before and after each residual operation, with half-precision floating-point numbers used throughout the residual operation process, and CPU read / write optimizations to reduce memory read / write operations. The time consumed per 40 frames of speech synthesis for each calculation strategy is 3000ms, 230ms, 210ms, and 141ms, respectively. The sound quality for each calculation strategy is rated as "good".

[0120] As shown in the above calculations, without affecting sound quality, the fourth calculation strategy, which uses a mixture of half-precision and single-precision floating-point calculations, reduces the computation time by 230-141=89ms compared to the second strategy, which uses only single-precision floating-point calculations, resulting in a speed improvement of 89 / 230=38.7%. Therefore, the mixed half-precision and single-precision floating-point calculation strategy used in this embodiment can significantly improve the computational efficiency of the speech synthesis system while ensuring sound quality.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0122] In one embodiment, such as Figure 9 As shown, a speech synthesis device is provided, including: a first determining module 11 and a first conversion module 12, wherein:

[0123] The first determining module 11 is used to obtain acoustic features based on the speech text;

[0124] The first conversion module 12 is used to input acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text; wherein, the speech conversion processing includes upsampling processing, which includes multiple calculation processes, and the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes.

[0125] In another embodiment, the data format of the first precision is half-precision floating-point number, and the data format of the second precision is single-precision floating-point number.

[0126] In another embodiment, the above upsampling process uses Single Instruction Multiple Data Stream (SIMD) for data processing.

[0127] In another embodiment, another speech synthesis device is provided. Based on the above embodiments, the first conversion module 12 may include:

[0128] A preconvolution unit is used to input acoustic features into the vocoder, perform preconvolution processing on the acoustic features according to a second-precision data format, and determine the preconvolution result; and,

[0129] The upsampling unit is used to perform upsampling processing on the preconvolution result a first preset number of times according to the second precision data format and the first precision data format, to determine the upsampling result; and,

[0130] The post-convolution unit is used to perform post-convolution processing on the upsampled results using a second-precision data format to determine the speech signal.

[0131] Optionally, the above upsampling unit may include:

[0132] The upsampling loop execution subunit is used to perform upsampling processing operations, which include: performing activation function operations on the pre-convolution result using a second-precision data format to determine a first activation result; performing deconvolution operations on the first activation result using a second-precision data format to determine a deconvolution result; and performing residual operations on the deconvolution result using a first-precision data format to determine an initial upsampling result.

[0133] The upsampling return execution subunit is used to take the initial upsampling result as the new pre-convolution result and return to perform the upsampling processing operation until the first preset number of times is reached to obtain the upsampling result.

[0134] Optionally, the sampling multiple of each upsampling operation is different during the first preset number of upsampling operations.

[0135] Optionally, the above-mentioned upsampling loop execution subunit performs residual calculation on the deconvolution result using a first-precision data format. Before determining the initial upsampling result, it is also used to convert the data format of the deconvolution result from a second-precision data format to a first-precision data format.

[0136] Optionally, the above-mentioned upsampling loop execution subunit performs residual operation on the deconvolution result using a first-precision data format. After determining the initial upsampling result, it is also used to convert the data format of the initial upsampling result from the first-precision data format to the second-precision data format.

[0137] In another embodiment, another speech synthesis device is provided. Based on the above embodiments, the upsampling loop execution subunit is specifically used to perform residual operation. The residual operation includes: performing activation function operation on the deconvolution result using a first-precision data format to determine a second activation result; performing symmetrical padding on the second activation result using a first-precision data format to determine a padding result; and performing one-dimensional convolution on the padding result using a first-precision data format to determine an initial convolution result.

[0138] The initial convolution result is used as the new deconvolution result, and the residual operation is performed until the second preset number of times is reached to obtain the initial upsampling result.

[0139] Optionally, the second preset number of times is the same as the first preset number of times.

[0140] In another embodiment, such as Figure 10As shown, another speech synthesis device is provided, including: a receiving module 21, a second determining module 22, a second conversion module 23, and a transmitting module 24, wherein:

[0141] Receiver module 21 is used to receive voice and text sent by the terminal;

[0142] The second determining module 22 is used to obtain acoustic features based on the speech text;

[0143] The second conversion module 23 is used to input acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text; wherein, the speech conversion processing includes upsampling processing, the upsampling processing includes multiple calculation processes, and the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes;

[0144] The transmitting module 24 is used to transmit voice signals to the terminal.

[0145] Specific limitations regarding the speech synthesis device can be found in the limitations of the speech synthesis method described above, and will not be repeated here. Each module in the aforementioned speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the terminal or server in hardware form, or stored in the memory of the terminal or server in software form, so that the processor can call and execute the corresponding operations of each module.

[0146] In one embodiment, a terminal is provided, the internal structure of which can be shown as follows: Figure 11 As shown, the terminal includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a speech synthesis method. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the terminal casing, or an external keyboard, touchpad, or mouse.

[0147] Those skilled in the art will understand that Figure 11The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal to which the present application is applied. A specific terminal may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0148] In one embodiment, a terminal is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described terminal-side method embodiments.

[0149] In one embodiment, a server is provided, the internal structure of which can be shown as follows: Figure 12 As shown, the server includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The server's database stores speech data involved in the speech synthesis process. The server's network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech synthesis method.

[0150] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the server to which the solution of this application is applied. A specific server may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] In one embodiment, a server is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described server-side method embodiments.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described terminal-side and server-side method embodiments.

[0153] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described terminal-side and server-side method embodiments.

[0154] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, and displayed data) are all information and data authorized by the user or fully authorized by all parties.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above-described embodiments are merely illustrative of several implementation methods of the present disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present disclosure embodiments, and these all fall within the protection scope of the present disclosure embodiments. Therefore, the protection scope of the patent for the embodiments of the present disclosure should be determined by the appended claims.

Claims

1. A speech synthesis method, characterized in that, The method includes: Acoustic features are obtained from speech text; The acoustic features are input into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text. The speech conversion process includes upsampling, which includes multiple calculation processes. In these multiple calculation processes, the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes. The step of inputting the acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text includes: The acoustic features are input into a vocoder, and pre-convolution processing is performed on the acoustic features according to the data format of the second precision to determine the pre-convolution result; and... According to the data format of the second precision and the data format of the first precision, the preconvolution result is upsampled cyclically a first preset number of times to determine the upsampling result; and, The upsampling result is post-convolutionally processed using the second precision data format to determine the speech signal; The step of performing upsampling processing on the preconvolution result a first preset number of times according to the data format of the second precision and the data format of the first precision, and determining the upsampling result, includes: Perform an upsampling operation, the upsampling operation including: performing activation function operation on the preconvolution result using the second precision data format to determine a first activation result; performing deconvolution operation on the first activation result using the second precision data format to determine a deconvolution result; and performing residual operation on the deconvolution result using the first precision data format to determine an initial upsampling result. The initial upsampling result is used as the new pre-convolution result, and the upsampling processing operation is performed again until the first preset number of times is reached to obtain the upsampling result.

2. The method according to claim 1, characterized in that, The data format for the first precision is half-precision floating-point number, and the data format for the second precision is single-precision floating-point number.

3. The method according to claim 1 or 2, characterized in that, The upsampling process uses Single Instruction Multiple Data Stream (SIMD) for data computation.

4. The method according to claim 1, characterized in that, The sampling multiple is different for each upsampling process during the first preset number of upsampling operations.

5. The method according to claim 1, characterized in that, Before performing residual calculations on the deconvolution result using the data format of the first precision to determine the initial upsampling result, the method further includes: The data format of the deconvolution result is converted from the second precision data format to the first precision data format.

6. The method according to claim 1, characterized in that, After performing residual calculations on the deconvolution result using the data format of the second precision to determine the initial upsampling result, the method further includes: The data format of the initial upsampling result is converted from the first precision data format to the second precision data format.

7. The method according to claim 1, characterized in that, The step of performing residual calculations on the deconvolution result using the data format of the first precision to determine the initial upsampling result includes: Perform residual operation, the residual operation including: performing activation function operation on the deconvolution result using the data format of the first precision to determine a second activation result; and performing symmetric padding on the second activation result using the data format of the first precision to determine a padding result; and performing one-dimensional convolution on the padding result using the data format of the first precision to determine an initial convolution result. The initial convolution result is used as the new deconvolution result, and the residual operation is performed again until the second preset number of times is reached to obtain the initial upsampling result.

8. The method according to claim 7, characterized in that, The second preset number of times is the same as the first preset number of times.

9. A speech synthesis method, characterized in that, The method includes: The voice and text sent by the receiving terminal; Acoustic features are obtained from the spoken text; The acoustic features are input into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text; wherein, the speech conversion processing includes upsampling processing, the upsampling processing includes multiple calculation processes, and the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes; The voice signal is sent to the terminal; The step of inputting the acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text includes: The acoustic features are input into a vocoder, and pre-convolution processing is performed on the acoustic features according to the data format of the second precision to determine the pre-convolution result; and... According to the data format of the second precision and the data format of the first precision, the preconvolution result is upsampled cyclically a first preset number of times to determine the upsampling result; and, The upsampling result is post-convolutionally processed using the second precision data format to determine the speech signal; The step of performing upsampling processing on the preconvolution result a first preset number of times according to the data format of the second precision and the data format of the first precision, and determining the upsampling result, includes: Perform an upsampling operation, the upsampling operation including: performing activation function operation on the preconvolution result using the second precision data format to determine a first activation result; performing deconvolution operation on the first activation result using the second precision data format to determine a deconvolution result; and performing residual operation on the deconvolution result using the first precision data format to determine an initial upsampling result. The initial upsampling result is used as the new pre-convolution result, and the upsampling processing operation is performed again until the first preset number of times is reached to obtain the upsampling result.

10. A speech synthesis device, characterized in that, The device includes: The first determining module is used to obtain acoustic features based on the speech text; The first conversion module is used to input the acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text; wherein, the speech conversion processing includes upsampling processing, the upsampling processing includes multiple calculation processes, and the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes; The first conversion module includes: A preconvolution unit is used to input the acoustic features into a vocoder, perform preconvolution processing on the acoustic features according to the second precision data format, and determine the preconvolution result; and, An upsampling unit is configured to perform upsampling processing on the preconvolution result a first preset number of times according to the data format of the second precision and the data format of the first precision, to determine the upsampling result; and, The post-convolution unit is used to perform post-convolution processing on the upsampling result using the data format of the second precision to determine the speech signal; The upsampling unit includes: An upsampling loop execution subunit is used to perform upsampling processing operations, the upsampling processing operations including: performing activation function operations on the preconvolution result using the second precision data format to determine a first activation result; performing deconvolution operations on the first activation result using the second precision data format to determine a deconvolution result; and performing residual operations on the deconvolution result using the first precision data format to determine an initial upsampling result. The upsampling return execution subunit is used to take the initial upsampling result as a new pre-convolution result and return to execute the upsampling processing operation until the first preset number of times is reached to obtain the upsampling result.

11. A speech synthesis device, characterized in that, The device includes: The receiving module is used to receive voice and text messages sent by the terminal. The second determining module is used to obtain acoustic features based on the spoken text; The second conversion module is used to input the acoustic features into a vocoder for speech conversion processing to obtain the speech signal corresponding to the speech text; wherein, the speech conversion processing includes upsampling processing, the upsampling processing includes multiple calculation processes, and the first precision of the data format used by the target calculation process is less than the second precision of the data format used by other calculation processes; A transmitting module is used to transmit the voice signal to the terminal; The second conversion module includes: A preconvolution unit is used to input the acoustic features into a vocoder, perform preconvolution processing on the acoustic features according to the second precision data format, and determine the preconvolution result; and, An upsampling unit is configured to perform upsampling processing on the preconvolution result a first preset number of times according to the data format of the second precision and the data format of the first precision, to determine the upsampling result; and, The post-convolution unit is used to perform post-convolution processing on the upsampling result using the data format of the second precision to determine the speech signal; The upsampling unit includes: An upsampling loop execution subunit is used to perform upsampling processing operations, the upsampling processing operations including: performing activation function operations on the preconvolution result using the second precision data format to determine a first activation result; performing deconvolution operations on the first activation result using the second precision data format to determine a deconvolution result; and performing residual operations on the deconvolution result using the first precision data format to determine an initial upsampling result. The upsampling return execution subunit is used to take the initial upsampling result as a new pre-convolution result and return to execute the upsampling processing operation until the first preset number of times is reached to obtain the upsampling result.

12. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

13. A server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 9.

14. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images