Audio generation method and apparatus, electronic device, and readable storage medium

Abstract

The application provides an audio generation method and device, electronic equipment and storage medium, and belongs to the technical field of audio processing. The audio generation method comprises the following steps: converting phoneme data of a text into a phoneme feature vector sequence; dividing the phoneme feature vector sequence into at least two feature vector blocks, and determining one feature vector block in the at least two feature vector blocks as a target vector block; sequentially performing acoustic feature mapping processing and speech conversion processing on the target vector block, and outputting synthesized audio corresponding to the target vector block; in the case where the target vector block is not a last feature vector block, updating the target vector block to the next feature vector block, returning to performing acoustic feature mapping processing on the target vector block to obtain acoustic feature data of the target vector block until the target vector block is the last feature vector block. The application effectively improves the speed of text conversion speech, reduces speech conversion delay, and reduces resource occupancy rate in the speech conversion process.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio generation method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Text-to-speech (TTS) technology is a technology that converts text information into natural and fluent speech output. It is widely used in various scenarios such as navigation, e-readers, in-vehicle systems, and voice assistants.

[0003] However, although TTS is currently widely used, it suffers from slow text-to-speech conversion speed, high latency, and high resource consumption on devices with lower computing power, such as in-vehicle systems, because it requires strong computing power. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this application provides an audio generation method, apparatus, electronic device, and readable storage medium.

[0005] According to a first aspect of the embodiments of this application, an audio generation method is provided, the method comprising:

[0006] Get the text to be processed;

[0007] The text is subjected to phoneme conversion processing to obtain phoneme data;

[0008] The phoneme data is converted into a sequence of phoneme feature vectors;

[0009] The phoneme feature vector sequence is divided into at least two feature vector blocks, and one of the at least two feature vector blocks is determined as the target vector block;

[0010] The target vector block is subjected to acoustic feature mapping processing to obtain the acoustic feature data of the target vector block;

[0011] The acoustic feature data is input into a vocoder for speech conversion processing, and the synthesized audio corresponding to the target vector block is output.

[0012] If the target vector block is not the last feature vector block, update the target vector block to the next feature vector block, and return to the step of performing acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block, until the target vector block is the last feature vector block, which is the last feature vector block in the phoneme feature vector sequence.

[0013] Optionally, the step of performing phoneme conversion processing on the text to obtain phoneme data includes:

[0014] Obtain the first sentence of the text and determine the first sentence as the target sentence;

[0015] The target statement is subjected to phoneme conversion processing to obtain the phoneme data of the target statement;

[0016] The method further includes: if the target vector block is the last feature vector block, and the target statement is not the last statement, then the target statement is updated to the next statement in the text, and the step of performing phoneme conversion processing on the target statement to obtain the phoneme data of the target statement is returned, until the target statement is the last statement, and the last statement is the last statement of the text.

[0017] Optionally, converting the phoneme data into a phoneme feature vector sequence includes: calling the encoder of the acoustic model to convert the phoneme data of the target sentence into a phoneme feature vector sequence of the target sentence;

[0018] The step of performing acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block includes: calling the decoder of the acoustic model to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block.

[0019] Optionally, before updating the target vector block to the next feature vector block, the method further includes:

[0020] Extract the last portion of the vector from the target vector block;

[0021] Before performing acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block, the method further includes:

[0022] If the target vector block is not the first feature vector block, the partial vector is concatenated with the target vector block to obtain the concatenated target vector block.

[0023] Optionally, obtaining the text to be processed includes:

[0024] Get the text to be processed;

[0025] The text is preprocessed according to the target rules to obtain the preprocessed text. The target rules are used to limit the content and / or format of the text that can be converted into audio.

[0026] Optionally, the audio generation method is applied to an electronic device; the size of each feature vector block satisfies at least one of the following:

[0027] It is positively correlated with the size of the cache memory of the electronic device;

[0028] It is positively correlated with the computing power of the decoder of the acoustic model;

[0029] It is positively correlated with the computing power of the vocoder.

[0030] Optionally, the step of performing phoneme conversion processing on the text to obtain phoneme data includes: calling a text front-end module of the target data type to perform phoneme conversion processing on the text to obtain phoneme data;

[0031] The step of calling the encoder of the acoustic model to convert the phoneme data of the target sentence into a phoneme feature vector sequence of the target sentence includes: calling the encoder in the acoustic model of the target data type to convert the phoneme data of the target sentence into a phoneme feature vector sequence of the target sentence;

[0032] The step of calling the decoder of the acoustic model to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block includes: calling the decoder in the acoustic model of the target data type to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block;

[0033] The step of inputting the acoustic feature data into a vocoder for speech conversion processing and outputting the synthesized audio corresponding to the target vector block includes: calling a vocoder of the target data type, performing speech conversion processing on the acoustic feature data, and outputting the synthesized audio corresponding to the target vector block, wherein the target data type includes a half-precision floating-point number type.

[0034] According to a second aspect of the embodiments of this application, an audio generation apparatus is provided, the apparatus comprising:

[0035] The acquisition module is used to acquire the text to be processed.

[0036] The first conversion module is used to perform phoneme conversion processing on the text to obtain phoneme data;

[0037] The second conversion module is used to convert the phoneme data into a phoneme feature vector sequence;

[0038] The determining module is used to divide the phoneme feature vector sequence into at least two feature vector blocks, and determine one of the at least two feature vector blocks as the target vector block;

[0039] The mapping module is used to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block;

[0040] The third conversion module is used to input the acoustic feature data into a vocoder for speech conversion processing and output the synthesized audio corresponding to the target vector block.

[0041] An update module is configured to update the target vector block to the next feature vector block if the target vector block is not the last feature vector block, until the target vector block is the last feature vector block, which is the last feature vector block in the phoneme feature vector sequence.

[0042] According to a third aspect of the present application, an electronic device is provided, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the audio generation method as described in the first aspect.

[0043] According to a fourth aspect of the embodiments of this application, an embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the audio generation method as described in the first aspect.

[0044] According to a fifth aspect of the embodiments of this application, an embodiment of this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the audio generation method as described in the first aspect.

[0045] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0046] In this embodiment, after obtaining the text to be processed, a phoneme feature vector sequence corresponding to the text is obtained through conversion. This phoneme feature vector sequence is then divided into at least two feature vector blocks. Acoustic feature mapping and speech conversion processing are then performed sequentially on each of the at least two feature vector blocks to obtain synthesized audio corresponding to each of the at least two feature vector blocks, thus achieving text-to-audio conversion. This technical solution can split the phoneme feature vector sequence corresponding to the text into multiple feature vector blocks, allowing for the sequential output of synthesized audio from each feature vector block, achieving a text-to-audio conversion scheme under streaming processing. Furthermore, since the computational cost of converting a single feature vector block into audio is relatively small compared to a phoneme feature vector sequence, the method of sequentially outputting audio using feature vector blocks as processing units can effectively improve the speed of text-to-speech conversion and reduce resource consumption during speech conversion. Additionally, the computational cost of a single feature vector block is smaller than that of a phoneme feature vector sequence. Therefore, compared to converting the entire phoneme feature vector sequence and then outputting the synthesized audio, converting individual feature vector blocks in batches and outputting their synthesized audio can effectively shorten the time between the start of text conversion and the output audio, improve audio output speed, reduce speech conversion latency, and output text-to-speech conversion faster.

[0047] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0049] Figure 1 This is a flowchart of an audio generation method provided in an embodiment of this application;

[0050] Figure 2 This is a flowchart of another audio generation method provided in the embodiments of this application;

[0051] Figure 3 This is a block diagram of an audio generation system provided in an embodiment of this application;

[0052] Figure 4 This is a block diagram of an audio generation apparatus provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0055] Text-to-speech (TTS) technology is a technology that converts text information into natural and fluent speech output. It is widely used in various scenarios such as navigation, e-readers, in-vehicle systems, and voice assistants.

[0056] However, although TTS is currently widely used, it suffers from slow text-to-speech conversion speed, high latency, and high resource consumption on devices with lower computing power, such as in-vehicle systems, because it requires strong computing power.

[0057] Taking a TTS system requiring 100MB of resources as an example, when using an in-vehicle device to run a TTS system provided by related technologies, the Qualcomm 8155 chip in the device has a real-time audio synthesis rate between 0.8s and 1.0s. Therefore, assuming a 2-second audio clip needs to be synthesized, the TTS system would require 1.6s to 2s to perform the audio synthesis. Clearly, current TTS systems on devices with limited computing power, such as in-vehicle systems, suffer from slow text-to-speech conversion speeds, high latency, and high resource consumption.

[0058] Please refer to Figure 1 This document illustrates a flowchart of an audio generation method provided in an embodiment of this application. The audio generation method can be applied to electronic devices such as in-vehicle devices, computers, and tablets. This embodiment uses the application of the audio generation method to an in-vehicle device as an example for illustration. Figure 1 As shown, the audio generation method includes:

[0059] Step 101: Obtain the text to be processed.

[0060] In this embodiment, the text to be processed refers to the text to be converted into audio. Optionally, the text may refer to text entered by the user, text searched by software running on the in-vehicle device, etc.

[0061] Step 102: Perform phoneme conversion on the text to obtain the phoneme data of the text.

[0062] Optionally, the in-vehicle device can utilize a text front-end module to perform phoneme conversion processing on the text to obtain the text's phoneme data. Specifically, the in-vehicle device can optionally call a phoneme conversion model to convert the text into phonemes, obtaining the text's phoneme data. The phoneme conversion model is used to convert text into phonemes. The phoneme data includes at least one phoneme for each sentence in the text.

[0063] In one optional implementation, the phoneme conversion model can be a text front-end. The text front-end can convert the input raw text into a phoneme representation, making it easier for the back-end system to process and generate speech. Specifically, the text front-end can achieve the conversion from raw text to phonemes through processes such as text preprocessing, text regularization, word segmentation, part-of-speech tagging, prosodic prediction, and character-to-phoneme conversion. In another optional implementation, the phoneme conversion model can be a grapheme-to-phoneme (G2P) model. The in-vehicle device can input text into the G2P model and obtain the phoneme data (also known as a phoneme sequence) output by the G2P model.

[0064] Step 103: Convert the phoneme data into a sequence of phoneme feature vectors.

[0065] In this embodiment, the phoneme feature vector sequence is called the embedding sequence. The phoneme feature vector sequence includes multiple phoneme feature vectors arranged in the order of sentences in the text. This phoneme feature vector is a lower-dimensional, more compact representation of the original phoneme data, and it fully satisfies the key features of the phoneme data.

[0066] Optionally, the in-vehicle device can convert phoneme data into a series of phoneme feature vectors using a feature extraction model. This feature extraction model can be a neural network module. For example, the feature extraction model can be a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a combination of CNN and RNN, etc.

[0067] In some embodiments of this application, the vehicle-mounted device can invoke the encoder of the acoustic model to convert phoneme data into a sequence of phoneme feature vectors. Optionally, the vehicle-mounted device can invoke the feature extraction model through the encoder of the acoustic model to convert phoneme data into a series of phoneme feature vectors, thereby obtaining a sequence of phoneme feature vectors.

[0068] Step 104: Divide the phoneme feature vector sequence into at least two feature vector blocks, and determine one of the feature vector blocks as the target vector block.

[0069] Optionally, the size of each feature vector block can be a preset value. The vehicle-mounted device can obtain the preset feature vector block size and divide the phoneme feature vector sequence into at least two feature vector blocks. For example, suppose the size of a single feature vector block is 32 phoneme feature vectors. If the phoneme feature vector sequence includes 320 phoneme feature vectors, the vehicle-mounted device can sequentially divide the phoneme feature vector sequence into 10 feature vector blocks. Each feature vector block includes 32 phoneme feature vectors.

[0070] Alternatively, in some embodiments, the number of feature vector blocks into which a single phoneme feature vector sequence can be divided is a preset value. The vehicle-mounted device can obtain the number of feature vector blocks into which a single phoneme feature vector sequence can be divided. Based on the obtained number, the phoneme feature vector sequence is sequentially divided into at least two feature vector blocks. The size of each feature vector block can be equal, or the size of each feature vector block can be unequal.

[0071] Optionally, the in-vehicle device can divide the phoneme feature vector sequence into at least two feature vector blocks. First, the first feature vector block is designated as the target vector block for subsequent audio conversion processing to generate the synthesized audio corresponding to the target vector block. Then, each subsequent feature vector block is designated as the target vector block for subsequent audio conversion processing to generate the synthesized audio corresponding to the next feature vector block, and so on, until the last feature vector block is designated as the target vector block for subsequent audio conversion processing to generate the synthesized audio corresponding to the last feature vector block. This yields the synthesized audio after complete text conversion.

[0072] In this embodiment, after dividing the phoneme feature vector sequence into at least two feature vector blocks, subsequent steps 105 to 106 can be performed sequentially on at least two feature vector blocks to output a synthesized audio after each feature vector block is processed. Thus, by continuously processing at least two feature vector blocks of the phoneme feature vector sequence to output continuous synthesized audio, text-to-audio conversion is achieved. It is easy to understand that this allows for the splitting and conversion of text, sequentially outputting synthesized audio corresponding to portions of the text, until the synthesized audio of the completely converted text is output.

[0073] Step 105: Perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block.

[0074] Optionally, the vehicle-mounted device can perform acoustic feature mapping processing on the target vector block using a spectroscopic feature model to obtain the acoustic feature data corresponding to the target vector block. The spectroscopic feature model is used to map the acoustic feature data of the target vector block. For example, the spectroscopic feature model can be a Mel-spectrum feature model or an LPC feature model, etc. The Mel-spectrum feature module is used to map the Mel-spectrum feature data of the phoneme feature vector sequence. The LPC feature model is used to map the LPC feature data of the phoneme feature vector sequence.

[0075] For example, the vehicle-mounted device inputs the target vector block into the Mel-spectrum feature model and obtains the Mel-spectrum feature data corresponding to the target vector block output by the Mel-spectrum feature model. The Mel-spectrum feature data can be Mel-Frequency Cepstral Coefficients (MFCCs).

[0076] In some embodiments of this application, the vehicle-mounted device can call the decoder of the acoustic model to perform acoustic feature mapping processing on the target vector block, thereby obtaining the acoustic feature data corresponding to the target vector block. Optionally, the vehicle-mounted device can use the decoder to call the acoustic spectrum feature model to perform acoustic feature mapping processing on the target vector block, thereby obtaining the acoustic feature data corresponding to the target vector block.

[0077] Step 106: Input the acoustic feature data of the target vector block into the vocoder for speech conversion processing, and output the synthesized audio corresponding to the target vector block.

[0078] Optionally, the vehicle-mounted device can perform feature decoding on the acoustic feature data of the target vector block to obtain the acoustic spectrum data corresponding to the acoustic feature data. Then, the acoustic spectrum is input into a pre-trained vocoder for speech waveform conversion to obtain synthesized audio in the media format corresponding to the text, i.e., the synthesized audio corresponding to the target vector block. The media format corresponding to the synthesized audio can be a Moving Picture Experts Group Audio Layer III (MP3), Moving Picture Experts Group Audio Layer III (MP4), or other media file formats, which are not limited in this embodiment. It should be noted that, obviously, after completing steps 105 to 106, the synthesized audio of the target vector blocks corresponding to the characters in the text can be output.

[0079] In some embodiments of this application, the vehicle-mounted device can extract a portion of the vector from the previous feature vector block and concatenate it with the next feature vector block. Then, the concatenated feature vector block is subjected to acoustic feature mapping processing and speech conversion processing in sequence, and a synthesized audio with a transition effect to the previous feature vector block is output.

[0080] Based on this, the audio generation method may optionally further include: truncating the last part of the target vector block before updating the target vector block to the next feature vector block.

[0081] Accordingly, before performing acoustic feature mapping processing on the target vector block in step 105 to obtain the acoustic feature data of the target vector block, the audio generation method further includes: when the target vector block is not the first feature vector block, concatenating a portion of the vectors with the target vector block to obtain the concatenated target vector block.

[0082] In this process, after generating the synthesized audio corresponding to the target vector block in step 106, a portion of the vector at the end of the target vector block can be extracted. Then, if the target vector block is not the last feature vector block, it is updated to the next feature vector block. Furthermore, the portion of the vector is concatenated with the updated target vector block to obtain a concatenated target vector block. Steps 105 and 106 are then performed on the concatenated target vector block to output the synthesized audio corresponding to that target vector block.

[0083] Optionally, the vehicle-mounted device can extract the last x phoneme feature vectors from the target vector block. Then, before concatenating these x phoneme feature vectors to the first phoneme feature vector of the updated target vector block, a concatenated target vector block is obtained. Here, x is a positive integer.

[0084] In this way, by concatenating the phoneme vector features at the end of the previous feature vector block with the next feature vector block, the synthesized audio from the next feature vector block can have a high degree of flow between the two. That is, when the in-vehicle device uses the cache for audio generation processing, it can pre-extract a portion of the operators calculated by the previous feature vector block (i.e., the phoneme feature vectors in the feature vector block) from the cache and use it for concatenating the next feature vector block. This results in a high degree of flow between the synthesized audio from the two feature vector blocks.

[0085] In one alternative implementation, the audio generation method is applied to an electronic device; the size of each feature vector block satisfies at least one of the following: positively correlated with the size of the electronic device's cache memory; positively correlated with the computing power of the decoder of the acoustic model; and positively correlated with the computing power of the vocoder.

[0086] Optionally, the vehicle-mounted device can test feature vector blocks of different sizes during the execution of the audio generation method provided in this application embodiment, considering the computing power data of the vocoder or the decoder of the acoustic model. Furthermore, the size of the feature vector block corresponding to the minimum computing power can be selected as the final size of each feature vector block.

[0087] Similarly, alternatively, the in-vehicle device can also test feature vector blocks of different sizes during the execution of the audio generation method provided in this application embodiment, considering the computing power data of the vocoder and the decoder of the acoustic model. Then, based on the computing power data of the vocoder and decoder, the size of the corresponding feature vector block is selected as the final size of each feature vector block.

[0088] Similarly, alternatively, the in-vehicle device can select the size of the feature vector block based on the size of the electronic device's cache memory, so that the size of a single feature vector block is positively correlated with the size of the electronic device's cache memory. This is because audio synthesis is essentially an upsampling process. During audio synthesis, the sampled data doubles, and correspondingly, the data to be processed also doubles. Assuming the sampled data volume is 80 × 512, then after upsampling by 4 times, the data volume becomes 80 × 2048. Obviously, the computational load of the chip in the electronic device doubles by 4 × 4 times. At this point, the electronic device's cache memory may not be able to accommodate the doubled processing data, resulting in a low hit rate for data to be processed from the cache memory. This necessitates reading data from memory, placing a significant burden on the memory, and the processing speed of reading and writing data from memory is slower than reading and writing data from the cache memory. Based on this, by optimizing the size of the feature vector block according to the size of the cache memory, the individual feature vector block can hit the data in the cache memory as much as possible during the calculation process, thereby improving the data read and write processing speed, finding a balance between data calculation and cache memory hit rate, effectively improving the calculation speed, and thus improving the overall text-to-speech conversion speed and reducing speech conversion latency.

[0089] Step 107: Determine if the target vector block is the last feature vector block. If not, proceed to step 108; if yes, end the audio generation process. The last feature vector block is the last feature vector block in the phoneme feature vector sequence.

[0090] In one alternative implementation, the vehicle-mounted device can determine the target vector block by sequentially marking effect vector blocks in the phoneme feature vector sequence. Correspondingly, the vehicle-mounted device can determine whether the current target vector block is the last feature vector block by checking if the number of unmarked feature vector blocks is zero.

[0091] For example, suppose the phoneme feature vector sequence is divided into 3 feature vector blocks. The in-vehicle device can first mark the first feature vector block in the phoneme feature vector sequence, determining that this feature vector block is the target vector block. The target vector block is then subjected to acoustic feature mapping processing and speech conversion processing sequentially to generate the synthesized audio corresponding to the target vector block. The in-vehicle device determines that the number of unmarked feature vector blocks in the phoneme feature vector sequence (2) is not equal to 0, thus indicating that the target vector block is not the last feature vector block.

[0092] In another alternative implementation, the vehicle-mounted device can determine the target vector block by sequentially writing feature vector blocks from the phoneme feature vector sequence into a processed queue. Correspondingly, the vehicle-mounted device can determine whether the current target vector block is the last feature vector block by checking if the number of feature vector blocks in the processed queue is the target number. Here, the target number refers to the total number of feature vector blocks divided from the phoneme feature vector sequence.

[0093] For example, suppose the phoneme feature vector sequence is divided into 3 feature vector blocks. The in-vehicle device can initialize and create an empty processed queue. First, the first feature vector block in the phoneme feature vector sequence is written into the processed queue, and this first feature vector block is used as the target vector block. The in-vehicle device performs acoustic feature mapping processing and speech conversion processing on the target vector block sequentially to generate the synthesized audio corresponding to the target vector block. The in-vehicle device determines that the number of feature vector blocks in the processed queue is not equal to 3 (1), and therefore determines that the target vector block is not the last feature vector block.

[0094] Step 108: Update the target vector block to the next feature vector block, and return to step 105 to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block.

[0095] In this embodiment of the application, if the target vector block is not the last feature vector block, the vehicle-mounted device updates the target vector block to the next feature vector block and returns to step 104 to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block, until the target vector block is the last feature vector block, and then ends the audio generation process.

[0096] For example, after acquiring the text to be processed, the vehicle-mounted device can perform phoneme conversion processing on the text to obtain phoneme data, and then convert the phoneme data into a phoneme feature vector sequence. The vehicle-mounted device divides the phoneme feature vector sequence into 3 feature vector blocks, and marks the first feature vector block among the 3 feature vector blocks, determining that feature vector block as the target vector block.

[0097] The in-vehicle device sequentially performs acoustic feature mapping and speech conversion processing on the target vector block to generate the corresponding synthesized audio. Then, the device determines that the number of unlabeled feature vector blocks in the phoneme feature vector sequence is not equal to 0 (2), thus confirming that the target vector block is not the last feature vector block. The device then labels the second of the three feature vector blocks, identifying it as the target vector block, and sequentially performs acoustic feature mapping and speech conversion processing on it to generate the corresponding synthesized audio.

[0098] Subsequently, the in-vehicle device determines that the number of unlabeled feature vector blocks in the phoneme feature vector sequence is not equal to 0 (1), thus confirming that the target vector block is not the last feature vector block. The in-vehicle device labels the third feature vector block out of the three, identifying it as the target vector block, and then sequentially performs acoustic feature mapping and speech conversion processing on the target vector block to generate the synthesized audio corresponding to the target vector block. The in-vehicle device again determines that the number of unlabeled feature vector blocks in the phoneme feature vector sequence is equal to 0 (0), ending the audio generation process. Clearly, the technical solution provided in this application embodiment can employ a text splitting method, sequentially outputting the synthesized audio corresponding to portions of the text until the synthesized audio of the completely converted text is output, thus realizing a text-to-audio conversion scheme under streaming processing.

[0099] Since converting a single feature vector block into audio requires relatively less computation compared to a sequence of phoneme feature vectors, streaming text-to-audio conversion schemes offer advantages such as high processing speed, low latency, and low resource consumption. Furthermore, because the computational cost of a single feature vector block is smaller than that of a phoneme feature vector sequence, converting individual feature vector blocks in batches and outputting their synthesized audio, compared to converting the entire phoneme feature vector sequence and then outputting the synthesized audio, can effectively shorten the time from the start of text conversion to audio output, increase audio output speed, reduce speech conversion latency, and output text-to-speech conversion faster.

[0100] In this embodiment, after obtaining the text to be processed, a phoneme feature vector sequence corresponding to the text is obtained through conversion. This phoneme feature vector sequence is then divided into at least two feature vector blocks. Acoustic feature mapping and speech conversion processing are then performed sequentially on each of the at least two feature vector blocks to obtain synthesized audio corresponding to each of the at least two feature vector blocks, thus achieving text-to-audio conversion. This technical solution can split the phoneme feature vector sequence corresponding to the text into multiple feature vector blocks, allowing for the sequential output of synthesized audio from each feature vector block, achieving a text-to-audio conversion scheme under streaming processing. Furthermore, since the computational cost of converting a single feature vector block into audio is relatively small compared to a phoneme feature vector sequence, the method of sequentially outputting audio using feature vector blocks as processing units can effectively improve the speed of text-to-speech conversion and reduce resource consumption during speech conversion. Additionally, the computational cost of a single feature vector block is smaller than that of a phoneme feature vector sequence. Therefore, compared to converting the entire phoneme feature vector sequence and then outputting the synthesized audio, converting individual feature vector blocks in batches and outputting their synthesized audio can effectively shorten the time between the start of text conversion and the output audio, improve audio output speed, reduce speech conversion latency, and output text-to-speech conversion faster.

[0101] In some embodiments of this application, effective format conversion can be performed before the text to be processed is converted to speech, in order to ensure the success of the text-to-speech conversion. Based on this, step 101, which involves obtaining the text to be processed, may optionally include:

[0102] Obtain the text to be processed. Preprocess the text according to the target rules to obtain the preprocessed text. The target rules are used to limit the content and / or format of the text that can be converted into audio.

[0103] In one optional implementation, the target rule can be a full-match rule. A full-match rule includes both poor-quality text and its corresponding standard text. Based on this, the process by which the in-vehicle device preprocesses the text according to the target rule to obtain the preprocessed text may include: performing a consistency match between the text to be processed and the poor-quality text to determine whether the text to be processed is poor-quality text.

[0104] If a match is successful, it indicates that the text to be processed is substandard text, and the in-vehicle device can replace the substandard text with the target standard text to obtain the preprocessed text. The target standard text is the standard text corresponding to the matched substandard text. If a match is unsuccessful, it indicates that the text to be processed is not substandard text, and the in-vehicle device can directly identify the substandard text as the preprocessed text. In some embodiments, the full-scale matching rule can be a pre-set rule. Alternatively, the full-scale matching rule can also be a rule collected and corrected during processing.

[0105] In another alternative implementation, the target rule can be a character matching rule. The character matching rule includes inferior characters and their corresponding standard characters. Based on this, the in-vehicle device preprocesses the text according to the target rule to obtain the preprocessed text. This process can include: performing consistency matching between each character in the text to be processed and the inferior characters to determine whether the text to be processed contains inferior characters.

[0106] If a match is successful, it indicates that the text to be processed contains substandard characters. The in-vehicle device can replace these substandard characters with target standard characters to obtain preprocessed text. The target standard character is the standard character corresponding to the matched substandard character. If a match is unsuccessful, it indicates that the text to be processed does not contain substandard characters, and the in-vehicle device can directly identify the text to be processed as preprocessed text. In some embodiments, the character matching rule can be a pre-set rule. Alternatively, the character matching rule can also be a rule collected and corrected during processing.

[0107] In another alternative implementation, the target rule can be a fuzzy matching rule. The fuzzy matching rule includes a regular expression. This regular expression is used to match substandard content and / or rules in the text that do not meet the requirements, and to define standard content and / or rules that can be replaced by these substandard content and / or rules. By utilizing fuzzy matching rules, text content detection and format correction can be performed on the text to be processed, thereby converting the text into text that is more recognizable by the TTS system, thus effectively improving the accuracy of text-to-speech conversion.

[0108] For example, the target rules may also include: full matching rules, character matching rules, and fuzzy matching rules. Among these, full matching rules have higher priority than character matching rules, and character matching rules have higher priority than fuzzy matching rules. Based on this, the process by which the in-vehicle device preprocesses the text according to the target rules to obtain the preprocessed text may include:

[0109] The system performs a consistency matching process between the text to be processed and the substandard text to determine whether the text to be processed is substandard. If a match is successful, it indicates that the text to be processed is substandard, and the in-vehicle device can replace the text to be processed with the target standard text to obtain the preprocessed text. The target standard text is the standard text corresponding to the matched substandard text.

[0110] If a match fails, the text to be processed undergoes another consistency match with the inferior character to determine if the text contains the inferior character. If a match is successful, it indicates that the text contains the inferior character, and the in-vehicle device can replace the inferior character with the target standard character to obtain the preprocessed text. The target standard character is the standard character corresponding to the matched inferior character.

[0111] If a match fails, the text content to be processed and / or the rules are matched and replaced according to the regular expression to obtain the preprocessed text.

[0112] In some embodiments of this application, the process of the vehicle-mounted device performing phoneme conversion processing on text to obtain phoneme data may include: calling a text front-end module of the target data type to perform phoneme conversion processing on the text to obtain phoneme data.

[0113] Correspondingly, the process by which the vehicle-mounted device calls the encoder of the acoustic model to convert the phoneme data of the target statement into a sequence of phoneme feature vectors of the target statement may also include: calling the encoder in the acoustic model of the target data type to convert the phoneme data of the target statement into a sequence of phoneme feature vectors of the target statement.

[0114] Correspondingly, the process by which the vehicle-mounted device calls the decoder of the acoustic model to perform acoustic feature mapping processing on the target vector block and obtain the acoustic feature data of the target vector block may also include: calling the decoder in the acoustic model of the target data type to perform acoustic feature mapping processing on the target vector block and obtain the acoustic feature data of the target vector block.

[0115] Accordingly, the process by which the on-board equipment inputs acoustic feature data into a vocoder for speech conversion processing and outputs the synthesized audio corresponding to the target vector block can also include: calling a vocoder of the target data type, performing speech conversion processing on the acoustic feature data, and outputting the synthesized audio corresponding to the target vector block. The target data type includes half-precision floating-point number type (FP16). Of course, the target data type can also be single-precision floating-point number type (FP32).

[0116] When the target data type is FP16, meaning the TTS system used to execute the audio generation method provided in this application supports FP16, the processing efficiency of a vector computing unit on the same computing chip with the same bit width differs for the target data type. For example, a 128-bit vector computing unit can simultaneously compute 4 FP32 and 8 FP16. Thus, theoretically, the speedup can be doubled when computing data of FP16 type. Tests have shown that the TTS system used to execute the audio generation method provided in this application can achieve a 20% to 30% speedup when supporting FP16. Similarly, by using the FP16 data type to cache the relevant parameters of the TTS system, nearly half of the memory can be saved. Furthermore, by using a memory pool to reuse memory, the memory consumption of the audio generation method provided in this application during execution can be effectively reduced.

[0117] In some embodiments of this application, the in-vehicle device can also perform operator kernel acceleration processing to further optimize the text-to-speech conversion speed and reduce speech conversion latency. The core of audio generation processing is the invocation of various operator kernels. Therefore, by fully utilizing the hardware characteristics of the in-vehicle device and deeply accelerating the operator kernels based on the number of registers supported by the device chip, the audio generation processing speed can be effectively improved. Optionally, in audio generation processing, registers can be called as many times as possible for synchronous calculation to improve calculation speed, thereby achieving reverse acceleration of operator kernel invocation and improving the audio generation processing speed.

[0118] For example, suppose the device chip supports 32 registers. In audio generation and processing, 12 registers can be used for memory read / write operations, and the remaining 20 registers can be used for computation. Thus, if we consider 4 registers as the cache for each set of data, the 20 registers can simultaneously compute 5 sets of data. This drives the operator kernel to accelerate computation as much as possible, thereby improving the audio generation and processing speed.

[0119] In summary, the audio generation method provided in this application, after obtaining the text to be processed, converts it to obtain a phoneme feature vector sequence corresponding to the text, divides the phoneme feature vector sequence into at least two feature vector blocks, and then sequentially performs acoustic feature mapping processing and speech conversion processing on at least two feature vector blocks to sequentially obtain synthesized audio corresponding to at least two feature vector blocks, thereby realizing text-to-audio conversion. In this technical solution, the phoneme feature vector sequence corresponding to the text can be split into multiple feature vector blocks so that the synthesized audio after conversion of multiple feature vector blocks can be output sequentially, realizing a text-to-audio conversion scheme under streaming processing. Furthermore, since the computational cost of converting a single feature vector block into audio is relatively small compared to a phoneme feature vector sequence, the method of sequentially outputting audio using feature vector blocks as processing units can effectively improve the speed of text-to-speech conversion and reduce the resource consumption during speech conversion. In addition, the computational cost of a single feature vector block is smaller than that of a phoneme feature vector sequence. Therefore, compared to converting the entire phoneme feature vector sequence and then outputting the synthesized audio, converting individual feature vector blocks in batches and outputting their synthesized audio can effectively shorten the time between the start of text conversion and the output audio, improve audio output speed, reduce speech conversion latency, and output text-to-speech conversion faster.

[0120] Please refer to Figure 2 This document illustrates a flowchart of an audio generation method provided in an embodiment of this application. The audio generation method can be applied to electronic devices such as in-vehicle devices, computers, and tablets. This embodiment uses the application of the audio generation method to an in-vehicle device as an example for illustration. Figure 2 As shown, the audio generation method includes:

[0121] Step 201: Obtain the text to be processed.

[0122] In this embodiment, the text to be processed refers to the text to be converted into audio. Optionally, the text may refer to text entered by the user, text searched by software running on the in-vehicle device, etc.

[0123] Step 202: Obtain the first sentence of the text and identify it as the target sentence.

[0124] Optionally, the in-vehicle device splits the text into sentences, obtaining individual sentences within the text. The in-vehicle device identifies the first sentence in the text as the target sentence.

[0125] Step 203: Perform phoneme conversion on the target sentence to obtain the phoneme data of the target sentence.

[0126] The explanation and implementation of this step can be found in the explanation and implementation of step 102 above, and will not be repeated in this embodiment.

[0127] Step 204: Convert the phoneme data of the target sentence into a sequence of phoneme feature vectors.

[0128] The explanation and implementation of this step can be found in the explanation and implementation of step 103 above, and will not be repeated in this embodiment.

[0129] Step 205: Divide the phoneme feature vector sequence into at least two feature vector blocks, and determine one of the feature vector blocks as the target vector block.

[0130] The explanation and implementation of this step can be found in the explanation and implementation of step 104 above, and will not be repeated in this embodiment.

[0131] Step 206: Perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block.

[0132] The explanation and implementation of this step can be found in the explanation and implementation of step 105 above, and will not be repeated in this embodiment.

[0133] Step 207: Input the acoustic feature data into the vocoder for speech conversion processing and output the synthesized audio corresponding to the target vector block.

[0134] The explanation and implementation of this step can be found in the explanation and implementation of step 106 above, and will not be repeated in this embodiment.

[0135] Step 208: Determine if the target vector block is the last feature vector block. If not, proceed to step 209. If yes, proceed to step 210.

[0136] The explanation and implementation of this step can be found in the explanation and implementation of step 106 above, and will not be repeated in this embodiment.

[0137] Step 209: Update the target vector block to the next feature vector block, and return to step 206 to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block.

[0138] The explanation and implementation of this step can be found in the explanation and implementation of step 107 above, and will not be repeated in this embodiment.

[0139] Step 210: Determine if the target statement is the last statement. If not, proceed to step 211; if yes, end the audio generation process. The target statement is the last statement, which is the last statement in the text.

[0140] In one alternative implementation, the vehicle-mounted device can identify the target statement by sequentially marking statements in the text. Correspondingly, the vehicle-mounted device can determine whether the current target statement is the last statement by checking if the number of unmarked statements is zero.

[0141] For example, suppose the text contains three sentences. The in-vehicle device can first mark the first sentence in the text to determine that the first sentence is the target sentence. The in-vehicle device performs steps 203 to 209 on the target sentence, outputting the synthesized audio of each feature vector block in the phoneme feature vector sequence corresponding to the target sentence. The in-vehicle device determines whether the target sentence is the last sentence by checking if the number of unmarked sentences in the text (2) is not equal to 0.

[0142] In another alternative implementation, the vehicle-mounted device can determine the target statement by sequentially writing the statements in the text into a second processed queue. Accordingly, the vehicle-mounted device can determine whether the current target statement is the last statement by checking if the number of statements in the second processed queue is a second quantity. Here, the second quantity refers to the total number of statements in the text.

[0143] For example, suppose the text contains three statements. The onboard device can initialize a second processed queue that is initially empty. The first statement from the text is written to the second processed queue, and this first statement is then used as the target statement.

[0144] The vehicle-mounted device executes steps 203 to 209 on the target statement, outputting the synthesized audio of each feature vector block in the phoneme feature vector sequence corresponding to the target statement. The vehicle-mounted device determines whether the number of statements in the second processed queue (1 is not equal to 3) and whether the target statement is the last statement.

[0145] Step 211: Update the target statement to the next statement in the text, and return to step 203 to perform phoneme conversion on the target statement to obtain the phoneme data of the target statement.

[0146] In this embodiment of the application, if the target statement is not the last statement, the vehicle-mounted device updates the target statement to the next statement in the text, returns to steps 203 to 209, and outputs the synthesized audio of each feature vector block in the phoneme feature vector sequence corresponding to the target statement, until the target statement is the last statement, and ends the audio generation process.

[0147] For example, suppose the text contains two sentences. After acquiring the text to be processed, the in-vehicle device can mark the first sentence in the text as the target sentence. The in-vehicle device performs phoneme conversion processing on the target sentence to obtain phoneme data, and then converts the phoneme data into a phoneme feature vector sequence of the target sentence. The in-vehicle device divides the phoneme feature vector sequence of the target sentence into two feature vector blocks, and marks the first feature vector block of the two feature vector blocks, determining that feature vector block as the target vector block.

[0148] The in-vehicle device sequentially performs acoustic feature mapping and speech conversion processing on the target vector block to generate the synthesized audio corresponding to the target vector block. Then, the in-vehicle device determines that the number of unlabeled feature vector blocks in the phoneme feature vector sequence of the target sentence is not equal to 0, thus determining that the target vector block is not the last feature vector block.

[0149] The in-vehicle device marks the second feature vector block out of two, identifying it as the target feature vector block. It then performs acoustic feature mapping and speech conversion processing on the target feature vector block sequentially to generate the synthesized audio corresponding to it. Afterward, the in-vehicle device again determines that the number of unmarked feature vector blocks in the phoneme feature vector sequence is equal to 0, ending the output of the synthesized audio corresponding to the first sentence in the text.

[0150] Next, the onboard device determines that the number of unmarked sentences in the text is not equal to 0, thus confirming that the target sentence is not the last sentence. The onboard device marks the second sentence in the text as the target sentence. The onboard device performs phoneme conversion processing on the target sentence to obtain phoneme data, and then converts the phoneme data into a phoneme feature vector sequence of the target sentence. The onboard device divides the phoneme feature vector sequence of the target sentence into two feature vector blocks, and marks the first feature vector block of the two feature vector blocks, determining that the feature vector block is the target vector block.

[0151] The in-vehicle device sequentially performs acoustic feature mapping and speech conversion processing on the target vector block to generate the synthesized audio corresponding to the target vector block. Then, the in-vehicle device determines that the number of unlabeled feature vector blocks in the phoneme feature vector sequence of the target sentence is not equal to 0, thus determining that the target vector block is not the last feature vector block.

[0152] The in-vehicle device marks the second feature vector block out of two, identifying it as the target vector block. It then sequentially performs acoustic feature mapping and speech conversion processing on the target vector block to generate the corresponding synthesized audio. Afterward, the in-vehicle device again determines that the number of unmarked feature vector blocks in the phoneme feature vector sequence is zero, ending the output of the synthesized audio corresponding to the second sentence in the text. Subsequently, the in-vehicle device again determines that the number of unmarked sentences in the text is zero, ending the audio generation process. Clearly, the technical solution provided in this application embodiment can employ a method of splitting the text into sentences, first outputting the synthesized audio corresponding to the first sentence based on the phoneme feature vector sequence corresponding to the first sentence. Then, while processing the phoneme feature vector sequences corresponding to subsequent sentences, it outputs the synthesized audio of the sentences, ultimately outputting the synthesized audio after complete text conversion, thus realizing a sentence-level streaming text-to-audio conversion scheme.

[0153] Furthermore, the technical solution of this application also adopts a sentence-splitting approach for individual sentences. First, based on the first feature vector block corresponding to the first few characters in the sentence, the synthesized audio of those characters is output. Then, the feature vectors are processed while the synthesized audio is output, ultimately producing the synthesized audio of the complete converted single sentence, thus achieving a character-level streaming text-to-audio conversion solution. The combination of sentence-level and character-level streaming text-to-audio conversion solutions can further improve the speed of text-to-speech conversion, reduce speech conversion latency, and decrease resource consumption during the speech conversion process.

[0154] In this embodiment, after obtaining the text to be processed, a phoneme feature vector sequence corresponding to the text is obtained through conversion. This phoneme feature vector sequence is then divided into at least two feature vector blocks. Acoustic feature mapping and speech conversion processing are then performed sequentially on each of the at least two feature vector blocks to obtain synthesized audio corresponding to each of the at least two feature vector blocks, thus achieving text-to-audio conversion. This technical solution can split the phoneme feature vector sequence corresponding to the text into multiple feature vector blocks, allowing for the sequential output of synthesized audio from each feature vector block, achieving a text-to-audio conversion scheme under streaming processing. Furthermore, since the computational cost of converting a single feature vector block into audio is relatively small compared to a phoneme feature vector sequence, the method of sequentially outputting audio using feature vector blocks as processing units can effectively improve the speed of text-to-speech conversion and reduce resource consumption during speech conversion. Additionally, the computational cost of a single feature vector block is smaller than that of a phoneme feature vector sequence. Therefore, compared to converting the entire phoneme feature vector sequence and then outputting the synthesized audio, converting individual feature vector blocks in batches and outputting their synthesized audio can effectively shorten the time between the start of text conversion and the output audio, improve audio output speed, reduce speech conversion latency, and output text-to-speech conversion faster.

[0155] Furthermore, this technical solution can also employ a method of splitting sentences within the text. First, based on the phoneme feature vector sequence corresponding to the first sentence, the synthesized audio corresponding to the first sentence is output. Then, while processing the phoneme feature vector sequences corresponding to subsequent sentences, the synthesized audio of the sentences is output simultaneously. This achieves a streaming text-to-audio conversion scheme that combines sentence-level and character-level processing, further improving the speed of text-to-speech conversion, reducing speech conversion latency, and lowering resource consumption during the speech conversion process.

[0156] For example, such as Figure 3 As shown, the audio synthesis system, or TTS system, includes: a text front-end module, an intervention text module, an acoustic model module, and a vocoder module. The acoustic model module includes an encoder and a decoder.

[0157] In an audio synthesis method provided in this application embodiment, an intervention text module can acquire the text to be processed and preprocess the text according to target rules to obtain the preprocessed text. The intervention text module outputs the preprocessed text to the text front-end module.

[0158] The text front-end module splits the received text into sentences, obtaining the individual sentences. The text front-end module then performs phoneme conversion on each sentence sequentially, and outputs the phoneme data of each sentence to the encoder of the acoustic model module after obtaining the phoneme data of that sentence.

[0159] The encoder of the acoustic model module converts the phoneme data of a received statement into a sequence of phoneme feature vectors and outputs a sequence of phoneme feature vectors corresponding to the statement to the decoder of the acoustic model module.

[0160] The decoder of the acoustic model module divides the phoneme feature vector sequence corresponding to a sentence into at least two feature vector blocks, and performs acoustic feature mapping processing on each feature vector block in sequence, so that after obtaining the acoustic feature data of each feature vector block, it outputs the acoustic feature data of one feature vector block to the vocoder module.

[0161] The vocoder module processes the acoustic feature data of a received feature vector block into speech and outputs the synthesized audio corresponding to the feature vector block. This process continues until the synthesized audio of each feature vector block corresponding to each sentence in the text is output, thus completing the output of the synthesized audio corresponding to the entire text.

[0162] Clearly, in the audio synthesis system provided in this application embodiment, the encoders of the text front-end module and the acoustic model module employ statement-level streaming processing. The decoder and vocoder modules of the acoustic model module employ character-level (i.e., feature vector block-level) streaming processing.

[0163] In summary, the audio synthesis method provided in this application, after obtaining the text to be processed, converts it to obtain a phoneme feature vector sequence corresponding to the text, divides the phoneme feature vector sequence into at least two feature vector blocks, and then sequentially performs acoustic feature mapping processing and speech conversion processing on at least two feature vector blocks to sequentially obtain synthesized audio corresponding to at least two feature vector blocks, thereby realizing text-to-audio conversion. In this technical solution, the phoneme feature vector sequence corresponding to the text can be split into multiple feature vector blocks so that the synthesized audio after conversion of multiple feature vector blocks can be output sequentially, realizing a text-to-audio conversion scheme under streaming processing. Furthermore, since the computational cost of converting a single feature vector block into audio is relatively small compared to a phoneme feature vector sequence, the method of sequentially outputting audio using feature vector blocks as processing units can effectively improve the speed of text-to-speech conversion and reduce resource consumption during speech conversion. In addition, the computational cost of a single feature vector block is smaller than that of a phoneme feature vector sequence. Therefore, compared to converting the entire phoneme feature vector sequence and then outputting the synthesized audio, converting individual feature vector blocks in batches and outputting their synthesized audio can effectively shorten the time between the start of text conversion and the output audio, improve audio output speed, reduce speech conversion latency, and output text-to-speech conversion faster.

[0164] Furthermore, this technical solution can also employ a method of splitting sentences within the text. First, based on the phoneme feature vector sequence corresponding to the first sentence, the synthesized audio corresponding to the first sentence is output. Then, while processing the phoneme feature vector sequences corresponding to subsequent sentences, the synthesized audio of the sentences is output simultaneously. This achieves a streaming text-to-audio conversion scheme that combines sentence-level and character-level processing, further improving the speed of text-to-speech conversion, reducing speech conversion latency, and lowering resource consumption during the speech conversion process.

[0165] Please refer to Figure 4 The diagram illustrates a block diagram of an audio generation apparatus according to an embodiment of this application. Optionally, the audio generation apparatus can be installed in electronic devices such as in-vehicle equipment, computers, tablets, etc. Figure 4 As shown, the audio generation device 400 includes: an acquisition module 401, a first conversion module 402, a second conversion module 403, a determination module 404, a mapping module 405, a third conversion module 406, and an update module 407.

[0166] Module 401 is used to acquire the text to be processed;

[0167] The first conversion module 402 is used to perform phoneme conversion processing on the text to obtain phoneme data;

[0168] The second conversion module 403 is used to convert phoneme data into a phoneme feature vector sequence;

[0169] The determination module 404 is used to divide the phoneme feature vector sequence into at least two feature vector blocks and determine one of the at least two feature vector blocks as the target vector block;

[0170] The mapping module 405 is used to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block;

[0171] The third conversion module 406 is used to input acoustic feature data into a vocoder for speech conversion processing and output the synthesized audio corresponding to the target vector block.

[0172] The update module 407 is used to update the target vector block to the next feature vector block when the target vector block is not the last feature vector block, until the target vector block is the last feature vector block, which is the last feature vector block in the phoneme feature vector sequence.

[0173] In this embodiment, after obtaining the text to be processed, a phoneme feature vector sequence corresponding to the text is obtained through conversion. This phoneme feature vector sequence is then divided into at least two feature vector blocks. Acoustic feature mapping and speech conversion processing are then performed sequentially on each of the at least two feature vector blocks to obtain synthesized audio corresponding to each of the at least two feature vector blocks, thus achieving text-to-audio conversion. This technical solution can split the phoneme feature vector sequence corresponding to the text into multiple feature vector blocks, allowing for the sequential output of synthesized audio from each feature vector block, achieving a text-to-audio conversion scheme under streaming processing. Furthermore, since the computational cost of converting a single feature vector block into audio is relatively small compared to a phoneme feature vector sequence, the method of sequentially outputting audio using feature vector blocks as processing units can effectively improve the speed of text-to-speech conversion and reduce resource consumption during speech conversion. Additionally, the computational cost of a single feature vector block is smaller than that of a phoneme feature vector sequence. Therefore, compared to converting the entire phoneme feature vector sequence and then outputting the synthesized audio, converting individual feature vector blocks in batches and outputting their synthesized audio can effectively shorten the time between the start of text conversion and the output audio, improve audio output speed, reduce speech conversion latency, and output text-to-speech conversion faster.

[0174] Optionally, the first conversion module 402 is also used for:

[0175] Get the first sentence of the text and set it as the target sentence;

[0176] The target sentence is processed by phoneme conversion to obtain the phoneme data of the target sentence;

[0177] The update module 407 is also used to update the target statement to the next statement in the text if the target statement is not the last statement when the target vector block is the last feature vector block, until the target statement is the last statement in the text.

[0178] Optionally, the second conversion module 403 is also used to call the encoder of the acoustic model to convert the phoneme data of the target sentence into a sequence of phoneme feature vectors of the target sentence;

[0179] The mapping module 405 is also used to call the decoder of the acoustic model to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block.

[0180] The device also includes: a truncation module for truncating the last portion of the vector in the target vector block;

[0181] The concatenation module is used to concatenate a portion of the vectors with the target vector block when the target vector block is not the first feature vector block, to obtain the concatenated target vector block.

[0182] Optionally, the acquisition module 401 is also used for:

[0183] Get the text to be processed;

[0184] The text is preprocessed according to the target rules to obtain the preprocessed text. The target rules are used to limit the content and / or format of the text that can be converted into audio.

[0185] Optionally, the audio generation method is applied to an electronic device; the size of each feature vector block satisfies at least one of the following: positively correlated with the size of the electronic device's cache memory;

[0186] It is positively correlated with the computing power of the decoder of the acoustic model;

[0187] It is positively correlated with the computing power of the vocoder.

[0188] Optionally, the first conversion module 402 is also used to call the text front-end module of the target data type to perform phoneme conversion processing on the text to obtain phoneme data;

[0189] The second conversion module 403 is also used to call the encoder in the acoustic model of the target data type to convert the phoneme data of the target sentence into a sequence of phoneme feature vectors of the target sentence.

[0190] The mapping module 405 is also used to call the decoder in the acoustic model of the target data type to perform acoustic feature mapping processing on the target vector block and obtain the acoustic feature data of the target vector block;

[0191] The third conversion module 406 is also used to call the vocoder of the target data type to perform speech conversion processing on the acoustic feature data and output the synthesized audio corresponding to the target vector block. The target data type includes half-precision floating-point type.

[0192] In summary, the audio synthesis apparatus provided in this application, after acquiring the text to be processed, converts it to obtain a phoneme feature vector sequence corresponding to the text, divides the phoneme feature vector sequence into at least two feature vector blocks, and then sequentially performs acoustic feature mapping processing and speech conversion processing on at least two feature vector blocks to sequentially obtain synthesized audio corresponding to at least two feature vector blocks, thereby realizing text-to-audio conversion. In this technical solution, the phoneme feature vector sequence corresponding to the text can be split into multiple feature vector blocks so that the synthesized audio converted from multiple feature vector blocks can be output sequentially, realizing a text-to-audio conversion scheme under streaming processing. Furthermore, since the computational cost of converting a single feature vector block into audio is relatively small compared to a phoneme feature vector sequence, the method of sequentially outputting audio using feature vector blocks as processing units can effectively improve the speed of text-to-speech conversion and reduce resource consumption during speech conversion. In addition, the computational cost of a single feature vector block is smaller than that of a phoneme feature vector sequence. Therefore, compared to converting the entire phoneme feature vector sequence and then outputting the synthesized audio, converting individual feature vector blocks in batches and outputting their synthesized audio can effectively shorten the time between the start of text conversion and the output audio, improve audio output speed, reduce speech conversion latency, and output text-to-speech conversion faster.

[0193] This application also provides an electronic device. For example... Figure 5 As shown, the electronic device 500 includes a processor 501, a memory 502, and a computer program stored in the memory 501 and executable on the processor 501. When the computer program is executed by the processor 501, it implements the various processes of the above-described audio generation method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.

[0194] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described audio generation method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0195] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An audio generation method, characterized in that, The method includes: Get the text to be processed; The text is subjected to phoneme conversion processing to obtain phoneme data; The phoneme data is converted into a sequence of phoneme feature vectors; The phoneme feature vector sequence is divided into at least two feature vector blocks, and one of the at least two feature vector blocks is determined as the target vector block; The target vector block is subjected to acoustic feature mapping processing to obtain the acoustic feature data of the target vector block; The acoustic feature data is input into a vocoder for speech conversion processing, and the synthesized audio corresponding to the target vector block is output. If the target vector block is not the last feature vector block, update the target vector block to the next feature vector block, and return to the step of performing acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block, until the target vector block is the last feature vector block, which is the last feature vector block in the phoneme feature vector sequence.

2. The method according to claim 1, characterized in that, The process of phoneme conversion of the text to obtain phoneme data includes: Obtain the first sentence of the text and determine the first sentence as the target sentence; The target statement is subjected to phoneme conversion processing to obtain the phoneme data of the target statement; The method further includes: If the target vector block is the last feature vector block, and the target statement is not the last statement, then the target statement is updated to the next statement in the text, and the process of performing phoneme conversion on the target statement to obtain the phoneme data of the target statement is repeated until the target statement is the last statement, which is the last statement in the text.

3. The method according to claim 2, characterized in that, The step of converting the phoneme data into a phoneme feature vector sequence includes: calling the encoder of the acoustic model to convert the phoneme data of the target sentence into a phoneme feature vector sequence of the target sentence; The step of performing acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block includes: calling the decoder of the acoustic model to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block.

4. The method according to claim 1, characterized in that, Before updating the target vector block to the next feature vector block, the method further includes: Extract the last portion of the vector from the target vector block; Before performing acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block, the method further includes: If the target vector block is not the first feature vector block, the partial vector is concatenated with the target vector block to obtain the concatenated target vector block.

5. The method according to claim 1, characterized in that, The process of obtaining the text to be processed includes: Get the text to be processed; The text is preprocessed according to the target rules to obtain the preprocessed text. The target rules are used to limit the content and / or format of the text that can be converted into audio.

6. The method according to claim 3, characterized in that, The audio generation method is applied to an electronic device; the size of each feature vector block satisfies at least one of the following: It is positively correlated with the size of the cache memory of the electronic device; It is positively correlated with the computing power of the decoder of the acoustic model; It is positively correlated with the computing power of the vocoder.

7. The method according to claim 3, characterized in that, The step of performing phoneme conversion processing on the text to obtain phoneme data includes: calling the text front-end module of the target data type to perform phoneme conversion processing on the text to obtain phoneme data; The step of calling the encoder of the acoustic model to convert the phoneme data of the target sentence into a phoneme feature vector sequence of the target sentence includes: calling the encoder in the acoustic model of the target data type to convert the phoneme data of the target sentence into a phoneme feature vector sequence of the target sentence; The step of calling the decoder of the acoustic model to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block includes: calling the decoder in the acoustic model of the target data type to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block; The step of inputting the acoustic feature data into a vocoder for speech conversion processing and outputting the synthesized audio corresponding to the target vector block includes: calling a vocoder of the target data type, performing speech conversion processing on the acoustic feature data, and outputting the synthesized audio corresponding to the target vector block, wherein the target data type includes a half-precision floating-point number type.

8. An audio generation apparatus, characterized in that, The method includes: The acquisition module is used to acquire the text to be processed. The first conversion module is used to perform phoneme conversion processing on the text to obtain phoneme data; The second conversion module is used to convert the phoneme data into a phoneme feature vector sequence; A determining module is used to divide the phoneme feature vector sequence into at least two feature vector blocks, and determine one of the at least two feature vector blocks as the target vector block; The mapping module is used to perform acoustic feature mapping processing on the target vector block to obtain the acoustic feature data of the target vector block; The third conversion module is used to input the acoustic feature data into a vocoder for speech conversion processing and output the synthesized audio corresponding to the target vector block. An update module is configured to update the target vector block to the next feature vector block if the target vector block is not the last feature vector block, until the target vector block is the last feature vector block, which is the last feature vector block in the phoneme feature vector sequence.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the audio generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the audio generation method as described in any one of claims 1 to 7.

Images