Method for a neural processing unit (NPU) based streaming speech synthesis vocoder and related products

Abstract

Embodiments of the present application provide a method for NPU-based streaming speech synthesis vocoder and related products. The method includes obtaining input features to be processed for the streaming speech, wherein the input features to be processed are of fixed length; processing the input features of fixed length based on a vocoder model deployed on the NPU to output audio information; and determining a synthesis result for the streaming speech based on the audio information. By deploying the vocoder model on the NPU to perform synthesis of the streaming speech, the high performance of the NPU (especially the advantages in neural network inference, etc.) is combined to ensure high-quality speech synthesis while effectively reducing latency and real-time rate to meet actual market demand. In addition, the present application also provides a device and a computer-readable storage medium.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of information processing technology, and more specifically, the embodiments of the present invention relate to a method for an NPU-based streaming speech synthesis vocoder, an apparatus for performing the aforementioned method, and a computer-readable storage medium. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention set forth in the claims. The description herein may include concepts that may be explored, but not necessarily concepts that have been previously conceived or explored. Therefore, unless otherwise stated, what is described in this section is not prior art for the purposes of this application's specification and claims, and is not acknowledged as prior art simply by virtue of its inclusion in this section.

[0003] With the rapid development of artificial intelligence, text-to-speech (TTS) algorithms are becoming increasingly mature. At the same time, in order to achieve faster response speeds and better privacy protection, the importance of edge-side TTS applications is increasing.

[0004] TTS (Text-to-Speech) is a text-to-speech synthesis process, generally consisting of three parts: a text front-end, an acoustic model, and a vocoder. The vocoder plays a decisive role in the sound quality of the synthesized speech. Vocoders can be broadly categorized into phase reconstruction-based vocoders and neural network-based vocoders. Phase reconstruction-based vocoders use algorithms to infer phase features and reconstruct the speech waveform because the acoustic features used in TTS (such as mel features) have lost phase characteristics. Neural network-based vocoders, on the other hand, directly model the acoustic features and speech waveform, resulting in higher sound quality. Currently, most existing technologies use neural network-based vocoders, which largely rely on a Central Processing Unit (CPU). However, with the increasing limitations of CPU computing power and the growing demand for high-quality speech synthesis, CPU-based edge TTS can no longer meet the requirements of low latency and high quality in speech synthesis. Summary of the Invention

[0005] Known CPU-based vocoders produce unsatisfactory speech synthesis results, which is a very frustrating process.

[0006] Therefore, there is a great need for an improved solution and related products for a streaming speech synthesis vocoder based on an embedded neural network processor (NPU), which can effectively improve the sound quality of speech synthesis while reducing latency and real-time rate.

[0007] In this context, embodiments of the present invention aim to provide a method for an NPU-based streaming speech synthesis vocoder and related products.

[0008] In a first aspect of the present invention, a method for a streaming speech synthesis vocoder based on an embedded neural network processor (NPU) is provided, comprising: acquiring input features to be processed regarding the streaming speech, wherein the input features to be processed are of fixed length; processing the fixed-length input features based on a vocoder model deployed on the NPU to output audio information; and determining a synthesis result of the streaming speech based on the audio information.

[0009] In one embodiment of the present invention, obtaining the input features to be processed regarding the streaming speech includes: preprocessing the streaming speech to split the input features to be processed into one or more fixed-length input features.

[0010] In another embodiment of the present invention, preprocessing the streaming speech includes: extracting fixed-length input features from the streaming speech using a sliding window method.

[0011] In another embodiment of the present invention, the sliding window method includes: sequentially sliding a sliding window of a fixed length to extract input features of the fixed length from the streaming speech, wherein the sliding step size of each slide is less than the fixed length, so that there is overlap between the input features extracted by adjacent sliding windows.

[0012] In another embodiment of the invention, the sliding window covers an effective portion located in the middle of the window and an overlapping portion located at both ends of the window, the size of the effective portion being equal to the sliding step size, and the size of the overlapping portion being determined according to the receptive field of the vocoder model structure.

[0013] In one embodiment of the present invention, determining the synthesis result of the streaming speech based on the audio information includes: segmenting the output of the vocoder model based on each input feature to obtain audio information corresponding to the effective part of each input feature; concatenating the audio information corresponding to the effective part of each input feature; and determining the synthesis result based on the concatenated audio information.

[0014] In another embodiment of the present invention, the method further includes: directly using the spliced ​​audio information as the synthesis result; or using a cosine signal to perform fade-in and fade-out post-processing on the spliced ​​audio information to obtain the synthesis result.

[0015] In another embodiment of the present invention, the post-processing of the spliced ​​audio information by using a cosine signal for fade-in and fade-out includes: using the first half-cycle of the cosine signal to fade out the audio information within the first half-cycle window; using the second half-cycle of the cosine signal to fade in the audio information within the second half-cycle window; and performing noise reduction processing on the spliced ​​audio information based on the fade-in processing and the fade-out processing.

[0016] In another embodiment of the present invention, the method further includes: in response to the existence of an input feature with a length less than the fixed length, performing a padding process on the input feature with a length less than the fixed length to adjust its length to the fixed length.

[0017] In one embodiment of the present invention, filling input features whose length is less than the fixed length includes: filling the length of the input feature to the fixed length using a fixed value.

[0018] In another embodiment of the invention, the vocoder model deployed on the NPU is converted from a trained vocoder model that supports streaming speech synthesis.

[0019] In another embodiment of the invention, the vocoder model supporting streaming speech synthesis is converted into a vocoder model deployed on the NPU via an NPU toolchain.

[0020] In a second aspect of the present invention, an apparatus is provided, comprising: an embedded neural network processor (NPU); and a memory storing computer instructions for a streaming speech synthesizer based on the NPU, wherein when the computer instructions are executed by the NPU, the apparatus causes the apparatus to perform the methods described according to the foregoing and the following embodiments.

[0021] In a third aspect of the present invention, a computer-readable storage medium is provided, comprising program instructions for an NPU-based streaming speech synthesizer vocoder, which, when executed by the NPU, causes the device to perform the methods described according to the foregoing and the following embodiments.

[0022] The method and related products of the NPU-based streaming speech synthesis vocoder according to embodiments of the present invention can utilize a vocoder model deployed on an NPU to process the input features of streaming speech with a fixed step size to achieve streaming speech synthesis. It can be seen that the solution of the present invention effectively deploys the vocoder model on the NPU to perform streaming speech synthesis, thereby combining the high performance of the NPU (especially its advantages in neural network inference) to ensure high-quality speech synthesis, while effectively reducing latency and real-time rate to meet actual market demands. Attached Figure Description

[0023] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example, not limitation, in which:

[0024] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0025] Figure 2 A flowchart illustrating a method for an NPU-based streaming speech synthesis vocoder according to an embodiment of the present invention is shown schematically.

[0026] Figure 3 A flowchart illustrating a method for an NPU-based streaming speech synthesis vocoder according to another embodiment of the present invention is shown schematically.

[0027] Figure 4 A flowchart illustrating a method for an NPU-based streaming speech synthesizer vocoder according to yet another embodiment of the present invention is shown.

[0028] Figure 5 A schematic diagram illustrating the use of a sliding window method to process streaming speech according to an embodiment of the present invention is shown.

[0029] Figure 6 A schematic diagram of a cosine signal according to an embodiment of the present invention is shown; and

[0030] Figure 7 A schematic block diagram of a device according to an embodiment of the present invention is shown.

[0031] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0032] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0033] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention is shown. For example... Figure 1As shown, the computing system 100 may include: a central processing unit (CPU) 101, random access memory (RAM) 102, read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that... Figure 1 The structural block diagrams described are for illustrative purposes only and are not intended to limit the scope of the invention. In some cases, certain devices may be added or removed depending on the specific circumstances.

[0034] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0035] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0036] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0037] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0038] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0039] Embodiments of the present invention will now be described with reference to flowchart illustrations and block diagrams of apparatus (or systems) according to embodiments of the invention. It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by the computer or other programmable data processing apparatus, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0040] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0041] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0042] According to an embodiment of the present invention, a method for a streaming speech synthesis vocoder based on an embedded neural network processor (NPU) and related products are proposed.

[0043] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0044] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments. Invention Overview

[0046] The inventors have discovered that existing CPU-based vocoders can no longer meet the demands for low-latency, high-quality speech synthesis. For example, in some high-quality speech synthesis tasks, CPU computing power is significantly limited, particularly its performance disadvantage in neural network inference. Furthermore, as the requirements for the sound quality of synthesized speech increase, CPU-based vocoders are clearly hindering the further development of speech synthesis technology. To address this, the inventors have also discovered that with the rapid development of artificial intelligence, a new type of NPU has emerged. This NPU is a dedicated chip designed for neural network algorithms and computational processes, and it currently shows promising results in fields such as computer vision and Natural Language Processing (NLP). Therefore, the performance advantages of NLP in neural networks can be combined to help vocoders overcome the current bottlenecks in speech synthesis technology.

[0047] Based on this, the inventors discovered that a vocoder model can be effectively deployed on an NPU, and that the processing of streaming speech based on this vocoder model can achieve speech synthesis, thereby effectively improving the sound quality of speech synthesis while reducing latency and real-time rate.

[0048] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0049] Exemplary methods

[0050] The following is for reference. Figure 2This section describes a method for an NPU-based streaming speech synthesizer vocoder according to an exemplary embodiment of the present invention. It should be noted that the embodiments of the present invention can be applied to any applicable scenario.

[0051] Figure 2 A flowchart of a method 200 for an NPU-based streaming speech synthesis vocoder according to an embodiment of the present invention is shown schematically.

[0052] like Figure 2 As shown, in step S201, the input features to be processed for the streaming speech can be obtained, where the input features are of fixed length. Traditional CPU-based vocoder models supporting streaming speech synthesis have variable input lengths, varying depending on the length of the input features, and the range is very large. However, NPUs are generally designed to run after compilation at a static size. This means that combining the NPU with the vocoder model is not simply a matter of superimposing the two, but requires specific adjustments based on their respective performance advantages. Therefore, by obtaining fixed-length input features for the streaming speech and adjusting the traditional variable-length input features to fixed-length input features, it is easier to effectively combine the advantages of the NPU for subsequent processing.

[0053] Next, in step S202, the fixed-length input features can be processed based on the vocoder model deployed on the NPU to output audio information. The NPU has performance advantages such as static graph fixed-length inference and neural network computation. By deploying the vocoder model on the NPU to process the fixed-length input features, the performance advantages of the NPU can be effectively combined to meet the synthesis requirements of high audio quality, low real-time rate and low latency for streaming speech.

[0054] Next, in step S203, the synthesis result of the streaming speech can be determined based on the aforementioned audio information.

[0055] As can be seen, the solution of this invention achieves streaming speech synthesis by adjusting the input features of the streaming speech to a fixed length and utilizing a vocoder model deployed on the NPU to process the fixed-length input features. Therefore, it effectively combines the high performance of the NPU (especially its advantages in neural network inference) to ensure high-quality speech synthesis, while effectively reducing latency and real-time performance to meet actual market demands.

[0056] Figure 3 A flowchart of a method 300 for an NPU-based streaming speech synthesizer vocoder according to another embodiment of the present invention is illustrated schematically. It should be noted that method 300 can be understood as... Figure 2 This is a further supplement or expansion of Chinese method 200. Therefore, the preceding text, combined with... Figure 2 The relevant detailed descriptions also apply to the following text.

[0057] like Figure 3 As shown, in step S301, the aforementioned streaming speech can be preprocessed to split the input features to be processed into one or more fixed-length input features. In practical applications, the splitting of the input features of streaming speech can be achieved in various ways. For example, a sliding window method can be used to extract fixed-length input features from the streaming speech. Specifically, a sliding window of fixed length can be used to sequentially extract fixed-length input features from the streaming speech, with each sliding step being smaller than the fixed length, so that there is overlap between the input features extracted by adjacent sliding windows.

[0058] In some embodiments, a sliding window covers the effective portion located in the middle of the window and the overlapping portions located at both ends of the window, where the size of the effective portion is equal to the sliding stride. The size of the overlapping portion can be determined based on the receptive field of the vocoder model structure. It can be understood that a vocoder model is a neural network model, which typically contains several convolutional layers. A characteristic of convolutional layer operations is that the value of an output feature is derived from contributions from multiple input features; these multiple input features can be called the receptive field of that output feature. When multiple convolutional layers exist, the receptive field is passed down layer by layer; that is, the receptive field can be understood as the size of the region mapped onto the original input data by the features in the output data of each layer of the convolutional neural network. When the receptive field of the model's final output feature as a whole is determined, the model's receptive field can be obtained. Specifically, the receptive field can be calculated based on the kernel size, stride, and padding length of the convolutional layers in the vocoder model, and the model's receptive field is obtained by accumulating the receptive fields, thereby determining the length of the overlapping portion.

[0059] For example, the receptive field of the output features of the first convolutional layer can generally be equal to the filter size, while the receptive field of deeper convolutional layers is related to the filter size, stride, padding length, etc., of all previous layers. Specifically, in the calculation process, the receptive field of the deepest layer on the previous layer can be calculated first, and then gradually passed to the first layer to accumulate the receptive field of the vocoder model structure. In some embodiments, assuming that a minimum overlap of length l is required for receptive field analysis based on the vocoder model structure, the length of the overlap at both ends of the window can be set to be greater than or equal to l. It should be noted that the detailed description of sliding window-related aspects here is merely illustrative.

[0060] It can be seen that by using sliding windows and overlapping methods to split the input features to be processed into one or more fixed-length input features, the effective part of the input features can be preserved to the greatest extent, thereby further ensuring the quality of speech synthesis.

[0061] Furthermore, in some embodiments, during the process of splitting input features, input features with a length less than a fixed length may occur. In this case, in response to the existence of input features with a length less than the fixed length, the input features with a length less than the fixed length are padded to adjust their length to the fixed length. For example, the length of the input features can be padded to the fixed length using fixed values ​​(e.g., a matrix of all zeros).

[0062] After the fixed-length input features are split, step S302 proceeds to process the fixed-length input features based on a vocoder model deployed on the NPU. In some embodiments, a trained vocoder model supporting streaming speech synthesis can be converted into a vocoder model deployed on the NPU. The vocoder model supporting streaming speech synthesis may include an Open Neural Network Exchange (ONNX) model. Specifically, an NPU toolchain (e.g., the RKNN toolchain) can be used to convert the trained ONNX model into an NPU vocoder model (e.g., an RKNN model), and the vocoder model conversion can be performed on the NPU platform using the RKNN SDK. It should be noted that the detailed description of the vocoder model deployment described here is merely illustrative.

[0063] Next, in step S303, the output of the aforementioned vocoder model (specifically, the vocoder model deployed on the NPU) can be segmented based on each input feature to obtain the audio information corresponding to the valid part of each input feature. Therefore, by obtaining the audio information corresponding to the valid part through segmentation, the influence of invalid information on the speech synthesis result can be avoided, and the consumption of subsequent computing resources can be reduced.

[0064] Next, in step S304, the audio information corresponding to the valid parts of each input feature can be concatenated. Thus, by concatenating the audio information corresponding to all valid parts, the integrity of the speech synthesis result can be ensured.

[0065] Finally, at step S305, the synthesis result can be determined based on the spliced ​​audio information. In some embodiments, the spliced ​​audio information can be directly used as the synthesis result. In other embodiments, considering that NPU operations are generally accelerated using low precision, some computational errors may be introduced, which will manifest as noise in the vocoder model (especially noise is easily introduced before and after the audio). Therefore, noise reduction processing can be performed on the spliced ​​audio information.

[0066] In some embodiments, noise reduction of the spliced ​​audio information can be achieved by using a cosine signal to perform fade-in and fade-out post-processing. For example, the first half-cycle of the cosine signal can be used to fade out the audio information within the first half-cycle window, and the second half-cycle of the cosine signal can be used to fade in the audio information within the second half-cycle window. Then, noise reduction of the spliced ​​audio information can be achieved based on the aforementioned fade-in and fade-out processing. It should be noted that the detailed description of the audio information noise reduction processing here is merely illustrative, and the solution of the present invention is not limited thereto. For example, other noise reduction methods such as filtering can also be used.

[0067] Figure 4 A flowchart of a method 400 for an NPU-based streaming speech synthesizer vocoder according to another embodiment of the present invention is illustrated schematically. It should be noted that... Figure 4 It can be understood as Figure 2 Chinese method 200 and Figure 3 A feasible technical implementation process for Chinese method 300. Therefore, the preceding text combines... Figure 2 and Figure 3 The relevant details described in the text also apply to the following text.

[0068] like Figure 4 As shown, model training can be performed in step S401. Specifically, in practical applications, a pre-trained vocoder model supporting streaming speech synthesis can be used directly, or an open neural network exchange model can be trained into a vocoder model supporting streaming speech synthesis.

[0069] Next, in step S402, model conversion can be performed. For example, some NPU toolchains can be used to convert the trained model into an NPU vocoder model and deploy it on the NPU. For example, the ONNX model can be converted into an RKNN model using the RKNN toolchain, and the RKNN model can be deployed on the NPU platform using the RKNN SDK.

[0070] Next, in step S403, pre- and post-processing of the model can be performed. Since the input length of a traditional vocoder model is variable and calculated only once, while the NPU supports fixed-length calculations, in practical applications, a sliding window method can be used to calculate fixed-size blocks each time, and the overlapping portions before and after calculations can be combined to ensure the quality of the synthesized image.

[0071] Specifically, the basic operator in the vocoder model deployed on the NPU can be a convolution operator. The receptive field is calculated based on the convolution kernel size and padding length. Then, the overlapping portion is calculated and analyzed by accumulating the receptive field. For example, if at least 14 frames are needed to analyze the overlapping portion, a fixed-length block of 140 frames can be extracted sequentially from the streaming speech using a fixed sliding block of 140 frames. The sliding step size for each slide can be a fixed 100-frame step size, with 20 frames overlapping on each side. For example... Figure 5 As shown, the sliding window can be of a fixed length (e.g., 140 frames), and the sliding window covers the valid portion in the middle of the window (e.g., 100 frames) and the overlapping portions at both ends of the window (e.g., 20 frames). In addition, the output of the vocoder model deployed on the NPU is segmented based on each input feature to extract only the audio information corresponding to the valid portion.

[0072] When processing input features, the vocoder model deployed on the NPU performs calculations in fixed-length blocks (e.g., 140 frames) each time, and padding is performed when the length is less than the fixed length. For example, when calculating in fixed-length blocks of 140 frames, if the input feature length is less than 140 frames, the input block will be padded with a fixed value of 0. For example, if the original input is 1x80x100, it needs to be padded with a 1x80x40 matrix of all zero values, and then concatenated in the last dimension.

[0073] Furthermore, cosine signals can be used for fade-in and fade-out processing of audio information. Specifically, cosine signals have a slope that gradually increases and then decreases within half a cycle, exhibiting advantages such as a smaller slope and smoother operation near 0 and near 1. For example, if the original audio signal is Pulse Code Modulation (PCM) data x[i], it can be faded in and out using the cosine function f(x) = 0.5 + 0.5 × cos(2 × π × x - 1) / window_len / 2. Specifically, the first half of the cosine function cycle can be used for fade-out processing, and the second half for fade-in processing. For example, assuming the period of the cosine function is 4000, and the corresponding processing window size is 4000, then the cosine function f(x) = 0.5 + 0.5 × cos(2 × π × x - 1) / 4000 / 2. See details. Figure 7 As shown, Figure 7The diagram shows a cosine function with a period of 4000, exhibiting a slope that gradually increases and then decreases (the slope is smaller and smoother near 0 and near 1). The first half of the period corresponds to [0, 2000], which can be faded out using a function segment of [0, 2000]. The second half of the period corresponds to [2000, 4000], which can be faded in using a function segment of [2000, 4000]. Specifically, this involves performing a Hadamard product operation on the corresponding two vectors (i.e., multiplying the cosine function by the corresponding position of the data x[i]).

[0074] Based on the characteristic that the slope of a cosine signal increases and then decreases again within half a cycle, the original signal amplitude is gradually increased from 0 to 1 to achieve a fade-in effect, and decreased from 1 to 0 to achieve a fade-out effect. This can achieve a smooth signal transition to reduce noise, thereby eliminating or reducing noise problems caused by errors in low-precision NPU calculations before and after the audio signal.

[0075] Compared to traditional CPU-based embedded TTS implementations, the solution of this invention can support larger neural network vocoder models and improve the audio quality of speech synthesis. At the same model size, the real-time rate is reduced from 0.96 in the CPU version to 0.09, and the latency is reduced from 675ms in the CPU version to 63ms.

[0076] Exemplary device

[0077] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 7 Products related to the NPU-based streaming speech synthesizer vocoder, which are exemplary embodiments of the present invention, are described below.

[0078] Figure 7 A schematic block diagram of a device 700 according to an embodiment of the present invention is shown. Figure 7 As shown, device 700 may include an NPU 701 and a memory 702. The memory 702 stores computer instructions for an NPU-based streaming speech synthesizer vocoder. When these computer instructions are executed by the NPU 701, the device 700 performs actions according to the preceding description. Figures 2 to 4 The described method. For example, in some embodiments, device 700 can perform tasks such as acquiring fixed-length input features of streaming speech, processing the fixed-length input features, and determining the synthesis result. Based on this, device 700 can effectively deploy a vocoder model on an NPU to perform streaming speech synthesis, combining the high performance of the NPU to ensure high-quality speech synthesis while effectively reducing latency and real-time rate to meet actual market demands.

[0079] In some implementation scenarios, device 700 may include devices with an NPU and support for speech synthesis (such as mobile phones, tablets, computers, or servers). The present invention does not limit the structural design that device 700 can have.

[0080] It should be noted that although several devices or sub-devices of an NPU-based streaming speech synthesizer have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0081] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0082] The use of the verbs "including" and "contains" and their inflections in the application documents does not preclude the existence of elements or steps other than those described in the application documents. The article "a" or "one" preceding an element does not preclude the existence of multiple such elements.

[0083] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A method for a streaming speech synthesis vocoder based on an embedded neural network processor (NPU), characterized in that, include: Obtaining input features to be processed from the streaming speech, wherein the input features to be processed are of fixed length, specifically, preprocessing the streaming speech to split the input features to be processed into one or more input features of fixed length, wherein the preprocessing of the streaming speech includes extracting fixed-length input features from the streaming speech using a sliding window method, the sliding window method including: Input features of a fixed length are extracted sequentially from the streaming speech using a sliding window of the fixed length. The sliding step size of each slide is less than the fixed length, so that there is overlap between the input features extracted by adjacent sliding windows. The sliding window covers the effective part located in the middle of the window and the overlapping part located at both ends of the window. The size of the effective part is equal to the sliding step size. The size of the overlapping part is determined according to the receptive field of the vocoder model structure on the NPU. In response to the existence of input features with a length less than the fixed length, the input features with a length less than the fixed length are padded to adjust their length to the fixed length; Based on the vocoder model deployed on the NPU, fixed-length input features are processed to output audio information; and The output of the vocoder model based on each input feature is segmented to obtain the audio information corresponding to the effective part of each input feature; The audio information corresponding to the effective portion of each input feature is concatenated; and The synthesis result is determined based on the audio information obtained by splicing.

2. The method according to claim 1, characterized in that, Also includes: The spliced ​​audio information is directly used as the synthesis result; or The synthesized result is obtained by applying a cosine signal to the spliced ​​audio information through fade-in and fade-out post-processing.

3. The method according to claim 2, characterized in that, Post-processing of the spliced ​​audio information using cosine signals, including fade-in and fade-out, includes: The audio information within the first half-cycle window of the cosine signal is faded out using the first half-cycle of the cosine signal. The audio information within the second half-cycle window of the cosine signal is faded in using the second half-cycle of the cosine signal; and The fade-in and fade-out processes are used to perform noise reduction on the spliced ​​audio information.

4. The method according to claim 1, characterized in that, The padding process for input features whose length is less than the fixed length includes: The length of the input feature is filled to the fixed length using a fixed value.

5. The method according to any one of claims 1-4, characterized in that, The vocoder model deployed on the NPU is converted from a pre-trained vocoder model that supports streaming speech synthesis.

6. The method according to claim 5, characterized in that, The vocoder model supporting streaming speech synthesis is converted into a vocoder model deployed on the NPU via the NPU toolchain.

7. A device, characterized in that, include: Embedded Neural Processing Unit (NPU); as well as A memory storing computer instructions for an NPU-based streaming speech synthesizer vocoder, which, when executed by the NPU, cause the device to perform the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The program instructions include an NPU-based streaming speech synthesizer vocoder, which, when executed by the NPU, causes the method according to any one of claims 1-6 to be implemented.

Images