Low-resource based vocoder speech synthesis processing method and device and related equipment

Abstract

The application provides a low-resource-based vocoder speech synthesis processing method and device and related equipment. In order to solve the problem of low efficiency of vocoder speech synthesis in the low-resource situation in the prior art, the T-frame acoustic features are obtained, the first target features corresponding to the features including but not limited to the up-down correlation between the T-frame acoustic features are determined, the first target features are convolved based on the respective corresponding channels, the features including but not limited to the correlation between the different channels of the acoustic features contained in the first target features are determined to obtain the second target features, the sampling points corresponding to the acoustic features of each frame of speech signal are determined according to the second target features to obtain the target sampling points, and finally the target sampling points are reconstructed into a speech waveform to obtain the target speech. The efficiency of the vocoder speech synthesis in the low-resource situation can be improved while ensuring the speech synthesis effect. The technical scheme can be applied to the field of financial science and technology.

Description

Technical Field

[0001] This application relates to the fields of speech processing and financial technology, and in particular to a low-resource vocoder-based speech synthesis processing method, apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] The vocoder module in speech synthesis functions by generating corresponding speech waveforms from acoustic features to obtain the target speech. Traditional vocoder models include, but are not limited to, Hifi-GAN and Mel-GAN vocoders. To prioritize the quality of the synthesized speech, they typically employ an upsampling structure in the neural network corresponding to the vocoder and gradually reconstruct the speech waveform based on the inverse short-time fourier transform (iSTFT). In speech synthesis, upsampling refers to increasing the sampling rate of the speech signal from a lower frequency to a higher frequency, while iSTFT is the process of resynthesizing the spectral signal into a time-domain signal. The aforementioned vocoder model has been proven to effectively learn the relationship between acoustic features and speech waveforms, generating speech quality comparable to that of humans. For example, in the fintech field, such as the banking, securities, and insurance industries, an increasing number of business scenarios are using self-service voice service devices based on intelligent robots to provide related services. These all involve using vocoders to generate corresponding service voices to interact with users and provide services.

[0003] However, the superposition of the above-mentioned vocoder model structure based on upsampling and inverse STFT will affect the inference efficiency of the vocoder model. Typically, the inference efficiency on the CPU can only achieve a 1:1 ratio of generated speech length to time consumption, that is, generating 1 minute of speech length requires 1 minute of inference. Such inference efficiency will significantly affect the speech generation efficiency of the vocoder model in some low-resource scenarios, because the hardware resources corresponding to inference, including but not limited to the CPU, have low performance, and other business data processing is also required at the same time. Here, low resources refer to low hardware resources with low power consumption and relatively low hardware performance, including but not limited to smart wearable devices, such as low-performance processor resources corresponding to low power consumption, such as single-core CPU resources.

[0004] Therefore, how to efficiently synthesize speech using a vocoder under low-resource conditions has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a vocoder-based speech synthesis processing method, apparatus, computer device, and computer-readable storage medium based on low resources, which can solve the technical problem of low efficiency of vocoder speech synthesis under low resource conditions in traditional technology.

[0006] In a first aspect, this application provides a low-resource vocoder-based speech synthesis processing method, comprising: acquiring acoustic features of T frames, and determining features including the vertical correlation between the acoustic features of the T frames based on a preset one-dimensional convolutional block included in a preset vocoder model to obtain a first target feature, wherein T is a natural number; convolving the first target feature based on its corresponding channels using a preset depth residual convolutional module included in the preset vocoder model, and then determining the features included in the first target feature to obtain a second target feature, wherein the second target feature includes the correlation between different channels of the acoustic feature corresponding to the first target feature; determining the sampling points corresponding to the acoustic features of each frame of speech signal based on a preset linear layer block and according to the second target feature to obtain target sampling points; and reconstructing the target sampling points into a speech waveform to obtain target speech.

[0007] Secondly, this application provides a low-resource vocoder-based speech synthesis processing apparatus, comprising: a first determining unit, configured to acquire acoustic features of T frames, and based on a preset one-dimensional convolutional block included in a preset vocoder model, determine features including the vertical correlation between the acoustic features of the T frames to obtain a first target feature, wherein T is a natural number; a second determining unit, configured to convolve the first target feature based on its corresponding channels using a preset depth residual convolutional module included in the preset vocoder model, and then determine the features included in the first target feature to obtain a second target feature, wherein the second target feature includes the correlation between different channels of the acoustic feature corresponding to the first target feature; a third determining unit, configured to determine the sampling points corresponding to the acoustic features of each frame of speech signal based on a preset linear layer block and according to the second target feature, to obtain target sampling points; and a reconstruction unit, configured to reconstruct the target sampling points into a speech waveform to obtain target speech.

[0008] Thirdly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the low-resource vocoder-based speech synthesis processing method.

[0009] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the low-resource-based vocoder speech synthesis processing method.

[0010] This application provides a low-resource vocoder-based speech synthesis processing method, apparatus, computer device, and computer-readable storage medium. The method acquires T-frame acoustic features and determines, but is not limited to, features corresponding to the vertical correlation between the T-frame acoustic features, to increase the receptive field of a preset vocoder model. Based on a preset depth residual convolution module included in the preset vocoder model, the first target features are convolved based on their respective channels. Then, the features contained in the first target features are determined to obtain second target features. The second target features contain the correlation between different channels of the acoustic features corresponding to the first target features, further increasing the receptive field of the preset vocoder model. Finally, based on preset linear blocks and according to the second target features, the receptive field of the vocoder model is determined... The sampling points corresponding to the acoustic features of each frame of speech signal are determined to obtain the target sampling points. Finally, the target sampling points are reconstructed into a speech waveform to obtain the target speech. While ensuring that the input and output are consistent each time, the number of parameters of the preset vocoder model is reduced. At the same time, a linear layer is used to replace ISTFT. Compared with the traditional vocoder model structure based on upsampling and ISTFT, not only is the amount of computation reduced by reducing the number of parameters of the preset vocoder model, but the inference efficiency of the preset vocoder model is also greatly accelerated. At the same time, the overall model structure of the vocoder is simpler and can be applied to various low-resource hardware computing platforms, including but not limited to CPUs. It can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating the low-resource-based vocoder speech synthesis processing method provided in this application embodiment;

[0013] Figure 2 A schematic diagram of the framework flow of the low-resource vocoder-based speech synthesis processing method provided in the embodiments of this application;

[0014] Figure 3 A schematic diagram of the first sub-process of the low-resource vocoder-based speech synthesis processing method provided in the embodiments of this application;

[0015] Figure 4 A schematic diagram of the second sub-process of the low-resource vocoder-based speech synthesis processing method provided in the embodiments of this application;

[0016] Figure 5 A schematic block diagram of a low-resource vocoder-based speech synthesis processing apparatus provided for embodiments of this application;

[0017] Figure 6 A schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] This application provides a low-resource vocoder-based speech synthesis processing method. The method can be applied to computer devices including but not limited to smartphones, tablets, laptops, desktop computers, and self-service voice equipment, and is used in low-resource vocoder-based speech synthesis processing in fields including but not limited to fintech.

[0021] To address the technical problem of low efficiency in vocoder speech synthesis under low-resource conditions in traditional technologies, the inventors propose a low-resource vocoder speech synthesis processing method according to the embodiments of this application. The core idea of ​​this application embodiment is: based on low-resource hardware conditions, a deep residual convolution module based on deep separable convolution and 1*1 point convolutional layers is designed. While ensuring that the input and output sizes are consistent each time, the model parameters are reduced. At the same time, a linear layer is used to replace ISTFT. Compared with the vocoder model structure based on upsampling and ISTFT, the inference efficiency of the vocoder model is greatly accelerated. At the same time, the overall model structure of the vocoder is simpler and can be applied to various low-resource hardware computing platforms, including but not limited to CPUs. It can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect.

[0022] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating the low-resource-based vocoder-based speech synthesis processing method provided in an embodiment of this application. Figure 2 This is a schematic diagram illustrating the framework of a low-resource vocoder-based speech synthesis processing method provided in an embodiment of this application. Figure 1 and Figure 2 As shown, this method is applied to vocoders, including but not limited to, speech synthesis, and includes, but is not limited to, the following steps S11-S14:

[0024] S11. Obtain the acoustic features of frame T, and based on the preset one-dimensional convolutional block contained in the preset vocoder model, determine the features including the vertical correlation between the acoustic features of frame T, and obtain the first target feature, where T is a natural number.

[0025] Explained, acoustic features refer to physical quantities that represent the acoustic properties of speech, and are also a general term for the acoustic performance of various sound elements. Acoustic features include the energy concentration area, formant frequency, formant intensity and bandwidth representing timbre, as well as the duration, fundamental frequency, average speech power and other features representing the prosodic characteristics of speech. Acoustic features include, but are not limited to, MFCC features (Mel Frequency Cepstrum Coefficient) and spectrograms. Acoustic features in speech synthesis are derived from speech features extracted from the other end's voice during a call, and speech features generated from text in speech generation from text.

[0026] A pre-defined one-dimensional convolutional block, also known as a preset one-dimensional convolutional block, represents a neural network block based on one-dimensional convolution. The preset one-dimensional convolutional block includes, but is not limited to, a preset one-dimensional convolutional layer based on one-dimensional convolution. One-dimensional convolution is used for convolution of sequential data. It is a sliding window operation that performs a dot product operation between the data in the window and the convolution kernel to obtain the convolution result.

[0027] In speech synthesis, based on the output of the upstream module of the vocoder, T-frame acoustic features are obtained. The T-frame acoustic features are frequency domain signals, which describe the frequency characteristics of a signal. The frequency domain signal is a frequency diagram that describes the frequency structure of the signal and the relationship between the frequency and the amplitude of the signal at that frequency. The T-frame acoustic features are then convolved one-dimensionally using a preset one-dimensional convolution block contained in the preset vocoder model to determine the features including the contextual correlation between the T-frame acoustic features, thus obtaining the first target feature. Here, T is a natural number. The purpose of convolving the T-frame acoustic features one-dimensionally is to correlate the T-frame acoustic features with the context, thereby increasing the receptive field of the preset vocoder model. In a convolutional neural network (CNN), the receptive field is the region in the input space that affects a specific unit of the network.

[0028] For example, in the field of financial technology, including but not limited to the banking, securities, and insurance industries, in scenarios where self-service voice service devices based on intelligent robots are used to provide self-service voice services for related businesses, when it is necessary to generate voice based on preset corresponding business text to provide voice services to users, several frames of acoustic features corresponding to the business text are obtained, and based on the preset one-dimensional convolutional blocks contained in the preset vocoder model, features including the vertical correlation between the acoustic features of several frames are extracted to obtain the first target feature. Then, the business voice is generated based on the acoustic features corresponding to the business text and broadcast to the user to provide self-service voice services.

[0029] S12. Based on the preset depth residual convolution module included in the preset vocoder model, the first target feature is convolved based on its corresponding channel, and then the features included in the first target feature are determined to obtain the second target feature, wherein the second target feature includes the correlation between different channels of the acoustic feature corresponding to the first target feature.

[0030] Explained, a preset vocoder model includes a deep residual convolution module, i.e., a preset deep residual convolution module. The preset deep residual convolution module represents a neural network module containing several preset deep residual convolution blocks based on residual connections. The preset deep residual convolution block represents several neural network layers that perform, but are not limited to, depthwise separable convolution and 1*1 point convolution. The preset deep residual convolution block includes, but is not limited to, depthwise separable convolution layers and 1*1 point convolution layers. Among them, the depthwise separable convolution layer represents a neural network layer based on depthwise separable convolution. Depthwise separable convolution is also known as depthwise separable convolution. Depthwise separable convolution is a variant of convolution that decomposes standard convolution into two separate convolution operations: depthwise convolution and pointwise convolution. Depthwise convolution performs a separate convolution operation on each input channel, while pointwise convolution performs a dot product operation on the result of depthwise convolution. A 1*1 point convolutional layer represents a neural network layer based on 1*1 point convolution. 1*1 point convolution is a convolution operation with a kernel size of 1*1. This convolution operation does not change the spatial dimensions (width and height) of the input features, but it can change the number of channels.

[0031] Based on the preset vocoder model obtained from the above concept, which includes a preset depth residual convolution module, the first target features are subjected to convolution operations, including but not limited to depthwise separable convolution, based on their respective corresponding channels. The features of each channel of the acoustic features corresponding to the first target features are determined, as well as the correlation features between different channels of the acoustic features corresponding to the first target features, to obtain the second target features. Here, depthwise separable convolution can reduce the number of parameters and computational cost of convolution operations, improve the efficiency of the preset vocoder model, and is therefore particularly suitable for low-resource hardware conditions. Furthermore, the 1*1 convolution operation can fully extract the correlation features between different channels contained in the first target features without changing the channel structure of the T-frame acoustic features corresponding to the first target features. Thus, while retaining the channel structure of the T-frame acoustic features corresponding to the first target features, the receptive field of the preset vocoder model is further improved. In summary, the concept and configuration of the preset depth residual convolution module can maximize the receptive field of the preset vocoder model while reducing the number of parameters and computational load of convolution operations. This allows for improved vocoder speech synthesis efficiency under low-resource conditions on various hardware computing platforms, including but not limited to CPUs, while maintaining speech synthesis quality. Compared to the traditional technique of superimposing upsampling and inverse STFT vocoder model structures, this approach not only ensures speech synthesis quality but also improves efficiency under low-resource conditions. The preset vocoder model structure is simple yet sophisticated, making it particularly suitable for vocoder-based speech synthesis under low-resource conditions.

[0032] In acoustic features, "channels" refer to the way sound frequency ranges are divided into multiple parts for individual processing during acoustic processing. Each channel can be independently amplified, compressed, denoised, convolved, etc. The more channels there are, the more refined the sound processing, resulting in clearer, more natural, and more comfortable sound. Channels refer to how many parts of the sound frequency range you can hear are divided into for individual processing. Each part can be amplified, compressed, and denoised independently. The more channels there are, the more refined the sound processing, and the clearer and more natural the sound you hear.

[0033] S13. Based on the preset linear block and according to the second target feature, determine the sampling points corresponding to the acoustic features of each frame of speech signal to obtain the target sampling points.

[0034] Explained, a pre-set linear layer block, also known as a preset linear layer block, refers to a neural network block that performs linear transformations on the input data based on several linear layers. The preset linear layer block includes several linear layers. Among them, the linear layer (Linear Layer), also known as a fully connected layer or a dense layer, plays a role in the neural network mainly by mapping the input data to a new space through linear transformations, changing the dimension of the data, and thus obtaining data with the desired dimension.

[0035] According to the above settings, after obtaining the second target feature, the second target feature is transformed linearly based on a preset linear block to map the second target feature to a new data space, and the sampling points corresponding to the acoustic features of each frame of speech signal are determined to obtain target sampling points. The obtained target sampling points are time-domain signals. Here, time-domain signals refer to the representation of signals as they change over time. In the time domain, the signal is regarded as a function that changes with time. Time-domain signals focus on the values ​​of the signal at different time points and how these values ​​change over time. Time-domain signals are relative to frequency-domain signals. Frequency-domain signals refer to the representation of signals on the frequency axis. In the frequency domain, the signal is decomposed into combinations of different frequency components.

[0036] S14. Reconstruct the target sampling points into a speech waveform to obtain the target speech.

[0037] Explained, since several target sampling points are time-domain signals, based on this, functions including but not limited to Reshape are used to reconstruct the target sampling points into a speech waveform to obtain the target speech. The Reshape function is used to rearrange the array corresponding to several target sampling points into a new shape without changing its data, and returns an array with unchanged data but changed dimensions, thereby obtaining the speech waveform.

[0038] In this embodiment, by acquiring T-frame acoustic features and determining, based on a preset one-dimensional convolutional block included in a preset vocoder model, a first target feature corresponding to the features of vertical correlation between T-frame acoustic features is obtained, thereby increasing the receptive field of the preset vocoder model. Then, based on a preset depth residual convolutional module included in the preset vocoder model, the first target feature is convolved based on its corresponding channel. Next, features corresponding to the correlation between different channels of the first target feature (not limited to acoustic features) are determined to obtain a second target feature, further increasing the receptive field of the preset vocoder model. Finally, based on a preset linear layer block and according to the second target feature, the acoustic features corresponding to each frame of speech signal are determined. The system obtains target sampling points by sampling points and then reconstructs the target sampling points into a speech waveform to obtain the target speech. While ensuring that the input and output are consistent each time, the system reduces the number of parameters in the preset vocoder model and uses a linear layer to replace ISTFT. Compared with the traditional vocoder model structure based on upsampling and ISTFT, this not only reduces the amount of computation by reducing the number of parameters in the preset vocoder model, but also greatly accelerates the inference efficiency of the preset vocoder model. At the same time, it makes the overall model structure of the vocoder simpler and can be applied to various low-resource hardware computing platforms, including but not limited to CPUs. It can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect.

[0039] In one embodiment, based on a preset one-dimensional convolutional block included in a preset vocoder model, features including the vertical correlation between the acoustic features of the T frames are determined to obtain a first target feature, including:

[0040] Based on the preset one-dimensional convolutional layer contained in the preset one-dimensional convolutional block, the features including the vertical correlation between the acoustic features of the T-frame are determined to obtain the initial first feature;

[0041] Based on the preset first regularization layer contained in the preset one-dimensional convolutional block, the initial first feature is regularized to obtain the first target feature.

[0042] For an explanatory explanation, please refer to [link / reference]. Figure 2 ,like Figure 2As shown, based on the aforementioned preset one-dimensional convolutional block settings, a one-dimensional convolutional layer is pre-set within the preset one-dimensional convolutional block, i.e., a preset one-dimensional convolutional layer. This preset one-dimensional convolutional layer represents a neural network layer performing one-dimensional convolution. Furthermore, a first normalization layer is pre-set within the preset one-dimensional convolutional block, i.e., a preset first normalization layer. This preset first normalization layer represents a neural network layer that performs layer normalization on the output of the preset one-dimensional convolutional layer. Layer normalization, also known as layer normalization, normalizes the input of neurons to stabilize the input of each layer. It should be noted that the "first" in the preset first normalization layer is only used to distinguish the neural network layers corresponding to different layer normalizations, and is not used to limit the neural network layers corresponding to different layer normalizations.

[0043] Based on the above concept and setup, based on the preset one-dimensional convolutional layer contained in the preset one-dimensional convolutional block, features including the vertical correlation between acoustic features of T frames are first extracted and determined to obtain the initial first feature. Then, based on the preset first regularization layer contained in the preset one-dimensional convolutional block, the initial first feature is regularized to obtain the first target feature, thereby mitigating gradient explosion and gradient vanishing during the training process of the preset vocoder model, and helping the preset vocoder model converge faster and more stably.

[0044] In this embodiment, an initial first feature corresponding to the features of the vertical correlation between acoustic features of T frames is determined based on a preset one-dimensional convolutional layer. The initial first feature is then layer-normalized based on a preset first normalization layer to obtain a first target feature. This not only increases the receptive field of the preset vocoder model, but also prevents gradient explosion and gradient vanishing during the operation of the preset vocoder model. Thus, while ensuring the speech synthesis effect, the efficiency of vocoder speech synthesis under low resource conditions is improved.

[0045] In one embodiment, based on the preset depth residual convolution module included in the preset vocoder model, the first target features are convolved based on their respective corresponding channels, and then the features included in the first target features are determined to obtain the second target features, including:

[0046] Based on residual connections, N layers of residual convolutional blocks with preset depths are connected to obtain a residual convolutional module with preset depth, where N is a natural number;

[0047] Based on the N layers of preset depth residual convolutional blocks, the first target feature is convolved based on its corresponding channel, and the features of each channel of the acoustic feature corresponding to the first target feature are determined. The correlation features between different channels of the acoustic feature corresponding to the first target feature are also determined to obtain the second target feature.

[0048] For an explanatory explanation, please refer to [link / reference]. Figure 2 ,like Figure 2 As shown, a pre-set depth residual convolutional block is defined as a pre-set depth residual convolutional block. The pre-set depth residual convolutional block represents several neural network layers that perform, but are not limited to, depthwise separable convolution and 1*1 point convolution. Residual connection refers to the connection method of the deep neural network by introducing direct connections across some pre-set depth residual convolutional blocks to optimize the connection method of the deep neural network.

[0049] Based on residual connections, N layers of preset depth residual convolutional blocks are connected to form a preset depth residual convolutional module, where N is a natural number.

[0050] Based on the above concept and setup, the first target feature is convolved based on its corresponding channel by the N layers of preset depth residual convolution blocks contained in the preset depth residual convolution module. Then, the features of each channel of the acoustic feature corresponding to the first target feature are determined, and the correlation features between different channels of the acoustic feature corresponding to the first target feature are determined to obtain the second target feature.

[0051] In this embodiment, by setting up a preset depth residual convolution module corresponding to an N-layer preset depth residual convolution block based on residual connections, the first target feature is convolved and the feature is extracted to obtain the second target feature. By using the superposition of N layers of preset depth residual convolution blocks, the number of input and output channels can be kept consistent each time. This multi-layer superposition structure effectively expands the receptive field of the entire preset vocoder model, and at the same time enables the preset vocoder model to fully learn the relationship between acoustic features and speech signals. This can improve the efficiency of vocoder speech synthesis under low resource conditions while ensuring the speech synthesis effect.

[0052] In one embodiment, based on the N layers of the preset depth residual convolutional blocks, the first target feature is convolved based on its corresponding channel, and then the features of each channel of the acoustic feature corresponding to the first target feature are determined, and the correlation features between different channels of the acoustic feature corresponding to the first target feature are determined, to obtain the second target feature, including:

[0053] Based on the preset depth separation convolution block contained in the preset depth residual convolution block, the first target feature is subjected to depth convolution to obtain depth convolution feature;

[0054] Based on the preset ordinary convolutional blocks contained in the preset depth residual convolutional block, the depth convolutional features are convolved to obtain the second target features.

[0055] Explained, a preset depthwise convolutional block is defined as a pre-defined depthwise convolutional block containing a depthwise separable convolutional block. This pre-defined depthwise convolutional block represents a neural network block based on depthwise convolution. Depthwise convolution, also known as depthwise separable convolution, splits the convolution corresponding to the first target feature in speech synthesis into depthwise convolution and pointwise convolution. The two independent steps of depthwise convolution improve the efficiency and accuracy of the vocoder model in speech synthesis. This differs from the application scenarios of depthwise separable convolution, which is usually used in image processing and computer vision tasks. Depthwise convolution is performed channel-wise, meaning that each input channel is independently applied to a convolution kernel. This means that if there are M input channels, then M convolution kernels will be used, where M is a natural number. Each convolution kernel is only responsible for the convolution operation of one channel, which can significantly reduce the number of parameters and computational cost. Depthwise convolution usually uses smaller convolution kernels to effectively extract local features of the corresponding first target features in speech synthesis. Pointwise convolution, after depthwise convolution, uses a 1*1 convolution kernel to convolve the outputs of all channels. It is also called pointwise convolution, which mixes and combines the acoustic features of different channels to generate new features of speech features. Pointwise convolution uses a 1*1 convolution kernel to linearly combine the outputs of depthwise convolution in the channel dimension while keeping the feature size unchanged, thereby enhancing the expressive power of the model.

[0056] The preset depth residual convolutional block contains regular convolutional blocks, which are called preset regular convolutional blocks. Preset regular convolutional blocks represent neural network blocks based on regular convolutions. Regular convolution (or standard convolution) is used to extract useful features from the input depthwise convolutional features. Regular convolutions include, but are not limited to, 1x1 point convolutions and 1x2 point convolutions. In summary, the preset depth residual convolutional block, based on the preset depth separation convolutional block and the preset regular convolutional block, is a neural network block that combines reduced model parameters with improved efficiency. Therefore, the preset depth residual convolutional module is also a neural network module that combines reduced model parameters with improved efficiency.

[0057] Based on the above concept and setup, the first target feature is first depthwise convolved with the preset depthwise separation convolution block contained in the preset depthwise residual convolution block to obtain the corresponding depthwise convolution feature. Then, the depthwise convolution feature is convolved with the preset ordinary convolution block contained in the preset depthwise residual convolution block to extract the useful features contained in the depthwise convolution and obtain the second target feature.

[0058] In this embodiment, the second target feature is obtained by performing depthwise convolution on the first target feature based on a preset depthwise separation convolution block, and then convolution on the depthwise convolution feature based on a preset ordinary convolution block. This not only significantly reduces the number of parameters and computational load, but also effectively strengthens the correlation between different channels of the acoustic features corresponding to the first target feature, thereby mixing and combining the acoustic features of different channels to generate corresponding new features. This further increases the receptive field of the preset vocoder model, thereby reducing the computational load by reducing the number of parameters in the preset vocoder model, which greatly accelerates the inference efficiency of the vocoder model. At the same time, it also makes the overall model structure of the vocoder simpler and applicable to various low-resource hardware computing platforms, including but not limited to CPUs. It can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect.

[0059] In one embodiment, based on the preset depth separation convolution block included in the preset depth residual convolution module, the first target feature is depthwise convolved to obtain depthwise convolution features, including:

[0060] Based on the preset depth separation convolutional layer contained in the preset depth separation convolutional block, the first target feature is convolved according to its corresponding channel to obtain the initial depth convolutional feature;

[0061] Based on the preset second regularization layer contained in the preset depth separation convolution block, the initial depth convolution features are regularized to obtain depth convolution features.

[0062] For an explanatory explanation, please refer to [link / reference]. Figure 2 ,like Figure 2 As shown, the preset depth separation convolution block includes a depth separation convolution layer, i.e., a preset depth separation convolution layer, which represents a neural network layer performing depth separation convolution. Furthermore, the preset depth separation convolution block includes a neural network layer performing layer normalization, i.e., a preset second normalization layer. The preset second normalization layer is similar to the preset first normalization layer and can be referred to in the context of the preset first normalization layer; it will not be repeated here. It should be noted that the "second" in the preset second normalization layer and the "first" in the preset first normalization layer are only used to distinguish different neural network layers performing layer normalization and are not used to limit the neural network layers performing layer normalization. Other similar descriptions in this application embodiment follow the same principle. Therefore, the preset depth separation convolution block based on the preset depth separation convolution layer is a neural network block that combines reducing model parameters and improving efficiency. Thus, the preset depth residual convolution module based on the preset depth separation convolution block and the preset ordinary convolution block is a neural network module that combines reducing model parameters and improving efficiency.

[0063] Based on the above concept and setup, the first target features are first convolved according to their respective channels using the preset depth separation convolutional layers contained in the preset depth separation convolutional block to obtain initial depth convolutional features. Then, the initial depth convolutional features are layer-normalized using the preset second normalization layer contained in the preset depth separation convolutional block to obtain depth convolutional features.

[0064] In this embodiment, by performing deep convolution and layer regularization on the first target features, not only can the number of parameters and computational load be significantly reduced, but gradient explosion and gradient vanishing can also be prevented during the operation of the preset vocoder model, thereby increasing the corresponding receptive field of the preset vocoder model. This can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect.

[0065] In one embodiment, please refer to Figure 3 , Figure 3 This is a schematic diagram of the first sub-process of the low-resource vocoder-based speech synthesis processing method provided in an embodiment of this application. Figure 3 As shown, in this embodiment, based on the preset ordinary convolutional blocks included in the preset depth residual convolutional block, the depth convolutional features are convolved to obtain the second target features, including:

[0066] S31. Based on the preset first 1*1 point convolutional layer contained in the preset ordinary convolutional block, the depth convolutional features are convolved to obtain the initial 1*1 convolutional features.

[0067] S32. Based on the preset GELU activation layer contained in the preset ordinary convolutional block, the initial 1*1 convolutional features are nonlinearly transformed to obtain activation features;

[0068] S33. Based on the preset second 1*1 point convolutional layer contained in the preset ordinary convolutional block, the activation feature is convolved to obtain a second target feature with the same output dimension as the initial 1*1 convolutional feature.

[0069] For an explanatory explanation, please refer to [link / reference]. Figure 2 ,like Figure 2As shown, a preset ordinary convolutional block includes a first 1*1 point convolutional layer, i.e., a preset first 1*1 point convolutional layer. This preset first 1*1 point convolutional layer represents the first neural network layer based on 1*1 point convolutions. The 1*1 point convolution is described above and will not be repeated here. The preset ordinary convolutional block also includes a GELU activation layer, i.e., a preset GELU activation layer. This preset GELU activation layer represents a neural network layer based on the GELU activation function. The derivative of the GELU activation function is continuous, which makes gradient propagation easier during training of the preset ordinary convolutional block. Its smooth activation curve allows the preset ordinary convolutional block to better capture [the target vector] when processing speech synthesis tasks. By capturing relevant features, the performance of the preset ordinary convolutional block is improved. A second 1*1 point convolutional layer is pre-set within the preset ordinary convolutional block, i.e., the preset second 1*1 point convolutional layer. The preset second 1*1 point convolutional layer represents the second neural network layer based on 1*1 point convolution. The 1*1 point convolution is as described above and will not be repeated here. It should be noted that the "first" in the preset first 1*1 point convolutional layer and the "second" in the preset second 1*1 point convolutional layer are only used to distinguish the neural network layers corresponding to different 1*1 point convolutional layers, and are not used to limit the neural network layers corresponding to different 1*1 point convolutional layers. Other similar descriptions in the embodiments of this application are analogous. Therefore, the preset ordinary convolutional block containing two 1*1 point convolutional layers is a neural network block that further expands the receptive field of the preset vocoder model based on expanding the receptive field of the preset ordinary convolutional block. Thus, the preset depth residual convolutional module based on the preset depth separation convolutional block and the preset ordinary convolutional block is a neural network module that combines reducing model parameters and improving efficiency.

[0070] Based on the above concept and setup, the depthwise convolution features are first convolved using a preset first 1*1 point convolutional layer contained in a preset ordinary convolutional block to obtain initial 1*1 convolutional features. Since 1*1 point convolution significantly reduces the number of parameters and computational cost compared to other convolution operations, and is beneficial for improving the model's generalization and expressive capabilities, this approach, while reducing the number of parameters and computational cost of convolution operations to adapt to low-resource hardware conditions, fully extracts the correlation features between different channels contained in the depthwise convolution features. This improves the speech synthesis effect without changing the channel structure of the depthwise convolution features, ensuring consistency between input and output. Then, based on the preset ordinary convolutional block... The block contains a preset GELU activation layer that performs a non-linear transformation on the initial 1*1 convolutional features to obtain activation features. This allows for better capture of relevant features when processing speech synthesis tasks, improving the performance of the preset ordinary convolutional block. Finally, based on the preset second 1*1 point convolutional layer contained in the preset ordinary convolutional block, the activation features are convolved to obtain a second target feature with the same output dimension as the initial 1*1 convolutional features. This ensures that the output of the preset GELU activation layer returns to the same output dimension as the initial 1*1 convolutional features, preventing data loss through the preset GELU activation layer and ensuring good speech synthesis results.

[0071] In this embodiment, by employing a 1*1 point convolutional layer to convolve the depthwise convolutional features, it can not only adapt to situations where the input acoustic features are of arbitrary size under low resource conditions, but also ensure that the spatial structure corresponding to the spatial location of the acoustic features is not destroyed, thereby ensuring the consistency between the input and output of the preset vocoder model. Furthermore, by setting a preset second 1*1 point convolutional layer after the preset GELU activation layer, it can prevent data loss and errors during the nonlinear transformation of the initial 1*1 convolutional features based on the preset GELU activation layer. By expanding the receptive field of the preset vocoder model through the 1*1 point convolutional layer, it ensures that the data returns to the consistent output corresponding to the initial 1*1 convolutional features, thereby ensuring that the data does not produce errors. This can improve the efficiency of vocoder speech synthesis under low resource conditions while ensuring the speech synthesis effect.

[0072] In one embodiment, please refer to Figure 4 , Figure 4 This is a schematic diagram of the second sub-process of the low-resource vocoder-based speech synthesis processing method provided in an embodiment of this application. Figure 4 As shown, in this embodiment, based on a preset linear block and according to the second target feature, the sampling points corresponding to the acoustic features of each frame of speech signal are determined to obtain the target sampling points, including:

[0073] S41. Based on the preset first linear layer contained in the preset linear layer block, the second target feature is subjected to the first linear transformation to obtain the corresponding linear transformation feature after dimensionality increase.

[0074] S42. Based on the preset second linear layer contained in the preset linear layer block, the linear transformation features are subjected to a second linear transformation and sampled to determine the sampling points corresponding to the acoustic features of each frame of speech signal, thereby obtaining the target sampling points.

[0075] For an explanatory explanation, please refer to [link / reference]. Figure 2 ,like Figure 2 As shown, a first linear layer is pre-set in the preset linear layer block, i.e., the preset first linear layer. The preset first linear layer represents the first neural network layer based on a linear layer contained in the preset linear layer block. At the same time, a second linear layer is pre-set in the preset linear layer block, i.e., the preset second linear layer. The preset second linear layer represents the second neural network layer based on a linear layer contained in the preset linear layer block. By setting the preset second linear layer, it is ensured that the output of the preset second linear layer is consistent with the input of the preset first linear layer in terms of dimension. The linear layers are as described above and will not be repeated here. It should be noted that the "first" in the preset first linear layer and the "second" in the preset second linear layer are only used to distinguish different linear layers and are not used to limit different linear layers. Other similar descriptions in the embodiments of this application are analogous.

[0076] Based on the above concept and setup, the second target feature is subjected to a first linear transformation based on the first linear layer contained in the preset linear layer block to obtain the linear transformation feature corresponding to the dimensionality upgrade. Then, based on the second linear layer contained in the preset linear layer block, the linear transformation feature is subjected to a second linear transformation and sampled to determine the sampling point corresponding to the acoustic feature of each frame of speech signal, thus obtaining the target sampling point.

[0077] For example, the second target features extracted from the preset depth residual convolution module contained in the preset vocoder model are processed through a preset linear block. The preset linear block replaces the inverse Fourier transform (ISTFT) in the traditional technique, transforming the acoustic features of T frames into a speech signal of length 256*T. 256 represents the hop_size for extracting acoustic features from the speech signal, indicating that the acoustic features of one frame need to recover 256 sampling points of the speech signal. Here, hop_size refers to the number of sample points that each frame of data moves in the time series during speech signal processing. It is the number of samples that are staggered between two adjacent windows, which determines the degree of overlap between frames. The size of hop_size affects the segmentation and processing of audio data, and is usually determined by both frame length and frame shift.

[0078] In this embodiment, a preset first linear layer and a preset second linear layer are used to linearly transform the second target features and determine the sampling points corresponding to the acoustic features of each frame of speech signal. This replaces the inverse Fourier transform (ISTFT) in the vocoder in traditional technology. This not only ensures the consistency between the input and output of the linear layer but also significantly reduces the computational load, thereby improving the inference efficiency of the preset vocoder model. It also simplifies the overall model structure of the vocoder, making it suitable for various low-resource hardware computing platforms, including but not limited to CPUs. This improves the efficiency of vocoder speech synthesis under low-resource conditions while maintaining the speech synthesis effect.

[0079] It should be noted that the low-resource vocoder-based speech synthesis processing methods described in the above embodiments can be recombined as needed to obtain combined implementation schemes, but all are within the protection scope claimed in this application.

[0080] Please see Figure 5 , Figure 5 This is a schematic block diagram of a low-resource vocoder-based speech synthesis processing apparatus provided in an embodiment of this application. Corresponding to the above-described low-resource vocoder-based speech synthesis processing method, this application also provides a low-resource vocoder-based speech synthesis processing apparatus. Figure 5 As shown, the resource-low vocoder-based speech synthesis processing apparatus includes a unit for performing the aforementioned resource-low vocoder-based speech synthesis processing method. This resource-low vocoder-based speech synthesis processing apparatus can be configured in a computer device. Specifically, please refer to... Figure 5 The low-resource vocoder-based speech synthesis processing device 50 includes a first determining unit 51, a second determining unit 52, a third determining unit 53, and a reconstruction unit 54.

[0081] The first determining unit 51 is used to acquire the acoustic features of T frames and, based on the preset one-dimensional convolutional block contained in the preset vocoder model, determine the features including the vertical correlation between the acoustic features of T frames to obtain the first target feature, where T is a natural number.

[0082] The second determining unit 52 is used to perform convolution on the first target feature based on its corresponding channel based on the preset depth residual convolution module included in the preset vocoder model, and then determine the features included in the first target feature to obtain the second target feature, wherein the second target feature includes the correlation between different channels of the acoustic feature corresponding to the first target feature.

[0083] The third determining unit 53 is used to determine the sampling points corresponding to the acoustic features of each frame of speech signal based on a preset linear block and according to the second target features, so as to obtain the target sampling points.

[0084] The reconstruction unit 54 is used to reconstruct the target sampling points into a speech waveform to obtain the target speech.

[0085] In one embodiment, the first determining unit 51 includes:

[0086] The first determining subunit is used to determine the features including the vertical correlation between the acoustic features of the T-frame based on the preset one-dimensional convolutional layer contained in the preset one-dimensional convolutional block, and to obtain the initial first feature;

[0087] The first regularization subunit is used to perform layer regularization on the initial first feature based on the preset first regularization layer contained in the preset one-dimensional convolutional block to obtain the first target feature.

[0088] In one embodiment, the second determining unit 52 includes:

[0089] The connection subunit is used to connect N layers of residual convolutional blocks with a preset depth based on residual connections to obtain a residual convolutional module with a preset depth, where N is a natural number;

[0090] The first convolutional subunit is used to convolve the first target feature based on its corresponding channel based on the N layers of preset depth residual convolutional blocks, then determine the features of each channel of the acoustic feature corresponding to the first target feature, and determine the correlation features between different channels of the acoustic feature corresponding to the first target feature, to obtain the second target feature.

[0091] In one embodiment, the first convolutional subunit includes:

[0092] A depthwise convolutional subunit is used to perform depthwise convolution on the first target feature based on the preset depthwise separation convolutional block contained in the preset depthwise residual convolutional block, to obtain depthwise convolutional features;

[0093] A regular convolutional subunit is used to convolve the depth convolutional features based on the preset regular convolutional blocks contained in the preset depth residual convolutional block to obtain the second target features.

[0094] In one embodiment, the depthwise convolutional subunit includes:

[0095] The second convolutional subunit is used to convolve the first target features according to their respective corresponding channels based on the preset depth separation convolutional layers contained in the preset depth separation convolutional block to obtain the initial depth convolutional features.

[0096] The second regularization subunit is used to perform layer regularization on the initial depth convolution features based on the preset second regularization layer contained in the preset depth separation convolution block to obtain depth convolution features.

[0097] In one embodiment, the ordinary convolutional subunit includes:

[0098] The third convolutional subunit is used to convolve the depth convolutional features based on the preset first 1*1 point convolutional layer contained in the preset ordinary convolutional block to obtain the initial 1*1 convolutional features.

[0099] The activation subunit is used to perform a nonlinear transformation on the initial 1*1 convolutional features based on the preset GELU activation layer contained in the preset ordinary convolutional block to obtain activation features;

[0100] The fourth convolutional subunit is used to convolve the activation features based on the preset second 1*1 point convolutional layer contained in the preset ordinary convolutional block, so as to obtain a second target feature with the same output dimension as the initial 1*1 convolutional feature.

[0101] In one embodiment, the third determining unit 53 includes:

[0102] The transformation subunit is used to perform a first linear transformation on the second target feature based on the preset first linear layer contained in the preset linear layer block, so as to obtain the corresponding linear transformation feature after dimensionality increase.

[0103] The sampling subunit is used to perform a second linear transformation on the linear transformation features based on the preset second linear layer contained in the preset linear layer block and to sample them, thereby determining the sampling points corresponding to the acoustic features of each frame of speech signal and obtaining the target sampling points.

[0104] It should be noted that those skilled in the art will clearly understand that the specific implementation process of the above-mentioned low-resource vocoder-based speech synthesis processing device and its various units can be referred to the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these details will not be repeated here.

[0105] Meanwhile, the division and connection methods of the various units in the above-mentioned low-resource vocoder speech synthesis processing device are only for illustrative purposes. In other embodiments, the low-resource vocoder speech synthesis processing device can be divided into different units as needed, and the various units in the low-resource vocoder speech synthesis processing device can be connected in different orders and methods to complete all or part of the functions of the above-mentioned low-resource vocoder speech synthesis processing device.

[0106] The aforementioned low-resource-based vocoder speech synthesis processing device can be implemented as a computer program, which can, for example... Figure 6 It runs on the computer device shown.

[0107] Please see Figure 6 , Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a desktop computer or a server, or it can be a component or part of other devices.

[0108] See Figure 6 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504, or it may be a volatile storage medium.

[0109] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to perform one of the aforementioned low-resource-based vocoder speech synthesis processing methods.

[0110] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0111] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned low-resource vocoder speech synthesis processing method.

[0112] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. For example, in some embodiments, the computer device may include only a memory and a processor. In such embodiments, the structure and function of the memory and processor are different from those shown in the figures. Figure 6 The embodiments shown are consistent and will not be described again here.

[0113] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps: acquiring acoustic features of T frames, and determining features including the vertical correlation between the acoustic features of T frames based on a preset one-dimensional convolutional block included in a preset vocoder model, to obtain a first target feature, where T is a natural number; convolving the first target feature based on its corresponding channel based on a preset depth residual convolutional module included in the preset vocoder model, and then determining the features included in the first target feature to obtain a second target feature, where the second target feature includes the correlation between different channels of the acoustic feature corresponding to the first target feature; determining the sampling points corresponding to the acoustic features of each frame of speech signal based on a preset linear layer block and according to the second target feature, to obtain target sampling points; and reconstructing the target sampling points into a speech waveform to obtain target speech.

[0114] In one embodiment, when the processor 502 determines the features including the vertical correlation between the acoustic features of T frames based on the preset one-dimensional convolutional blocks contained in the preset vocoder model to obtain the first target feature, it specifically implements the following steps:

[0115] Based on the preset one-dimensional convolutional layer contained in the preset one-dimensional convolutional block, the features including the vertical correlation between the acoustic features of the T-frame are determined to obtain the initial first feature;

[0116] Based on the preset first regularization layer contained in the preset one-dimensional convolutional block, the initial first feature is regularized to obtain the first target feature.

[0117] In one embodiment, when the processor 502 implements the following steps to obtain the second target feature by performing convolution on the first target feature based on its corresponding channel using the preset depth residual convolution module contained in the preset vocoder model:

[0118] Based on residual connections, N layers of residual convolutional blocks with preset depths are connected to obtain a residual convolutional module with preset depth, where N is a natural number;

[0119] Based on the N layers of preset depth residual convolutional blocks, the first target feature is convolved based on its corresponding channel, and the features of each channel of the acoustic feature corresponding to the first target feature are determined. The correlation features between different channels of the acoustic feature corresponding to the first target feature are also determined to obtain the second target feature.

[0120] In one embodiment, when the processor 502 implements the following steps to obtain the second target feature by convolving the first target feature based on its corresponding channels using N layers of preset depth residual convolutional blocks, determining the features of each channel of the acoustic feature corresponding to the first target feature, and determining the correlation features between different channels of the acoustic feature corresponding to the first target feature:

[0121] Based on the preset depth separation convolution block contained in the preset depth residual convolution block, the first target feature is subjected to depth convolution to obtain depth convolution feature;

[0122] Based on the preset ordinary convolutional blocks contained in the preset depth residual convolutional block, the depth convolutional features are convolved to obtain the second target features.

[0123] In one embodiment, when the processor 502 performs depthwise convolution on the first target feature based on the preset depth separation convolution block included in the preset depth residual convolution module to obtain depthwise convolution features, it specifically implements the following steps:

[0124] Based on the preset depth separation convolutional layer contained in the preset depth separation convolutional block, the first target feature is convolved according to its corresponding channel to obtain the initial depth convolutional feature;

[0125] Based on the preset second regularization layer contained in the preset depth separation convolution block, the initial depth convolution features are regularized to obtain depth convolution features.

[0126] In one embodiment, when the processor 502 performs convolution on the depthwise convolution features based on the preset ordinary convolution blocks contained in the preset depthwise residual convolution block to obtain the second target features, it specifically implements the following steps:

[0127] Based on the preset first 1*1 point convolutional layer contained in the preset ordinary convolutional block, the depth convolutional features are convolved to obtain the initial 1*1 convolutional features.

[0128] Based on the preset GELU activation layer contained in the preset ordinary convolutional block, the initial 1*1 convolutional features are nonlinearly transformed to obtain activation features;

[0129] Based on the preset second 1*1 point convolutional layer contained in the preset ordinary convolutional block, the activation features are convolved to obtain a second target feature with the same output dimension as the initial 1*1 convolutional feature.

[0130] In one embodiment, when the processor 502 determines the sampling points corresponding to the acoustic features of each frame of speech signal based on a preset linear block and according to the second target feature, and obtains the target sampling points, it specifically implements the following steps:

[0131] Based on the preset first linear layer contained in the preset linear layer block, the second target feature is subjected to the first linear transformation to obtain the corresponding linear transformation feature after dimensionality increase;

[0132] Based on the preset second linear layer contained in the preset linear layer block, the linear transformation features are subjected to a second linear transformation and sampled to determine the sampling points corresponding to the acoustic features of each frame of speech signal, thereby obtaining the target sampling points.

[0133] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0134] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0135] Therefore, this application also provides a computer-readable storage medium. This computer-readable storage medium can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, storing a computer program that, when executed by a processor, causes the processor to perform the following steps:

[0136] A computer program product, when run on a computer, causes the computer to perform the steps of the low-resource-based vocoder speech synthesis processing method described in the above embodiments.

[0137] The computer-readable storage medium can be an internal storage unit of the aforementioned device, such as the device's hard drive or memory. The computer-readable storage medium can also be an external storage device of the device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the device.

[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0139] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing computer programs.

[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0142] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0144] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A low-resource vocoder-based speech synthesis processing method, characterized in that, include: Acquire the acoustic features of T frames, and based on the preset one-dimensional convolutional block contained in the preset vocoder model, determine the features including the vertical correlation between the acoustic features of T frames to obtain the first target feature, where T is a natural number; Based on the preset depth residual convolution module included in the preset vocoder model, the first target feature is convolved based on its corresponding channel, and then the features included in the first target feature are determined to obtain the second target feature, wherein the second target feature includes the correlation between different channels of the acoustic feature corresponding to the first target feature. Based on a preset linear block and according to the second target feature, the sampling points corresponding to the acoustic features of each frame of speech signal are determined to obtain the target sampling points; The target sampling points are reconstructed into a speech waveform to obtain the target speech.

2. The low-resource vocoder-based speech synthesis processing method according to claim 1, characterized in that, Based on the preset one-dimensional convolutional blocks contained in the preset vocoder model, features including the vertical correlation between the acoustic features of the T frames are determined to obtain the first target features, including: Based on the preset one-dimensional convolutional layer contained in the preset one-dimensional convolutional block, the features including the vertical correlation between the acoustic features of the T-frame are determined to obtain the initial first feature; Based on the preset first regularization layer contained in the preset one-dimensional convolutional block, the initial first feature is regularized to obtain the first target feature.

3. The low-resource vocoder-based speech synthesis processing method according to claim 1 or 2, characterized in that, Based on the preset depth residual convolution module included in the preset vocoder model, the first target features are convolved based on their respective channels, and then the features contained in the first target features are determined to obtain the second target features, including: Based on residual connections, N layers of residual convolutional blocks with preset depths are connected to obtain a residual convolutional module with preset depth, where N is a natural number; Based on the N layers of preset depth residual convolutional blocks, the first target feature is convolved based on its corresponding channel, and the features of each channel of the acoustic feature corresponding to the first target feature are determined. The correlation features between different channels of the acoustic feature corresponding to the first target feature are also determined to obtain the second target feature.

4. The low-resource vocoder-based speech synthesis processing method according to claim 3, characterized in that, Based on the N layers of preset depth residual convolutional blocks, the first target features are convolved based on their respective channels. Then, the features of each channel of the acoustic features corresponding to the first target features are determined, and the correlation features between different channels of the acoustic features corresponding to the first target features are determined to obtain the second target features, including: Based on the preset depth separation convolution block contained in the preset depth residual convolution block, the first target feature is subjected to depth convolution to obtain depth convolution feature; Based on the preset ordinary convolutional blocks contained in the preset depth residual convolutional block, the depth convolutional features are convolved to obtain the second target features.

5. The low-resource vocoder-based speech synthesis processing method according to claim 4, characterized in that, Based on the preset depth separation convolution block included in the preset depth residual convolution module, the first target feature is depthwise convolved to obtain depthwise convolution features, including: Based on the preset depth separation convolutional layer contained in the preset depth separation convolutional block, the first target feature is convolved according to its corresponding channel to obtain the initial depth convolutional feature; Based on the preset second regularization layer contained in the preset depth separation convolution block, the initial depth convolution features are regularized to obtain depth convolution features.

6. The low-resource vocoder-based speech synthesis processing method according to claim 4, characterized in that, Based on the preset ordinary convolutional blocks contained in the preset depth residual convolutional block, the depth convolutional features are convolved to obtain the second target features, including: Based on the preset first 1*1 point convolutional layer contained in the preset ordinary convolutional block, the depth convolutional features are convolved to obtain the initial 1*1 convolutional features. Based on the preset GELU activation layer contained in the preset ordinary convolutional block, the initial 1*1 convolutional features are nonlinearly transformed to obtain activation features; Based on the preset second 1*1 point convolutional layer contained in the preset ordinary convolutional block, the activation features are convolved to obtain a second target feature with the same output dimension as the initial 1*1 convolutional feature.

7. The low-resource vocoder-based speech synthesis processing method according to claim 1 or 2, characterized in that, Based on a preset linear block and according to the second target feature, the sampling points corresponding to the acoustic features of each frame of speech signal are determined to obtain the target sampling points, including: Based on the preset first linear layer contained in the preset linear layer block, the second target feature is subjected to the first linear transformation to obtain the corresponding linear transformation feature after dimensionality increase; Based on the preset second linear layer contained in the preset linear layer block, the linear transformation features are subjected to a second linear transformation and sampled to determine the sampling points corresponding to the acoustic features of each frame of speech signal, thereby obtaining the target sampling points.

8. A low-resource vocoder-based speech synthesis processing device, characterized in that, include: The first determining unit is used to acquire the acoustic features of T frames and, based on the preset one-dimensional convolutional block contained in the preset vocoder model, determine the features including the vertical correlation between the acoustic features of T frames to obtain the first target feature, where T is a natural number. The second determining unit is used to perform convolution on the first target feature based on its corresponding channel based on the preset depth residual convolution module included in the preset vocoder model, and then determine the features included in the first target feature to obtain the second target feature, wherein the second target feature includes the correlation between different channels of the acoustic feature corresponding to the first target feature. The third determining unit is used to determine the sampling points corresponding to the acoustic features of each frame of speech signal based on a preset linear block and according to the second target features, so as to obtain the target sampling points. The reconstruction unit is used to reconstruct the target sampling points into a speech waveform to obtain the target speech.

9. A computer device, characterized in that, The computer device includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to perform the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the steps of the method as described in any one of claims 1-7.

Images