Low-resource-based vocoder speech synthesis processing method and device and related equipment

Abstract

The invention provides a vocoder speech synthesis processing method and device based on low resources and related equipment, and aims to solve the problem of low speech synthesis efficiency of a vocoder under the condition of low resources in the prior art. The method comprises the following steps: acquiring T frames of acoustic features, determining first target features corresponding to features including but not limited to vertical correlation between the T frames of acoustic features, performing convolution on the first target features based on respective corresponding channels, and determining features including but not limited to correlation between different channels of the acoustic features included in the first target features to obtain second target features; and determining a sampling point corresponding to the acoustic feature of each frame of voice signal according to the second target feature to obtain a target sampling point, and finally reconstructing the target sampling point into a voice waveform to obtain a target voice, so that the voice synthesis effect can be ensured, the voice synthesis efficiency of the vocoder under the condition of low resources can be improved, and the user experience can be improved. The technical scheme can be applied to the technical field of financial science and technology.

Description

Technical Field

[0001] The present application relates to the fields of speech processing and financial technology, and in particular to a low-resource based vocoder speech synthesis processing method, apparatus, computer equipment and computer-readable storage medium. Background Art

[0002] The vocoder module in speech synthesis is used to generate the corresponding speech waveform from the acoustic features to obtain the target speech. The vocoder models in the traditional technology include but are not limited to Hifi-gan, Mel-gan and other vocoders. In order to give priority to the quality of synthesized speech, the speech waveform is gradually restored based on the inverse STFT (Inverse Short-Time Fourier Transform, iSTFT for short) through the upsampling structure of the neural network corresponding to the vocoder. In speech synthesis, upsampling refers to the process of increasing the sampling rate of the speech signal from a lower frequency to a higher frequency, and the inverse STFT is the process of resynthesizing the spectrum signal into a time domain signal. The above-mentioned vocoder model has been proven that this vocoder model structure based on upsampling and inverse STFT can well learn the relationship between acoustic features and speech waveforms, and the generated speech quality is comparable to that of humans. For example, in the field of financial technology, such as banking, securities, and insurance industries, more and more business scenarios use self-service voice service devices based on intelligent robots to provide related business services, all of which involve using vocoders to generate corresponding service voices to interact with users and provide services to users.

[0003] However, the superposition of the above-mentioned vocoder model structure based on upsampling and inverse STFT will affect the inference efficiency of the vocoder model. Usually, the inference efficiency on the CPU can only achieve a ratio of generated speech length and time consumption of 1:1, that is, it takes 1 minute to infer to generate 1 minute of speech length. Such inference efficiency, in some low-resource occasions, because the hardware resources corresponding to the inference, including but not limited to the CPU, have low performance themselves, and other business data need to be processed at the same time, resulting in low resources that greatly affect the speech generation efficiency of the vocoder model. Among them, low resources refer to low hardware resources with low power consumption and relatively low hardware performance corresponding to smart wearable devices, such as low processor resources with low performance corresponding to low power consumption such as single-core CPU resources.

[0004] Therefore, how to efficiently perform vocoder speech synthesis under low resource conditions has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a low-resource based vocoder speech synthesis processing method, apparatus, computer equipment and computer-readable storage medium, which can solve the technical problem of low efficiency of vocoder speech synthesis under low resource conditions in traditional technologies.

[0006] In a first aspect, the present application provides a low-resource based vocoder speech synthesis processing method, comprising: obtaining T-frame acoustic features, and based on a preset one-dimensional convolution block contained in a preset vocoder model, determining features including up and down correlations between the acoustic features of the T frames, to obtain a first target feature, wherein T is a natural number; based on a preset deep residual convolution module contained in the preset vocoder model, convolving the first target features based on their respective corresponding channels, and then determining features included in the first target features, to obtain a second target feature, wherein the second target feature includes correlations between different channels of the acoustic features corresponding to the first target features; based on a preset linear layer block and according to the second target feature, determining the sampling points corresponding to the acoustic features of each frame of speech signal, to obtain target sampling points; reconstructing the target sampling points into a speech waveform to obtain a target speech.

[0007] In a second aspect, the present application provides a low-resource based vocoder speech synthesis processing device, comprising: a first determination unit, used to obtain T-frame acoustic features, and based on a preset one-dimensional convolution block contained in a preset vocoder model, determine the features including the up-down correlation between the acoustic features of the T-frames, to obtain a first target feature, wherein T is a natural number; a second determination unit, used to convolve the first target features based on the channels corresponding to each of them based on a preset deep residual convolution module contained in the preset vocoder model, and then determine the features included in the first target features, to obtain a second target feature, wherein the second target feature includes the correlation between different channels of the acoustic features corresponding to the first target feature; a third determination unit, used to determine the sampling points corresponding to the acoustic features of each frame of speech signal based on a preset linear layer block and according to the second target feature, to obtain a target sampling point; a reconstruction unit, used to reconstruct the target sampling point into a speech waveform to obtain a target speech.

[0008] In a third aspect, the present application provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the low-resource-based vocoder speech synthesis processing method when executing the computer program.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the low-resource-based vocoder speech synthesis processing method.

[0010] The present application provides a low-resource based vocoder speech synthesis processing method, apparatus, computer equipment and computer-readable storage medium. The method increases the receptive field of a preset vocoder model by obtaining T-frame acoustic features and determining a first target feature corresponding to features including but not limited to up-and-down correlations between T-frame acoustic features, and based on a preset deep residual convolution module included in the preset vocoder model, convolves the first target feature based on the channels corresponding to each of them, and then determines the features included in the first target feature to obtain a second target feature, wherein the second target feature includes the correlation between different channels of the acoustic features corresponding to the first target feature, further increasing the receptive field of the preset vocoder model, and then determines based on a preset linear layer block and according to the second target feature. The sampling points corresponding to the acoustic features of each frame of speech signal are determined to obtain the target sampling points, and finally the target sampling points are reconstructed into a speech waveform to obtain the target speech. While ensuring that the size of each input and output is consistent, the number of parameters of the preset vocoder model is reduced, and a linear layer is used to replace ISTFT. Compared with the traditional vocoder model structure based on upsampling and ISTFT, not only the amount of calculation is reduced by reducing the number of parameters of the preset vocoder model, but also the reasoning efficiency of the preset vocoder model is greatly accelerated, and the overall model structure of the vocoder is also made simpler, which can be applied to various low-resource hardware computing platforms including but not limited to CPUs, and can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A flowchart of a low-resource vocoder speech synthesis processing method provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of a framework flow of a low-resource vocoder speech synthesis processing method provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of a first sub-flow of a low-resource based vocoder speech synthesis processing method provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of a second sub-flow of the low-resource-based vocoder speech synthesis processing method provided in an embodiment of the present application;

[0016] Figure 5 A schematic block diagram of a low-resource vocoder speech synthesis processing device provided in an embodiment of the present application;

[0017] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0020] The embodiment of the present application provides a low-resource based vocoder speech synthesis processing method, which can be applied to computer devices including but not limited to smart phones, tablet computers, laptop computers, desktop computers, self-service voice service devices, and the like, and can be used when performing low-resource based vocoder speech synthesis processing in the field of financial technology including but not limited to.

[0021] In the face of the technical problem of low efficiency of vocoder speech synthesis under low resource conditions in traditional technology, the inventor proposes a low-resource vocoder speech synthesis processing method according to the embodiment of the present application. The core idea of ​​the embodiment of the present application is: according to the hardware conditions of low resources, a deep residual convolution module based on deep separation convolution and 1*1 point convolution layer is designed. While ensuring that the size of each input and output is consistent, the model parameters are reduced, and a linear layer is used to replace ISTFT. Compared with the vocoder model structure based on upsampling and ISTFT, the reasoning efficiency of the vocoder model is greatly accelerated, and the overall model structure of the vocoder is also simpler. It can be applicable to various low-resource hardware computing platforms including but not limited to CPUs, and can improve the efficiency of vocoder speech synthesis under low resource conditions while ensuring the speech synthesis effect.

[0022] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0023] See also Figure 1 and Figure 2 , Figure 1 A flowchart of a low-resource vocoder speech synthesis processing method provided in an embodiment of the present application is provided. Figure 2 The following is a schematic diagram of the framework flow of the low-resource vocoder speech synthesis processing method provided in the embodiment of the present application. Figure 1 and Figure 2 As shown, the method is applied to, including but not limited to, a vocoder in speech synthesis, and the method includes but is not limited to the following steps S11-S14:

[0024] S11. Acquire acoustic features of T frames, and determine features including up-and-down correlations between the acoustic features of the T frames based on a preset one-dimensional convolution block included in a preset vocoder model to obtain a first target feature, where T is a natural number.

[0025] Explanatory terms, acoustic features refer to physical quantities that represent the acoustic characteristics of speech, and are also a general term for the acoustic manifestations of various sound elements. Acoustic features include energy concentration areas, resonance peak frequency, resonance peak intensity and bandwidth that represent timbre, as well as duration, fundamental frequency, average voice power and other features that represent the rhythmic characteristics of speech. Acoustic features include but are not limited to MFCC features (Mel Frequency Cepstrum Coefficient) and spectrograms. In speech synthesis, acoustic features are derived from, but are not limited to, speech features extracted from the sound of the other end of the call, and speech features generated from text in text-based speech generation.

[0026] A one-dimensional convolution block is pre-set, that is, a preset one-dimensional convolution block. The preset one-dimensional convolution block represents a neural network block based on one-dimensional convolution. The preset one-dimensional convolution block includes but is not limited to a preset one-dimensional convolution layer based on one-dimensional convolution. One-dimensional convolution is a convolution for sequence data, which is a sliding window operation. The data in the window is dot-convolved with the convolution kernel to obtain a convolution result.

[0027] In speech synthesis, based on the output of the upstream module of the vocoder, T-frame acoustic features are obtained. The T-frame acoustic features are frequency domain signals. The frequency domain signals refer to signals that describe the characteristics of the signal in terms of frequency. The frequency domain signals are frequency graphs that describe the frequency structure of the signal and the relationship between the frequency and the amplitude of the frequency signal. The T-frame acoustic features are one-dimensionally convolved using a preset one-dimensional convolution block included in a preset vocoder model, thereby determining features including upper and lower correlations between the T-frame acoustic features to obtain a first target feature, wherein T is a natural number, and the purpose of performing one-dimensional convolution on the T-frame acoustic features is to correlate the T-frame acoustic features with the context, thereby increasing the receptive field of the preset vocoder model, wherein in a convolutional neural network (CNN), the receptive field is an area in the input space that affects a specific unit of the network.

[0028] Exemplarily, in the field of financial technology including but not limited to the banking, securities, and insurance industries, in a scenario where self-service voice service equipment based on intelligent robots is used to provide self-service voice services for related businesses, when it is necessary to generate voice according to preset corresponding business texts to provide voice services to users, several frames of acoustic features corresponding to the business texts are obtained, and based on the preset one-dimensional convolution block contained in the preset vocoder model, features including the up and down correlations between several frames of acoustic features are extracted to obtain the first target feature, and then business voice is generated according to the acoustic features corresponding to the business text, and the voice is broadcast to the user to provide self-service voice service.

[0029] S12. Based on the preset deep residual convolution module included in the preset vocoder model, the first target features are convolved based on the channels corresponding to the first target features, and then the features included in the first target features are determined to obtain the second target features, wherein the second target features include the correlation between different channels of the acoustic features corresponding to the first target features.

[0030] Explanatoryly, a preset vocoder model includes a deep residual convolution module, namely a preset deep residual convolution module, which represents a neural network module including several preset deep residual convolution blocks based on residual connection, and the preset deep residual convolution block represents several neural network layers including but not limited to deep separable convolution and 1*1 point convolution, and the preset deep residual convolution block includes but is not limited to deep separable convolution layer and 1*1 point convolution layer, wherein the deep separable convolution layer represents a neural network layer based on deep separable convolution, and the deep separable convolution is also called depthwise separable convolution. Depthwise separable convolution is a variant of convolution, which decomposes the standard convolution into two separate convolution operations: depthwise convolution and pointwise convolution. The depthwise convolution is a separate convolution operation for each input channel, while the pointwise convolution is a dot product operation on the result of the depthwise convolution. The 1*1 point convolution layer represents a neural network layer based on 1*1 point convolution. 1*1 point convolution is a convolution operation with a convolution kernel size of 1*1. This convolution operation does not change the spatial size (width and height) of the input feature, but can change the number of channels.

[0031] According to the preset deep residual convolution module included in the preset vocoder model obtained by the above conception, the first target feature is subjected to a convolution operation including but not limited to a deep separable convolution based on the channels corresponding to each of them, and the characteristics of the channels of the acoustic features corresponding to the first target feature are determined, and the characteristics of the correlation between different channels of the acoustic features corresponding to the first target feature are determined to obtain the second target feature, wherein the deep separable convolution can reduce the number of parameters and the amount of calculation of the convolution operation, improve the efficiency of the preset vocoder model, and is particularly suitable for low-resource hardware conditions, and the 1*1 convolution operation can fully extract the characteristics of the correlation between different channels contained in the first target feature, and will not change the channel structure of the T-frame acoustic features corresponding to the first target feature, thereby further improving the receptive field of the preset vocoder model on the basis of retaining the channel structure of the T-frame acoustic features corresponding to the first target feature. In summary, based on the conception and setting of the preset deep residual convolution module, the receptive field of the preset vocoder model can be improved as much as possible on the basis of reducing the number of parameters and calculations of the convolution operation, so that on various low-resource hardware computing platforms including but not limited to CPU, the efficiency of vocoder synthesis of speech can be improved under low-resource conditions while ensuring the speech synthesis effect. Compared with the superposition of the vocoder model structure based on upsampling and inverse STFT in the traditional technology, it can not only ensure the speech synthesis effect, but also improve the efficiency of vocoder synthesis of speech under low-resource conditions, so that the structure of the preset vocoder model is simple but not minimalist, and is particularly suitable for the situation of synthesizing speech based on vocoder under low-resource conditions.

[0032] Among them, the "channel" in the acoustic characteristics refers to the method of dividing the sound frequency range into multiple parts for separate processing during the acoustic processing process. Each channel can independently perform operations such as amplification, compression, noise reduction, and convolution. The more channels there are, the more refined the sound processing is, making the sound sound clearer, more natural, and more comfortable. Channel refers to the number of parts into which the sound frequency range you can hear is divided for separate processing. Each part can be amplified, compressed, and noise-reduced separately. The more channels there are, the more refined the sound processing is, and the clearer and more natural the sound you hear is.

[0033] S13. Based on the preset linear layer block and according to the second target feature, determine the sampling point corresponding to the acoustic feature of each frame of the speech signal to obtain the target sampling point.

[0034] Explanatoryly, a linear layer block is pre-set, i.e., a preset linear layer block. The preset linear layer block represents a neural network block that performs linear transformation on the input data based on several linear layers. The preset linear layer block includes several linear layers, wherein the linear layer (Linear Layer) is also called the fully connected layer (Fully Connected Layer) or the dense layer (Dense Layer). The role of the linear layer in the neural network is mainly to map the input data to a new space through linear transformation, change the dimension of the data, and thus obtain data of the desired dimension.

[0035] According to the above settings, after obtaining the second target feature, the second target feature is linearly transformed based on the preset linear layer block to map the second target feature to a new data space, and the sampling points corresponding to the acoustic features of each frame of the speech signal are determined to obtain the target sampling points. The obtained target sampling points are time domain signals, wherein the time domain signal refers to a representation method of a signal changing over time. In the time domain, the signal is regarded as a function that changes over time. The time domain signal focuses on the value of the signal at different time points and how these values ​​change over time. The time domain signal is relative to the frequency domain signal. The frequency domain signal refers to the representation method of the signal on the frequency axis. In the frequency domain, the signal is decomposed into a combination of different frequency components.

[0036] S14: reconstruct the target sampling points into a speech waveform to obtain a target speech.

[0037] Explanatory, since several target sampling points are time domain signals, based on this, the target sampling points are reconstructed into a speech waveform using functions including but not limited to the Reshape function to obtain the target speech, wherein the Reshape function is used to rearrange the arrays corresponding to the several target sampling points into a new shape without changing their data, and return an array with unchanged data but changed dimension, so as to obtain a speech waveform.

[0038] In an embodiment of the present application, the acoustic features of T frames are obtained, and based on a preset one-dimensional convolution block contained in a preset vocoder model, a first target feature corresponding to a feature including but not limited to the up-and-down correlation between the acoustic features of the T frames is determined to increase the receptive field of the preset vocoder model, and based on a preset deep residual convolution module contained in the preset vocoder model, the first target feature is convolved based on the channels corresponding to each of them, and then the feature corresponding to the correlation between different channels including but not limited to the acoustic features contained in the first target feature is determined to obtain a second target feature, thereby further increasing the receptive field of the preset vocoder model, and then based on the preset linear layer block and according to the second target feature, the sampling rate corresponding to the acoustic feature of each frame of the speech signal is determined. The target sampling points are obtained, and finally the target sampling points are reconstructed into a speech waveform to obtain the target speech. While ensuring that the size of each input and output is consistent, the parameter amount of the preset vocoder model is reduced, and a linear layer is used to replace ISTFT. Compared with the traditional vocoder model structure based on upsampling and ISTFT, not only the amount of calculation is reduced by reducing the parameter amount of the preset vocoder model, but also the reasoning efficiency of the preset vocoder model is greatly accelerated, and the overall model structure of the vocoder is also simpler, which can be applied to various low-resource hardware computing platforms including but not limited to CPUs, and can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect.

[0039] In one embodiment, based on a preset one-dimensional convolution block included in a preset vocoder model, a feature including up-down correlation between the acoustic features of the T frames is determined to obtain a first target feature, including:

[0040] Based on a preset one-dimensional convolution layer included in the preset one-dimensional convolution block, determine a feature including up-down correlations between the acoustic features of the T frames to obtain an initial first feature;

[0041] Based on the preset first regularization layer included in the preset one-dimensional convolution block, the initial first feature is layer-regularized to obtain a first target feature.

[0042] Explanatory, please continue to see Figure 2 ,like Figure 2As shown, based on the setting of the above-mentioned preset one-dimensional convolution block, the one-dimensional convolution layer contained in the preset one-dimensional convolution block is preset, that is, the preset one-dimensional convolution layer, the preset one-dimensional convolution layer represents a neural network layer that performs one-dimensional convolution, and the first regularization layer contained in the preset one-dimensional convolution block is preset, that is, the preset first regularization layer, the preset first regularization layer represents a neural network layer that performs layer regularization on the output of the preset one-dimensional convolution layer, layer regularization, also known as layer normalization (Layer Normalization), by normalizing the input of neurons to keep the input of each layer stable. It should be noted that the "first" involved in the preset first regularization layer is only used to distinguish the neural network layers corresponding to different layer regularizations, and is not used to limit the neural network layers corresponding to different layer regularizations.

[0043] According to the above conception and arrangement, based on the preset one-dimensional convolution layer contained in the preset one-dimensional convolution block, the features including the up-down correlation between the acoustic features of the T frames are first extracted and determined to obtain the initial first feature, and then based on the preset first regularization layer contained in the preset one-dimensional convolution block, the initial first feature is layer-regularized to obtain the first target feature, thereby mitigating the gradient explosion and gradient vanishing in the training process of the preset vocoder model, so as to help the preset vocoder model converge faster and more stably.

[0044] In the embodiment of the present application, an initial first feature corresponding to features including but not limited to the up-and-down correlations between acoustic features of T frames is determined based on a preset one-dimensional convolutional layer, and the initial first feature is layer-regularized based on a preset first-layer regularization layer to obtain a first target feature. This not only increases the corresponding receptive field of the preset vocoder model, but also prevents gradient explosion and gradient disappearance during the operation of the preset vocoder model, thereby improving the efficiency of vocoder speech synthesis under low resource conditions while ensuring the speech synthesis effect.

[0045] In one embodiment, based on a preset deep residual convolution module included in the preset vocoder model, the first target features are convolved based on the channels corresponding to the first target features, and then the features included in the first target features are determined to obtain the second target features, including:

[0046] Based on residual connection, N layers of preset depth residual convolution blocks are connected to obtain a preset depth residual convolution module, where N is a natural number;

[0047] Based on the N layers of preset depth residual convolution blocks, the first target features are convolved based on their respective corresponding channels, and then the characteristics of the respective channels of the acoustic features corresponding to the first target features are determined, and the characteristics of the correlation between different channels of the acoustic features corresponding to the first target features are determined to obtain the second target features.

[0048] Explanatory, please continue to see Figure 2 ,like Figure 2 As shown, a depth residual convolution block is pre-set, that is, a preset depth residual convolution block, which represents several neural network layers including but not limited to depth separation convolution and 1*1 point convolution, and a residual connection (Residual Connection) represents a connection method of optimizing the deep neural network by introducing a direct connection across some preset depth residual convolution blocks.

[0049] Based on the residual connection, N layers of preset depth residual convolution blocks are connected, indicating that some preset depth residual convolution blocks in the N layers of preset depth residual convolution blocks are residually connected to obtain a preset depth residual convolution module, where N is a natural number.

[0050] According to the above conception and setting, based on the N layers of preset depth residual convolution blocks included in the preset depth residual convolution module, the first target feature is convolved based on the channels corresponding to each other, and then the characteristics of the respective channels of the acoustic features corresponding to the first target feature are determined, and the characteristics of the correlation between different channels of the acoustic features corresponding to the first target feature are determined to obtain the second target feature.

[0051] In an embodiment of the present application, a preset depth residual convolution module corresponding to an N-layer preset depth residual convolution block based on residual connection is set to convolve the first target feature and perform feature extraction to obtain the second target feature. By using the superposition of N layers of preset depth residual convolution blocks, it is possible to ensure that the number of input and output channels is consistent each time. Such a multi-layer superposition structure effectively expands the receptive field of the entire preset vocoder model, and at the same time enables the preset vocoder model to fully learn the relationship between acoustic features and speech signals, and can improve the efficiency of vocoder synthesis of speech under low resource conditions while ensuring the speech synthesis effect.

[0052] In one embodiment, based on the N layers of the preset depth residual convolution blocks, the first target features are convolved based on the channels corresponding to the first target features, and then the characteristics of the channels of the acoustic features corresponding to the first target features are determined, and the characteristics of the correlation between different channels of the acoustic features corresponding to the first target features are determined to obtain the second target features, including:

[0053] Based on the preset depth separation convolution block included in the preset depth residual convolution block, the first target feature is depth-convolved to obtain a depth convolution feature;

[0054] Based on the preset common convolution block included in the preset deep residual convolution block, the deep convolution feature is convolved to obtain a second target feature.

[0055] Explanation: the preset depth residual convolution block contains a depth separation convolution block, that is, a preset depth separation convolution block. The preset depth separation convolution block represents a neural network block based on depth separation convolution. The depth separation convolution is also called depth separable convolution. The depth separable convolution is achieved by splitting the convolution corresponding to the first target feature in speech synthesis into depthwise convolution and pointwise convolution. Convolution) is used to improve the efficiency and accuracy of the vocoder model in speech synthesis. This is different from the application scenarios of deep separable convolution, which is usually used in image processing and computer vision tasks. The convolution operation of deep convolution is performed channel by channel (Depthwise), that is, a convolution kernel is applied independently to each input channel, which means that if there are M input channels, then M convolution kernels will be used, where M is a natural number. Each convolution kernel is only responsible for the convolution operation of one channel, which can significantly reduce the number of parameters and the amount of calculation. Deep convolution usually uses smaller convolution kernels to effectively extract the local features of the first target feature corresponding to speech synthesis; point-by-point convolution, after deep convolution, uses a 1*1 convolution kernel to convolve the outputs of all channels, also known as point convolution, to mix and combine the acoustic features of different channels to generate new features of speech features. Point-by-point convolution uses a 1*1 convolution kernel to linearly combine the outputs of deep convolution in the channel dimension while keeping the feature size unchanged, thereby enhancing the expressiveness of the model.

[0056] The ordinary convolution block contained in the preset depth residual convolution block is pre-set, that is, the preset ordinary convolution block, and the preset ordinary convolution block represents a neural network block based on ordinary convolution, wherein the ordinary convolution (Regular Convolution or Standard Convolution) is used to extract useful features from the input depth convolution features, and the ordinary convolution includes but is not limited to 1*1 point convolution and 1*2 point convolution. In summary, the preset depth residual convolution block based on the preset depth separation convolution block and the preset ordinary convolution block is a neural network block that combines reducing model parameters and improving efficiency. Therefore, the preset depth residual convolution module is also a neural network module that combines reducing model parameters and improving efficiency.

[0057] According to the above conception and setting, the first target feature is firstly deep-convolved based on the preset depth separation convolution block contained in the preset depth residual convolution block to obtain the corresponding depth convolution feature, and then the depth convolution feature is convolved based on the preset ordinary convolution block contained in the preset depth residual convolution block to extract the useful features contained in the depth convolution to obtain the second target feature.

[0058] In the embodiment of the present application, by performing a deep convolution on the first target feature based on a preset deep separation convolution block, and then convolving the deep convolution feature based on a preset common convolution block to obtain a second target feature, not only can the number of parameters and the amount of calculation be significantly reduced, but also the acoustic features of different channels can be mixed and combined by effectively strengthening the correlation between different channels of the acoustic features corresponding to the first target feature to generate corresponding new features, so as to further increase the receptive field of the preset vocoder model, and reduce the amount of calculation by reducing the number of parameters of the preset vocoder model, so as to greatly accelerate the reasoning efficiency of the vocoder model, and also make the overall model structure of the vocoder simpler, which can be applicable to various low-resource hardware computing platforms including but not limited to CPUs, and can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect.

[0059] In one embodiment, based on a preset depth separation convolution block included in the preset depth residual convolution module, the first target feature is subjected to depth convolution to obtain a depth convolution feature, including:

[0060] Based on the preset depth separation convolution layer included in the preset depth separation convolution block, the first target features are convolved according to the channels corresponding to the first target features to obtain initial depth convolution features;

[0061] Based on the preset second regularization layer included in the preset depth separation convolution block, the initial depth convolution feature is layer-regularized to obtain the depth convolution feature.

[0062] Explanatory, please continue to see Figure 2 ,like Figure 2 As shown, the depth separation convolution layer included in the preset depth separation convolution block is preset, that is, the preset depth separation convolution layer. The preset depth separation convolution layer represents a neural network layer for performing depth separation convolution, and the neural network layer for performing layer regularization included in the preset depth separation convolution block is preset, that is, the preset second regularization layer. The preset second regularization layer is similar to the preset first regularization layer, and can refer to the preset first regularization layer. It will not be repeated here. It should be noted that the "second" involved in the preset second regularization layer and the "first" involved in the preset first regularization layer are only used to distinguish different neural network layers for layer regularization, and are not used to limit the neural network layers for layer regularization. Other similar descriptions in the embodiments of the present application are analogous. Therefore, the preset depth separation convolution block based on the preset depth separation convolution layer is a neural network block that combines reducing model parameters and improving efficiency. Therefore, the preset depth residual convolution module based on the preset depth separation convolution block and the preset ordinary convolution block is a neural network module that combines reducing model parameters and improving efficiency.

[0063] According to the above conception and setting, the first target features are first convolved according to the corresponding channels based on the preset depth separation convolution layer contained in the preset depth separation convolution block to obtain the initial depth convolution features, and then the initial depth convolution features are layer-regularized based on the preset second regularization layer contained in the preset depth separation convolution block to obtain the depth convolution features.

[0064] In the embodiment of the present application, by performing deep convolution on the first target feature and performing layer regularization, not only can the number of parameters and the amount of calculation be significantly reduced, but also the gradient explosion and gradient vanishing during the operation of the preset vocoder model can be prevented, so that the corresponding receptive field of the preset vocoder model is increased, and the efficiency of vocoder speech synthesis under low resource conditions can be improved while ensuring the speech synthesis effect.

[0065] In one embodiment, see Figure 3 , Figure 3 This is a schematic diagram of the first sub-flow of the low-resource vocoder speech synthesis processing method provided in the embodiment of the present application. Figure 3 As shown, in this embodiment, based on the preset common convolution block included in the preset depth residual convolution block, the depth convolution feature is convolved to obtain the second target feature, including:

[0066] S31, based on the preset first 1*1 point convolution layer included in the preset common convolution block, convolve the depth convolution feature to obtain an initial 1*1 convolution feature;

[0067] S32, based on the preset GELU activation layer included in the preset common convolution block, performing a nonlinear transformation on the initial 1*1 convolution feature to obtain an activation feature;

[0068] S33. Based on the preset second 1*1 point convolution layer included in the preset ordinary convolution block, the activation feature is convolved to obtain a second target feature consistent with the output dimension corresponding to the initial 1*1 convolution feature.

[0069] Explanatory, please continue to see Figure 2 ,like Figure 2As shown, the first 1*1 point convolution layer contained in the preset ordinary convolution block is preset, that is, the first 1*1 point convolution layer is preset. The preset first 1*1 point convolution layer represents the first neural network layer based on the 1*1 point convolution. The 1*1 point convolution is as described above and will not be repeated here. The GELU activation layer contained in the preset ordinary convolution block is preset, that is, the preset GELU activation layer. The preset GELU activation layer represents a neural network layer based on the GELU activation function. The derivative of the GELU activation function is continuous, which makes it easier to propagate gradients when training the preset ordinary convolution block. Its smooth activation curve enables the preset ordinary convolution block to better capture the speech when processing speech synthesis tasks. Capture the corresponding features and improve the performance of the preset ordinary convolution block; pre-set the second 1*1 point convolution layer contained in the preset ordinary convolution block, that is, preset the second 1*1 point convolution layer, the preset second 1*1 point convolution layer represents the second neural network layer based on the 1*1 point convolution, the 1*1 point convolution is as described above, and will not be repeated here. It should be noted that the "first" involved in the preset first 1*1 point convolution layer and the "second" involved in the preset second 1*1 point convolution layer are only used to distinguish the neural network layers corresponding to different 1*1 point convolution layers, and are not used to limit the neural network layers corresponding to different 1*1 point convolution layers. The other similar descriptions in the embodiments of the present application are analogous. Therefore, the preset ordinary convolution block containing two 1*1 point convolution layers is a neural network block that further expands the receptive field of the preset vocoder model on the basis of expanding the receptive field of the preset ordinary convolution block. Therefore, the preset depth residual convolution module based on the preset depth separation convolution block and the preset ordinary convolution block is a neural network module that combines reducing model parameters and improving efficiency.

[0070] According to the above conception and setting, the deep convolution feature is first convolved based on the preset first 1*1 point convolution layer contained in the preset ordinary convolution block to obtain the initial 1*1 convolution feature. Compared with other convolution operations, 1*1 point convolution can significantly reduce the number of parameters and the amount of calculation, and is conducive to improving the generalization and expression ability of the model. Therefore, while reducing the number of parameters and the amount of calculation of the convolution operation to adapt to the hardware conditions of low resources, the characteristics of the correlation between different channels contained in the deep convolution feature are fully extracted, which can improve the good effect of speech synthesis, and will not change the channel structure of the deep convolution feature to ensure the consistency of input and output. Then, based on the preset ordinary convolution block, the deep convolution feature is used as the input and output of the speech synthesis function. The preset GELU activation layer contained in the convolution block performs a nonlinear transformation on the initial 1*1 convolution feature to obtain the activation feature, so that the corresponding features can be better captured when processing the speech synthesis task, and the performance of the preset ordinary convolution block is improved. Finally, based on the preset second 1*1 point convolution layer contained in the preset ordinary convolution block, the activation feature is convolved to obtain the second target feature with the same output dimension as the initial 1*1 convolution feature, to ensure that the output of the preset GELU activation layer regresses to the output dimension corresponding to the initial 1*1 convolution feature, to prevent errors caused by the loss of data through the preset GELU activation layer, and to ensure good speech synthesis effects.

[0071] In the embodiment of the present application, by using a 1*1 point convolution layer to convolve the deep convolution feature, it can not only adapt to the situation where the input acoustic feature is of any size under low resource conditions, but also ensure that the spatial structure corresponding to the spatial position of the acoustic feature is not destroyed, thereby ensuring the consistency of the input and output of the preset vocoder model, and setting a preset second 1*1 point convolution layer after the preset GELU activation layer, which can prevent data loss and errors in the process of performing nonlinear transformation of the initial 1*1 convolution feature based on the preset GELU activation layer, and on the basis of expanding the receptive field of the preset vocoder model through the 1*1 point convolution layer, it can ensure that the data regresses to the consistent output corresponding to the initial 1*1 convolution feature, thereby ensuring that the data is error-free, and can improve the efficiency of vocoder speech synthesis under low resource conditions while ensuring the speech synthesis effect.

[0072] In one embodiment, see Figure 4 , Figure 4 The second sub-flow diagram of the low-resource vocoder speech synthesis processing method provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, in this embodiment, based on the preset linear layer block and according to the second target feature, the sampling point corresponding to the acoustic feature of each frame of the speech signal is determined to obtain the target sampling point, including:

[0073] S41, based on the preset first linear layer included in the preset linear layer block, performing a first linear transformation on the second target feature to obtain a corresponding linear transformation feature after dimension upgrading;

[0074] S42. Based on the preset second linear layer included in the preset linear layer block, the linear transformation feature is subjected to a secondary linear transformation and is sampled, the sampling point corresponding to the acoustic feature of each frame of the speech signal is determined, and the target sampling point is obtained.

[0075] Explanatory, please continue to see Figure 2 ,like Figure 2 As shown, the first linear layer included in the preset linear layer block is preset, that is, the preset first linear layer. The preset first linear layer represents the first linear layer-based neural network layer included in the preset linear layer block. At the same time, the second linear layer included in the preset linear layer block is preset, that is, the preset second linear layer. The preset second linear layer represents the second linear layer-based neural network layer included in the preset linear layer block. By setting the preset second linear layer, it is ensured that the output of the preset second linear layer is consistent with the input of the preset first linear layer in dimension. The linear layer is as described above and will not be repeated here. It should be noted that the "first" involved in the preset first linear layer and the "second" involved in the preset second linear layer are only used to distinguish different linear layers, and are not used to limit different linear layers. Other similar descriptions in the embodiments of the present application are analogous.

[0076] According to the above conception and setting, based on the preset first linear layer contained in the preset linear layer block, the second target feature is subjected to a first linear transformation to obtain the corresponding linear transformation feature after dimensionality increase, and then based on the preset second linear layer contained in the preset linear layer block, the linear transformation feature is subjected to a secondary linear transformation and sampled to determine the sampling points corresponding to the acoustic features of each frame of speech signal to obtain the target sampling points.

[0077] Exemplarily, the second target feature from the preset deep residual convolution module included in the preset vocoder model passes through a preset linear layer block, and the preset linear layer block replaces the inverse Fourier transform (ISTFT) in the traditional technology to transform the acoustic features of T frames into a speech signal of a length corresponding to 256*T, where 256 represents the hop_size for extracting acoustic features from speech signals, indicating that the acoustic features of 1 frame need to restore 256 sampling points of the speech signal, where hop_size refers to the number of sample points that each frame of data moves in the time series in speech signal processing. It is the number of samples staggered between two adjacent windows, which determines the degree of overlap between frames. The size of hop_size will affect the segmentation and processing of audio data, and is usually determined by the frame length and frame shift.

[0078] The embodiment of the present application adopts a preset first linear layer and a preset second linear layer to perform a linear transformation on the second target feature and determine the sampling points corresponding to the acoustic features of each frame of the speech signal, thereby replacing the inverse Fourier transform (ISTFT) in the vocoder in the traditional technology. It can not only ensure the consistency of the input and output of the linear layer, but also greatly reduce the amount of calculation, thereby improving the reasoning efficiency of the preset vocoder model, and also make the overall model structure of the vocoder simpler, which can be applicable to various low-resource hardware computing platforms including but not limited to CPUs, and can improve the efficiency of vocoder speech synthesis under low-resource conditions while ensuring the speech synthesis effect.

[0079] It should be noted that the low-resource based vocoder speech synthesis processing method described in the above embodiments can recombine the technical features contained in different embodiments as needed to obtain a combined implementation plan, but all are within the scope of protection required by this application.

[0080] See also Figure 5 , Figure 5 A schematic block diagram of a low-resource vocoder speech synthesis processing device provided in an embodiment of the present application. Corresponding to the above-mentioned low-resource vocoder speech synthesis processing method, an embodiment of the present application also provides a low-resource vocoder speech synthesis processing device. Figure 5 As shown, the low-resource based vocoder speech synthesis processing device includes a unit for executing the above-mentioned low-resource based vocoder speech synthesis processing method, and the low-resource based vocoder speech synthesis processing device can be configured in a computer device. Specifically, please refer to Figure 5 The low-resource based vocoder speech synthesis processing device 50 includes a first determining unit 51, a second determining unit 52, a third determining unit 53 and a reconstruction unit 54.

[0081] The first determination unit 51 is used to obtain T frame acoustic features, and determine features including up and down correlations between the acoustic features of the T frame based on a preset one-dimensional convolution block included in a preset vocoder model to obtain a first target feature, wherein T is a natural number;

[0082] A second determination unit 52 is configured to convolve the first target features based on the channels corresponding to the first target features based on the preset deep residual convolution module included in the preset vocoder model, and then determine the features included in the first target features to obtain a second target feature, wherein the second target feature includes the correlation between different channels of the acoustic features corresponding to the first target feature;

[0083] A third determining unit 53 is used to determine the sampling point corresponding to the acoustic feature of each frame of the speech signal based on the preset linear layer block and according to the second target feature, so as to obtain the target sampling point;

[0084] The reconstruction unit 54 is used to reconstruct the target sampling points into a speech waveform to obtain a target speech.

[0085] In one embodiment, the first determining unit 51 includes:

[0086] A first determining subunit is used to determine a feature including up-and-down correlations between acoustic features of the T frames based on a preset one-dimensional convolution layer included in the preset one-dimensional convolution block, to obtain an initial first feature;

[0087] The first regularization subunit is used to regularize the initial first feature based on the preset first regularization layer included in the preset one-dimensional convolution block to obtain the first target feature.

[0088] In one embodiment, the second determining unit 52 includes:

[0089] A connection subunit, used for connecting N layers of preset depth residual convolution blocks based on residual connection to obtain a preset depth residual convolution module, where N is a natural number;

[0090] The first convolution subunit is used to convolve the first target features based on the respective corresponding channels based on the N layers of preset depth residual convolution blocks, and then determine the characteristics of the respective channels of the acoustic features corresponding to the first target features, and determine the characteristics of the correlation between different channels of the acoustic features corresponding to the first target features, so as to obtain the second target feature.

[0091] In one embodiment, the first convolution subunit comprises:

[0092] A depth convolution subunit, configured to perform depth convolution on the first target feature based on a preset depth separation convolution block included in the preset depth residual convolution block to obtain a depth convolution feature;

[0093] The common convolution subunit is used to convolve the depth convolution feature based on the preset common convolution block included in the preset depth residual convolution block to obtain a second target feature.

[0094] In one embodiment, the depthwise convolution subunit includes:

[0095] A second convolution subunit is used to convolve the first target features according to the channels corresponding to the first target features based on the preset depth separation convolution layer included in the preset depth separation convolution block to obtain an initial depth convolution feature;

[0096] The second regularization subunit is used to regularize the initial depth convolution features based on the preset second regularization layer included in the preset depth separation convolution block to obtain depth convolution features.

[0097] In one embodiment, the common convolution subunit includes:

[0098] A third convolution subunit, configured to convolve the depth convolution feature based on a preset first 1*1 point convolution layer included in the preset common convolution block to obtain an initial 1*1 convolution feature;

[0099] An activation subunit, configured to perform a nonlinear transformation on the initial 1*1 convolution feature based on a preset GELU activation layer included in the preset common convolution block to obtain an activation feature;

[0100] The fourth convolution subunit is used to convolve the activation feature based on the preset second 1*1 point convolution layer included in the preset ordinary convolution block to obtain a second target feature consistent with the output dimension corresponding to the initial 1*1 convolution feature.

[0101] In one embodiment, the third determining unit 53 includes:

[0102] A transformation subunit, configured to perform a first linear transformation on the second target feature based on a preset first linear layer included in the preset linear layer block, so as to obtain a corresponding linear transformation feature after dimensionality increase;

[0103] The sampling subunit is used to perform a secondary linear transformation and sampling on the linear transformation features based on the preset second linear layer contained in the preset linear layer block, determine the sampling points corresponding to the acoustic features of each frame of the speech signal, and obtain the target sampling points.

[0104] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the above-mentioned low-resource-based vocoder speech synthesis processing device and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0105] At the same time, the division and connection methods of the various units in the above-mentioned low-resource-based vocoder speech synthesis processing device are only used for illustration. In other embodiments, the low-resource-based vocoder speech synthesis processing device can be divided into different units as needed, and the various units in the low-resource-based vocoder speech synthesis processing device can also be connected in different orders and methods to complete all or part of the functions of the above-mentioned low-resource-based vocoder speech synthesis processing device.

[0106] The low-resource vocoder speech synthesis processing device can be implemented in the form of a computer program. Figure 6 Runs on the computer device shown.

[0107] See also Figure 6 , Figure 6 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part in other devices.

[0108] See also Figure 6 The computer device 500 includes a processor 502, a memory and a network interface 505 connected via a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also be a volatile storage medium.

[0109] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the above-mentioned low-resource-based vocoder speech synthesis processing method.

[0110] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0111] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned low-resource-based vocoder speech synthesis processing method.

[0112] The network interface 505 is used to communicate with other devices over the network. Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device 500 to which the present application scheme is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the present application scheme. Figure 6 The embodiments shown are consistent and will not be described again here.

[0113] The processor 502 is used to run a computer program 5032 stored in a memory to implement the following steps: obtaining acoustic features of T frames, and based on a preset one-dimensional convolution block included in a preset vocoder model, determining features including up and down correlations between the acoustic features of the T frames to obtain a first target feature, wherein T is a natural number; based on a preset deep residual convolution module included in the preset vocoder model, convolving the first target features based on their respective corresponding channels, and then determining features included in the first target features to obtain a second target feature, wherein the second target feature includes correlations between different channels of the acoustic features corresponding to the first target feature; based on a preset linear layer block and according to the second target feature, determining sampling points corresponding to the acoustic features of each frame of speech signal to obtain target sampling points; reconstructing the target sampling points into a speech waveform to obtain a target speech.

[0114] In one embodiment, the processor 502 implements the following steps when implementing a preset one-dimensional convolution block included in a preset vocoder model, determining a feature including up-down correlations between acoustic features of T frames, and obtaining a first target feature:

[0115] Based on a preset one-dimensional convolution layer included in the preset one-dimensional convolution block, determine a feature including up-down correlations between the acoustic features of the T frames to obtain an initial first feature;

[0116] Based on the preset first regularization layer included in the preset one-dimensional convolution block, the initial first feature is layer-regularized to obtain a first target feature.

[0117] In one embodiment, when the processor 502 implements a preset deep residual convolution module included in the preset vocoder model, convolves the first target features based on the channels corresponding to the first target features, and then determines the features included in the first target features to obtain the second target features, the following steps are specifically implemented:

[0118] Based on residual connection, N layers of preset depth residual convolution blocks are connected to obtain a preset depth residual convolution module, where N is a natural number;

[0119] Based on the N layers of preset depth residual convolution blocks, the first target features are convolved based on their respective corresponding channels, and then the characteristics of the respective channels of the acoustic features corresponding to the first target features are determined, and the characteristics of the correlation between different channels of the acoustic features corresponding to the first target features are determined to obtain the second target features.

[0120] In one embodiment, the processor 502 implements the N-layer preset depth residual convolution block, convolves the first target feature based on the respective corresponding channels, determines the characteristics of the respective channels of the acoustic features corresponding to the first target feature, and determines the characteristics of the correlation between different channels of the acoustic features corresponding to the first target feature, and obtains the second target feature by specifically implementing the following steps:

[0121] Based on the preset depth separation convolution block included in the preset depth residual convolution block, the first target feature is depth-convolved to obtain a depth convolution feature;

[0122] Based on the preset common convolution block included in the preset deep residual convolution block, the deep convolution feature is convolved to obtain a second target feature.

[0123] In one embodiment, when the processor 502 performs depth convolution on the first target feature to obtain a depth convolution feature based on the preset depth separation convolution block included in the preset depth residual convolution module, the processor 502 specifically implements the following steps:

[0124] Based on the preset depth separation convolution layer included in the preset depth separation convolution block, convolve the first target features according to the channels corresponding to the first target features to obtain initial depth convolution features;

[0125] Based on the preset second regularization layer included in the preset depth separation convolution block, the initial depth convolution feature is layer-regularized to obtain the depth convolution feature.

[0126] In one embodiment, when the processor 502 implements the preset common convolution block included in the preset depth residual convolution block to convolve the depth convolution feature to obtain the second target feature, the following steps are specifically implemented:

[0127] Based on the preset first 1*1 point convolution layer included in the preset common convolution block, the depth convolution feature is convolved to obtain an initial 1*1 convolution feature;

[0128] Based on the preset GELU activation layer included in the preset common convolution block, the initial 1*1 convolution feature is nonlinearly transformed to obtain an activation feature;

[0129] Based on the preset second 1*1 point convolution layer included in the preset ordinary convolution block, the activation feature is convolved to obtain a second target feature consistent with the output dimension corresponding to the initial 1*1 convolution feature.

[0130] In one embodiment, when the processor 502 determines the sampling point corresponding to the acoustic feature of each frame of the speech signal based on the preset linear layer block and according to the second target feature to obtain the target sampling point, the processor 502 specifically implements the following steps:

[0131] Based on the preset first linear layer included in the preset linear layer block, the second target feature is subjected to a first linear transformation to obtain a corresponding linear transformation feature after dimensionality increase;

[0132] Based on the preset second linear layer included in the preset linear layer block, the linear transformation feature is subjected to a secondary linear transformation and sampled, the sampling point corresponding to the acoustic feature of each frame of the speech signal is determined, and the target sampling point is obtained.

[0133] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0134] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0135] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps:

[0136] A computer program product, when running on a computer, enables the computer to execute the steps of the low-resource based vocoder speech synthesis processing method described in the above embodiments.

[0137] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] The storage medium is a physical, non-transient storage medium, for example, it can be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store computer programs.

[0140] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0141] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0142] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0144] The software tools or components not provided by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A low-resource based vocoder speech synthesis processing method, characterized in that: include: Acquire T frames of acoustic features, and determine features including up-down correlations between the acoustic features of the T frames based on a preset one-dimensional convolution block included in a preset vocoder model to obtain a first target feature, where T is a natural number; Based on a preset deep residual convolution module included in the preset vocoder model, convolve the first target features based on the channels corresponding to the first target features, determine the features included in the first target features, and obtain a second target feature, wherein the second target feature includes correlations between different channels of acoustic features corresponding to the first target feature; Based on the preset linear layer block and according to the second target feature, determining the sampling point corresponding to the acoustic feature of each frame of the speech signal to obtain the target sampling point; The target sampling points are reconstructed into a speech waveform to obtain a target speech.

2. The low-resource vocoder speech synthesis processing method according to claim 1, characterized in that: Based on a preset one-dimensional convolution block included in a preset vocoder model, a feature including up-down correlation between the acoustic features of the T frames is determined to obtain a first target feature, including: Based on a preset one-dimensional convolution layer included in the preset one-dimensional convolution block, determine a feature including up-down correlations between the acoustic features of the T frames to obtain an initial first feature; Based on the preset first regularization layer included in the preset one-dimensional convolution block, the initial first feature is layer-regularized to obtain a first target feature.

3. The low-resource vocoder speech synthesis processing method according to claim 1 or 2, characterized in that: Based on the preset deep residual convolution module included in the preset vocoder model, the first target features are convolved based on the channels corresponding to the first target features, and then the features included in the first target features are determined to obtain the second target features, including: Based on residual connection, N layers of preset depth residual convolution blocks are connected to obtain a preset depth residual convolution module, where N is a natural number; Based on the N layers of preset depth residual convolution blocks, the first target features are convolved based on their respective corresponding channels, and then the characteristics of the respective channels of the acoustic features corresponding to the first target features are determined, and the characteristics of the correlation between different channels of the acoustic features corresponding to the first target features are determined to obtain the second target features.

4. The low-resource vocoder speech synthesis processing method according to claim 3, characterized in that: Based on the N layers of the preset depth residual convolution blocks, the first target features are convolved based on the channels corresponding to the first target features, and then the characteristics of the channels of the acoustic features corresponding to the first target features are determined, and the characteristics of the correlation between different channels of the acoustic features corresponding to the first target features are determined to obtain the second target features, including: Based on the preset depth separation convolution block included in the preset depth residual convolution block, the first target feature is depth-convolved to obtain a depth convolution feature; Based on the preset common convolution block included in the preset deep residual convolution block, the deep convolution feature is convolved to obtain a second target feature.

5. The low-resource vocoder speech synthesis processing method according to claim 4, characterized in that: Based on the preset depth separation convolution block included in the preset depth residual convolution module, the first target feature is subjected to depth convolution to obtain a depth convolution feature, including: Based on the preset depth separation convolution layer included in the preset depth separation convolution block, the first target features are convolved according to the channels corresponding to the first target features to obtain initial depth convolution features; Based on the preset second regularization layer included in the preset depth separation convolution block, the initial depth convolution feature is layer-regularized to obtain the depth convolution feature.

6. The low-resource vocoder speech synthesis processing method according to claim 4, characterized in that: Based on the preset common convolution block included in the preset deep residual convolution block, the deep convolution feature is convolved to obtain a second target feature, including: Based on the preset first 1*1 point convolution layer included in the preset common convolution block, the depth convolution feature is convolved to obtain an initial 1*1 convolution feature; Based on the preset GELU activation layer included in the preset common convolution block, the initial 1*1 convolution feature is nonlinearly transformed to obtain an activation feature; Based on the preset second 1*1 point convolution layer included in the preset ordinary convolution block, the activation feature is convolved to obtain a second target feature consistent with the output dimension corresponding to the initial 1*1 convolution feature.

7. The low-resource based vocoder speech synthesis processing method according to claim 1 or 2, characterized in that: Based on the preset linear layer block and according to the second target feature, a sampling point corresponding to the acoustic feature of each frame of the speech signal is determined to obtain a target sampling point, including: Based on the preset first linear layer included in the preset linear layer block, the second target feature is subjected to a first linear transformation to obtain a corresponding linear transformation feature after dimensionality increase; Based on the preset second linear layer included in the preset linear layer block, the linear transformation feature is subjected to a secondary linear transformation and sampled, the sampling point corresponding to the acoustic feature of each frame of the speech signal is determined, and the target sampling point is obtained.

8. A low-resource vocoder speech synthesis processing device, characterized in that: include: A first determination unit is used to obtain T frames of acoustic features, and determine features including up and down correlations between the acoustic features of the T frames based on a preset one-dimensional convolution block included in a preset vocoder model to obtain a first target feature, wherein T is a natural number; A second determination unit is configured to convolve the first target features based on the channels corresponding to the first target features based on the preset deep residual convolution module included in the preset vocoder model, and then determine the features included in the first target features to obtain second target features, wherein the second target features include correlations between different channels of acoustic features corresponding to the first target features; A third determining unit is used to determine the sampling point corresponding to the acoustic feature of each frame of the speech signal based on the preset linear layer block and according to the second target feature, so as to obtain the target sampling point; The reconstruction unit is used to reconstruct the target sampling points into a speech waveform to obtain a target speech.

9. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 can be implemented.

Images