Voice frequency band extension method, voice frequency band extension device and electronic equipment

Abstract

The embodiment of the invention discloses a voice frequency band extension method, a voice frequency band extension device and electronic equipment. A specific embodiment of the method comprises the following steps: preprocessing a voice data set to obtain a sample voice of a first sampling rate and a corresponding standard voice of a second sampling rate, the second sampling rate being greater than the first sampling rate; a corresponding cache array is set for a specified convolutional layer in a generator of the model, and the cache array is used for caching the previous frame of voice data in the sample voice processed by the convolutional layer; and inputting the sample voice into the generator, processing the sample voice according to the cache data in the cache array, and outputting extended voice of a second sampling rate of the sample voice. The embodiment is related to an audio processing technology, and the previous frame of voice data can be cached by setting the cache array. Therefore, the extension effect of the voice frequency band can be improved by combining the context information, and particularly, the frequency band extension processing of streaming voice can be realized.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of audio processing, and in particular to a voice band extension method, a voice band extension device, and an electronic device. Background Art

[0002] Bandwidth reduction is a distortion phenomenon produced by lossy speech codecs used in telecommunication networks, resulting in narrowband speech with a frequency of 4 kHz. This gives the received speech a familiar dull, fuzzy character. To address this problem and improve speech quality, signal processing-based band extension methods are often used, such as spectral mapping methods, or methods that synthesize missing high-frequency components based on the formant characteristics of the speech signal.

[0003] However, the inventors have found that the related frequency band extension methods can usually only process offline audio. Moreover, these processing methods often only rely on the current audio signal when performing speech processing, and cannot connect the upper and lower frame information, thus affecting the processing effect.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure provide a voice band extension method, a voice band extension device, an electronic device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a voice band extension method, comprising: preprocessing a speech data set to obtain sample speech of a first sampling rate and a corresponding standard speech of a second sampling rate, wherein the second sampling rate is greater than the first sampling rate; setting a corresponding cache array for a specified convolutional layer in a generator of a model, wherein the cache array is used to cache the previous frame of speech data in the sample speech processed by the convolutional layer; inputting the sample speech into the generator, processing the sample speech according to the cached data in the cache array, and outputting an extended speech of the second sampling rate of the sample speech.

[0008] In some embodiments, the generator of the model includes: an input layer, configured to perform vectorization processing on the input sample speech; an encoder layer, configured to encode the vector data output by the input layer; a communication convolution layer, configured to process the encoded data output by the encoder layer; a decoder layer, configured to decode the data output by the communication convolution layer; an output layer, configured to process the data output by the decoder layer and output the result of the generator; and setting a corresponding cache array for the specified convolution layer in the generator of the model, including: setting a corresponding cache array for the convolution layers in the input layer, encoder layer, communication convolution layer and decoder layer of the generator.

[0009] In some embodiments, the input layer is a one-dimensional convolution layer, corresponding to the first bit in the cache array; the encoder layer includes four downsampled encoding blocks, each encoding block includes three residual convolution layers and one one-dimensional convolution layer, corresponding to the second to seventeenth bits in the cache array; the communication convolution layer includes two one-dimensional convolution layers, corresponding to the eighteenth to nineteenth bits in the cache array; the decoder layer includes four upsampled decoding blocks, each decoding block includes a transposed convolution layer and three residual convolution layers, wherein the residual convolution layer in the decoder layer corresponds to the twentieth to thirty-first bits in the cache array.

[0010] In some embodiments, the method further includes: storing the sample speech in a pre-set residual array; and during the process of the generator processing the sample speech, storing the data output by the specified layer in the generator in the residual array for generating extended speech.

[0011] In some embodiments, the sample speech is processed according to the cache data in the cache array, including: for each convolution layer in the generator, in the process of processing the sample speech, determining whether the cache array corresponding to the convolution layer is empty; based on the determination result, supplementing a target number of data at a preset position of the current frame speech data to be processed by the convolution layer, wherein the target number is related to the convolution kernel and the step size of the convolution layer; processing the supplemented current frame speech data through the convolution layer; and updating the corresponding cache data in the cache array according to the current frame speech data processed by the convolution layer for processing the next frame speech data.

[0012] In some embodiments, based on the judgment result, a target number of data are supplemented at the preset position of the current frame voice data to be processed by the convolution layer, including: in response to the judgment being empty, the front end position of the current frame voice data to be processed by the convolution layer is supplemented with the target number of preset data; in response to the judgment being not empty, the front end position of the current frame voice data to be processed by the convolution layer is supplemented with the target number of voice data cached at the back end position of the previous frame voice data.

[0013] In some embodiments, the method further includes: inputting the extended speech output by the generator into the discriminator of the model, and outputting a discrimination result of the extended speech; determining the loss function value of the model based on the discrimination result and the standard speech to update relevant parameters in the generator and the discriminator.

[0014] In some embodiments, the discriminator includes: a first pooling layer, configured to downsample the extended speech by a first multiple to obtain the first speech; a second pooling layer, configured to downsample the extended speech by a second multiple to obtain the second speech, wherein the second multiple is greater than the first multiple; and three discriminant layers, configured to perform discriminative processing on the extended speech, the first speech, and the second speech, respectively.

[0015] In some embodiments, a speech data set is preprocessed to obtain sample speech of a first sampling rate and a corresponding standard speech of a second sampling rate, including: using the acquired streaming speech data set as the standard speech, wherein the sampling rate of the standard speech is 16 khz; downsampling the streaming speech data set to obtain sample speech, wherein the sampling rate of the sample speech is 8 khz.

[0016] In a second aspect, some embodiments of the present disclosure provide a voice band extension device, comprising: a data preprocessing unit, configured to preprocess a speech data set to obtain a sample speech of a first sampling rate and a corresponding standard speech of a second sampling rate, wherein the second sampling rate is greater than the first sampling rate; a cache setting unit, configured to set a corresponding cache array for a specified convolutional layer in a generator of a model, wherein the cache array is used to cache the previous frame of speech data in the sample speech processed by the convolutional layer; an extended speech generation unit, configured to input the sample speech into the generator, process the sample speech according to the cache data in the cache array, and output the extended speech of the second sampling rate of the sample speech.

[0017] In some embodiments, the generator of the model includes: an input layer, configured to perform vectorization processing on the input sample speech; an encoder layer, configured to encode the vector data output by the input layer; a communication convolution layer, configured to process the encoded data output by the encoder layer; a decoder layer, configured to decode the data output by the communication convolution layer; an output layer, configured to process the data output by the decoder layer and output the result of the generator; and the cache setting unit is further configured to set corresponding cache arrays for the convolution layers in the input layer, encoder layer, communication convolution layer and decoder layer of the generator.

[0018] In some embodiments, the input layer is a one-dimensional convolution layer, corresponding to the first bit in the cache array; the encoder layer includes four downsampled encoding blocks, each encoding block includes three residual convolution layers and one one-dimensional convolution layer, corresponding to the second to seventeenth bits in the cache array; the communication convolution layer includes two one-dimensional convolution layers, corresponding to the eighteenth to nineteenth bits in the cache array; the decoder layer includes four upsampled decoding blocks, each decoding block includes a transposed convolution layer and three residual convolution layers, wherein the residual convolution layer in the decoder layer corresponds to the twentieth to thirty-first bits in the cache array.

[0019] In some embodiments, the voice band extension device also includes a residual storage unit, which is configured to store the sample speech in a pre-set residual array; and during the process of the generator processing the sample speech, the data output by the specified layer in the generator is stored in the residual array for generating extended speech.

[0020] In some embodiments, the extended speech generation unit is further configured to, for each convolution layer in the generator, determine whether the cache array corresponding to the convolution layer is empty during the process of processing the sample speech; based on the judgment result, supplement the target number of data at the preset position of the current frame speech data to be processed by the convolution layer, wherein the target number is related to the convolution kernel and the step size of the convolution layer; process the supplemented current frame speech data through the convolution layer; and update the cache data in the corresponding cache array based on the current frame speech data processed by the convolution layer for processing the next frame speech data.

[0021] In some embodiments, the extended speech generation unit is further configured to, in response to being judged as empty, supplement the front end position of the current frame speech data to be processed by the convolution layer with a target number of preset data; in response to being judged as not empty, supplement the front end position of the current frame speech data to be processed by the convolution layer with a target number of speech data cached at the back end position of the previous frame speech data.

[0022] In some embodiments, the voice band extension device also includes a discriminator unit, which is configured to input the extended speech output by the generator into the discriminator of the model, and output the discrimination result of the extended speech; based on the discrimination result and the standard speech, determine the loss function value of the model to update the relevant parameters in the generator and the discriminator.

[0023] In some embodiments, the discriminator includes: a first pooling layer, configured to downsample the extended speech by a first multiple to obtain the first speech; a second pooling layer, configured to downsample the extended speech by a second multiple to obtain the second speech, wherein the second multiple is greater than the first multiple; and three discriminant layers, configured to perform discriminative processing on the extended speech, the first speech, and the second speech, respectively.

[0024] In some embodiments, the data preprocessing unit is further configured to use the acquired streaming speech data set as standard speech, wherein the sampling rate of the standard speech is 16 khz; downsample the streaming speech data set to obtain sample speech, wherein the sampling rate of the sample speech is 8 khz.

[0025] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the voice band extension method described in any implementation method of the first aspect above.

[0026] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the voice band extension method described in any implementation manner in the first aspect above is implemented.

[0027] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the voice band extension method described in any implementation manner in the first aspect above.

[0028] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the voice band expansion method of some embodiments of the present disclosure can improve the expansion effect of the voice band. Specifically, the related frequency band expansion method can usually only process offline audio. And the current output only depends on the current input, that is, each audio frame can only be processed separately, and the context information cannot be combined, which affects the processing effect of speech, especially streaming speech.

[0029] Based on this, the voice band expansion method of some embodiments of the present disclosure sets a corresponding cache array for the specified convolution layer in the generator of the model. The cache array here can be used to cache the previous frame of voice data in the sample voice processed by the convolution layer. That is, by setting the cache array, the previous frame of voice data can be cached. In this way, after the sample voice of the first sampling rate is input into the generator, the sample voice can be processed according to the cache data in the cache array. That is, the current frame of voice data can be processed in combination with the context information to obtain the extended voice of the second sampling rate of the sample voice. In this way, the expansion effect of the voice band can be improved, thereby effectively restoring the missing part in the frequency band, especially for streaming voice. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0031] Figure 1 is a flow chart of some embodiments of the voice band extension method disclosed in the present invention;

[0032] Figure 2 are flow charts of some embodiments of the generator of the present disclosure;

[0033] Figure 3 is a flowchart of some embodiments of the convolutional layer in the generator of the present disclosure;

[0034] Figure 4 is a flowchart of some embodiments of the discriminator of the present disclosure;

[0035] Figure 5 Schematic diagrams of some application scenarios of the voice band extension method disclosed in the present invention;

[0036] Figure 6 is a schematic diagram of the structure of some embodiments of the voice band extension device disclosed in the present invention;

[0037] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0039] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0040] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0041] In addition, the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0042] Figure 1 The process 100 of some embodiments of the voice band extension method according to the present disclosure is shown. The method may include the following steps:

[0043] Step 101 , preprocessing a speech data set to obtain sample speech at a first sampling rate and a corresponding standard speech at a second sampling rate.

[0044] In some embodiments, the execution subject (such as a server) of the voice band extension method of the present disclosure can be connected to other electronic devices (such as a user terminal, a database, etc.) through a wired connection or a wireless connection. Here, the user can send the voice data set for training the model to the execution subject through the user terminal. Alternatively, the user can store the voice data set in a database and send the storage location (such as a database identifier) ​​to the execution subject. In this way, the execution subject can obtain the voice data set from it.

[0045] It should be noted that the speech data set here can be collected by a microphone (such as Figure 5 As shown), it can also be obtained from public resources. As an example, a joint speech dataset can be constructed based on real meeting data and laboratory recording data. The speech dataset may include at least two languages ​​such as English and Chinese, with a total length of about 100 hours. Among them, real meeting data can come from the ICSIMeeting Corpus dataset, which contains an English dataset of 75 meeting recordings with a total length of about 72 hours. The laboratory recording data can come from an open source dataset such as the VCTK-26spk-train dataset, in which 26 recorders read sentences in a certain language, with a length of about 20 hours.

[0046] In some embodiments, the execution subject may preprocess the above-mentioned speech data set to obtain a sample speech of a first sampling rate and a corresponding standard speech of a second sampling rate. The second sampling rate is usually greater than the first sampling rate. The preprocessing method here can be set according to actual needs.

[0047] It should be noted that, after a large number of experiments, it has been proved that the spectrum of speech is mainly concentrated in 0-8khz, and the high-frequency components beyond the 8khz frequency band have little effect on the quality of daily speech. Therefore, the method of the embodiment of the present disclosure can be mainly aimed at the case of increasing the sampling frequency from 8khz to 16khz, thereby expanding the available bandwidth from 4khz to 8khz respectively. In this case, the sampling rate of the speech in the above-mentioned speech data set can be 16khz. At this time, the execution subject can use the acquired speech data set as the standard speech. That is, the sampling rate of the standard speech can be 16khz. And the speech data set can be downsampled to obtain sample speech in the low frequency band. Here, the sampling rate of the sample speech can be 8khz.

[0048] In addition, the speech data set in the embodiment of the present disclosure can be offline speech or streaming speech. It is understandable that offline speech is usually speech with a length of more than 1 second (s). Streaming speech is generally a plurality of 16 millisecond (ms) audio frames, and at a sampling rate of 8 khz, there are usually only 128 sampling points. It should be noted that the related band extension methods, namely the band extension based on signal processing and the band extension based on deep learning, often have the problem of not being able to accept streaming speech input and can only process offline audio. For offline speech, it is often necessary to process it after all the offline speech input is completed. For streaming speech, the duration is usually shorter, so it is generally real-time input and real-time processing.

[0049] In some application scenarios, in order to implement the frequency band extension processing of streaming voice, the voice data in the above voice data set may be streaming voice data. Figure 5 As shown, the execution entity can cache sample speech audio with a length of 16ms for each frame of audio. And the audio can also be further standardized and preprocessed. The standardized preprocessing method here is also not limited. For example, the sample speech can be read in sequence through an audio processing library (such as torch.audio) to perform statistics and processing of audio information, such as channel separation, length interception, format conversion, etc. As an example, the sample speech can be processed to obtain a single-channel, 8kHz sampling rate wav (an audio file format) format input audio, and each input audio segment has the same interception length, such as 2 seconds.

[0050] Step 102, setting a corresponding cache array for a specified convolutional layer in the generator of the model.

[0051] In some embodiments, before performing voice data processing, the execution entity may also set a corresponding cache array for a specified convolutional layer in the generator of the model. The cache array may be used to cache the previous frame of voice data in the sample voice processed by the convolutional layer. In other words, for the current frame of voice data processed by the convolutional layer, the cache array may be used to store voice data that is the same voice as the current frame of voice data and is located in the previous frame of the current frame of voice. The previous frame of voice data may be the voice data before processing or the voice data after processing. By saving the previous frame of voice data through the cache array, the voice data may be processed in combination with the context information, thereby helping to improve the processing effect.

[0052] It is understandable that the structure of the generator, the configuration of the specified convolutional layer and the corresponding buffer array are also not limited. Figure 2 As shown, the generator may include an input layer (Conv_in convolution), an encoder layer (Encoder encoding), a decoder layer (Decoder decoding) and an output layer (Conv_out convolution). The input layer may be used to vectorize the sample speech input to the generator; the encoder layer may be used to encode the vector data output by the input layer; the decoder layer may be used to decode the encoded data; and the output layer may be used to process the data output by the decoder layer and output the result of the generator.

[0053] Furthermore, the generator can also include a connected convolutional layer (such as Figure 2 The middle convolution layer is located between the encoder layer and the decoder layer, and serves as a link between the encoder and the decoder. In this case, the link convolution layer can process the encoded data output by the encoder layer. The decoder layer can be used to decode the data output by the link convolution layer. Here, the corresponding cache arrays can be set for the input layer, encoder layer, link convolution layer, and convolution layer in the decoder layer of the generator.

[0054] It should be noted that the structure of each layer in the generator can also be set according to actual needs. As an example, the input layer can be a one-dimensional convolution layer, corresponding to the first bit in the cache array. The encoder layer may include four downsampled encoding blocks. Each encoding block may include three residual convolution layers and one one-dimensional convolution layer, corresponding to the second to seventeenth bits in the cache array. The Unicom convolution layer may include two one-dimensional convolution layers, corresponding to the eighteenth to nineteenth bits in the cache array. The decoder layer may include four upsampled decoding blocks. Each decoding block may include a transposed convolution layer and three residual convolution layers. Among them, the residual convolution layer in the decoder layer corresponds to the twentieth to thirty-first bits in the cache array. In other words, each convolution layer in the generator may correspond to different cache bits in the same cache array. The voice data stored in each cache bit can be continuously updated with the voice currently processed by the corresponding convolution layer, that is, only the latest previous frame of voice data is stored. This can reduce the occupation of storage resources.

[0055] Optionally, a corresponding cache array can be separately set for each convolution layer, and an association relationship can be established between the two. The length of the cache array can be set according to the length of the sample speech, and each cache bit is used to store each frame of speech data. For example, for the cache array of the input layer, its length can be the number of frames of the input sample speech. For the cache arrays of other convolution layers, their lengths can be determined according to the number of frames of the sample speech and the data of the previous convolution layer.

[0056] Step 103, inputting the sample speech into the generator, processing the sample speech according to the buffer data in the buffer array, and outputting the extended speech of the sample speech at the second sampling rate.

[0057] In some embodiments, the execution subject may input the sample speech into the generator. The generator may process the sample speech according to the cache data in the cache array, for example, thereby outputting the extended speech of the second sampling rate of the sample speech. In addition, the data in the cache array may be continuously updated during the generator processing the sample speech.

[0058] In some application scenarios, in order to improve the accuracy of data processing, a residual array may be provided in the above generator for residual calculation. As an example, when the generator receives the input sample data, it may first store the sample speech in the residual array. In addition, during the process of the generator processing the sample speech, the data output by the specified layer in the generator may also be stored in the residual array for generating the extended speech.

[0059] Specifically, Figure 2 The process flow shown is:

[0060] (1) First, you can initialize a cache, that is, an array, whose length corresponds to the number of convolutional layers in the generator. For example, if the generator has a total of 31 convolutional layers, the length of the cache array is 31, and each convolutional layer corresponds to a cache bit). Initialize it to 0 to prepare for the subsequent reception of streaming input. Initialize a residual array for the subsequent construction of residual connections, also initialized to 0. At the same time, you can set the batch_size, that is, how many voice data to read each time. The batch_size depends on the existing size of the server.

[0061] (2) Next, the sample speech x (three dimensions, namely batch_size, number of channels, and audio signal) and cache[0] can be sent to the Conv_in convolution layer. The number of output channels is 8, and the convolution kernel size can be 7. At the same time, a copy of the sample speech x is saved in the residual array.

[0062] (3) The output of step (2) and cache[1-16] are sent to the encoder coding layer. The encoder coding layer consists of 4 encblocks, each of which can include three residual convolution layers and one standard convolution layer. Among them, the residual convolution layer can adopt a one-dimensional dilated convolution, and the dilation values ​​can be 1, 3, and 9 respectively. Here, the 4 encblocks can follow the downsampling scheme of (2, 2, 8, 8). That is, the length of the speech sequence is halved after passing through the first two encblocks, and the sequence length becomes 1 / 8 of the original after passing through the last two encblocks. Other downsampling schemes can also be used, such as (2, 4, 6, 8). Since the encoder coding layer has a total of 16 convolution layers, 16 caches are sent. In the encoder coding layer, the speech x output by each encblock is appended and saved in the residual array.

[0063] (4) After the output of the encoder layer, it is sent to two standard one-dimensional convolutional layers (i.e., middle convolutional layers), with the convolution kernel size of 7. At the same time, the cache[17-18] is sent to this layer.

[0064] (5) The decoder decoding layer is the opposite of the encoder encoding layer, including four decblocks (decoding blocks) and following the upsampling scheme of (8, 8, 2, 2). Each decblock can include a transposed convolution layer and three residual convolution layers. Among them, the transposed convolution layer is similar to deconvolution and is mainly used to expand the sequence. The speech x and cache[19-30] output in step (4) are also sent to the decoder decoding layer. In addition, the speech x output by each decblock is added to the corresponding residual block to ensure that the speech x has both the information of the current decoder and the encoder information. That is, it is added to the speech x output by the four encblocks in step (3) respectively.

[0065] (6) Finally, the speech x can be sent to the Conv_out layer, corresponding to the deconvolution of the Conv_in layer. It is added to the sample speech x initially stored in the residual array to obtain the output signal of the generator and update the cache at the same time.

[0066] It should be noted that, although compared with the traditional band extension method, the band extension method based on deep learning can learn effective features from a large amount of data and has a better band extension effect. However, this model is usually implemented using causal convolution, and the current output only depends on the current input. In other words, causal convolution cannot combine contextual information and processes each audio frame separately, which often makes the output streaming audio noisy. The processing effect on streaming audio is not very good.

[0067] From the above description, it can be seen that the generator in the embodiment of the present disclosure involves a large number of convolution operations, in order to adapt to 16ms streaming speech and solve the problem that streaming speech input cannot save context information. The voice band extension method disclosed in the present disclosure proposes a streaming convolution method, which can combine context information for speech processing by caching the previous frame of speech data.

[0068] As an example, for each convolution layer in the generator, in the process of processing sample speech, it can first be determined whether the cache array corresponding to the convolution layer is empty. Then, according to the judgment result, the target number of data can be supplemented at the preset position of the current frame speech data to be processed by the convolution layer. Among them, the target number is related to the convolution kernel and step size of the convolution layer. The preset position here can also be set according to actual needs, such as front end or back end. Afterwards, the supplemented current frame speech data is processed by the convolution layer.

[0069] Specifically, Figure 3 The process shown:

[0070] First, the sample speech (i.e. speech sequence) and the buffer array are fed as parameters into the streaming convolution, i.e. the convolution layer specified above. The convolution kernel size and stride are also set. The output signal length L of the standard one-dimensional convolution is OUT Satisfies the following formula:

[0071] L OUT =(L IN -Kernal size +1) / stride;

[0072] Among them, L IN Represents the input speech sequence; Kernal size Indicates the convolution kernel size; stride indicates the step size.

[0073] For example, if the input speech sequence length is 256, the convolution kernel size is 3, and the stride is 1, then the output signal length is 254. Therefore, in order to keep the output sequence length consistent with the input sequence length, the standard convolution is generally performed at L IN A 0 is added before and after. Causal convolution usually takes place in L IN Add two leading zeros to ensure that the output at the current moment depends only on the input at the current moment.

[0074] In this case, we can first determine whether the cache array is empty. If the cache array is empty, it means that the current audio frame is the first frame of the input streaming speech sequence. At this time, we can add the target number of preset data at the front position of the current frame speech data to be processed by the convolution layer. The preset data here is also not limited. For example, in order not to affect the original speech sequence, the preset data here can be 0. That is to say, in L IN The front end is filled with 0, the number is (Kernal size -1) / stride.

[0075] If the cache array is not empty, it proves that it saves the cache data of the previous frame. At this time, the data in the cache can be supplemented at the front position of the current frame speech data to be processed by the convolution layer. It can be understood that the biggest difference here from the causal convolution is that the causal convolution fills 0 in front of the sequence, while the streaming convolution fills the previous frame information in front of the sequence. In this way, the current frame information also contains the previous frame information, so that speech processing can be combined with context information.

[0076] It should be noted that, since the processing of voice data is generally carried out frame by frame in time sequence, in order to ensure the connection of information between frames, the buffered data added to the front end of the current frame voice data is usually the target number of voice data located at the rear end position of the previous frame voice data, that is, the last (kernal) of the previous frame. size-1) / stride sampling points.

[0077] After that, the supplemented speech sequence can be input into the convolution layer and the output can be obtained through the activation function. Here, by fusing the previous frame of speech data with the current frame of speech data before processing and then sending them into the convolution layer for processing, the convolution layer can learn the relationship between context information more effectively, enrich the learning features of the model, and thus help improve the processing effect of the model.

[0078] It is understandable that, regardless of whether the cache array is empty or not, the cache needs to be updated after zero padding or buffer data padding. According to the current frame of voice data processed by the convolutional layer, the cache data in the corresponding cache array is updated for processing the next frame of voice data. As an example, the previous frame of voice data can be stored in the cache array. For another example, in order to reduce storage space occupancy, the last (kernal) of the current frame of voice data can be updated. size -1) / stride number to replace the data in the buffer and save it. Then the output and updated buffer can be returned to continue processing the next frame of voice data.

[0079] In some embodiments, in order to improve data processing efficiency, the above-mentioned cache array can also be used to store the previous frame of voice data processed by the corresponding convolution layer, such as saving the target number of data located at the back end of the processed previous frame of voice data. In this way, when the convolution layer processes the current frame of voice data to obtain the processed current frame of voice data, the cache data in the cache array can be superimposed on the preset position of the processed current frame of voice data. Thereby, the superimposed current frame of voice data contains the previous frame of voice data. In addition, the cache array can be updated according to the processed current frame of voice data.

[0080] It can be seen from the above description that the voice band extension method of some embodiments of the present disclosure sets a corresponding cache array for the specified convolution layer in the generator of the model. The cache array here can be used to cache the previous frame of voice data in the sample voice processed by the convolution layer. In other words, by setting the cache array, the previous frame of voice data can be cached. In this way, after the sample voice of the first sampling rate is input into the generator, the sample voice can be processed according to the cache data in the cache array. That is, the current frame of voice data can be processed in combination with the context information to obtain the extended voice of the second sampling rate of the sample voice. In this way, the extension effect of the voice band can be improved, thereby effectively restoring the missing part in the frequency band, especially for streaming voice.

[0081] It should be noted that, through the voice band extension method of the above embodiment, the generator can output the extended speech of the second sampling rate. Here, the execution subject can determine the loss function value of the model (i.e., the generator) according to the extended speech and the corresponding standard speech of the second sampling rate, thereby adjusting the relevant parameters in the generator to continue training the adjusted generator until the training is completed.

[0082] In some application scenarios, the model of the present disclosure may also be a generative adversarial network model, that is, the model may also include a discriminator. In this case, the execution subject may input the extended speech output by the generator into the discriminator of the model, and obtain the discrimination result of the extended speech through the discriminator output. Then, according to the discrimination result and the standard speech, the loss function value of the model is determined to update the relevant parameters in the generator and the discriminator.

[0083] It is understandable that the result of the discriminator is not limited here either. As an example, the discriminator can adopt the same architecture as MelGAN (non-autoregressive feedforward convolution architecture), consisting of three discriminator layers (Discriminator_1D) and two pooling operations. Here, the first pooling layer can be used to downsample the extended voice by a first multiple (such as 2 times) to obtain the first voice. The second pooling layer can be used to downsample the extended voice by a second multiple (such as 4 times) to obtain the second voice. Wherein, the second multiple is greater than the first multiple. The three discriminator layers can be used to discriminate the extended voice, the first voice and the second voice respectively.

[0084] like Figure 4 As shown, the extended speech, the first speech and the second speech can be sent to the first discriminant layer, the second discriminant layer and the third discriminant layer respectively. Here, the discriminant layer can include 7 non-causal standard convolutional layers, and the number of output channels can be 16, 64, 256, 1024, 1024, 1024, 1 respectively. In addition, a residual array can also be initialized inside the discriminator. Each time a convolutional layer is passed, a copy is added to the residual array, and finally the processed extended speech (i.e., the discrimination results of the three discriminant layers) and the residual array are returned at the output. The loss function (loss) can be calculated through the returned discrimination results and residual array. The loss here usually includes generator loss and discriminant loss.

[0085] From the above description, it can be seen that the embodiment of the present disclosure proposes a streaming voice band extension method based on streaming convolution, adopts an improved U-Net (a deep learning model) model, and effectively learns spectral features through an encoder-decoder structure. All causal convolutions in the model use streaming convolution, save the previous frame information through caching, and predict future information by combining the previous frame information with the current information to achieve streaming output. That is, by adopting streaming convolution and combining context information, the missing part in the frequency band can be effectively restored, thereby improving the expansion effect of the voice band.

[0086] In the embodiment of the present disclosure, when the input is streaming voice, the voice band extension method of the present disclosure can output high-quality audio with a frame of 16ms and a sampling rate of 16khz. Multiple 16ms audio frames can form an audio stream, so that streaming audio can be continuously output at the output end. Figure 5 In the application scenario shown, the frequency band extension result obtained by this method can be saved for subsequent processing, such as audio denoising, audio downloading, etc.

[0087] Further references Figure 6 , as a response to the above Figures 1 to 5 The present disclosure provides some embodiments of a voice band extension device. These device embodiments are similar to Figures 1 to 5 The device can be specifically applied to various electronic devices.

[0088] like Figure 6 As shown, the voice band extension device 600 of some embodiments may include: a data preprocessing unit 601, configured to preprocess the speech data set to obtain a sample speech of a first sampling rate and a corresponding standard speech of a second sampling rate, wherein the second sampling rate is greater than the first sampling rate; a cache setting unit 602, configured to set a corresponding cache array for a specified convolution layer in a generator of the model, wherein the cache array is used to cache the previous frame of speech data in the sample speech processed by the convolution layer; an extended speech generation unit 603, configured to input the sample speech into the generator, process the sample speech according to the cache data in the cache array, and output the extended speech of the second sampling rate of the sample speech.

[0089] In some embodiments, the generator of the model may include: an input layer, configured to perform vectorization processing on the input sample speech; an encoder layer, configured to encode the vector data output by the input layer; a communication convolution layer, configured to process the encoded data output by the encoder layer; a decoder layer, configured to decode the data output by the communication convolution layer; an output layer, configured to process the data output by the decoder layer and output the result of the generator; and the cache setting unit 602 can be further configured to set corresponding cache arrays for the convolution layers in the input layer, encoder layer, communication convolution layer and decoder layer of the generator.

[0090] In some embodiments, the input layer is a one-dimensional convolution layer, corresponding to the first bit in the cache array; the encoder layer includes four downsampled encoding blocks, each encoding block includes three residual convolution layers and one one-dimensional convolution layer, corresponding to the second to seventeenth bits in the cache array; the communication convolution layer includes two one-dimensional convolution layers, corresponding to the eighteenth to nineteenth bits in the cache array; the decoder layer includes four upsampled decoding blocks, each decoding block includes a transposed convolution layer and three residual convolution layers, wherein the residual convolution layer in the decoder layer corresponds to the twentieth to thirty-first bits in the cache array.

[0091] In some embodiments, the voice band extension device 600 may also include a residual storage unit (not shown in the figure), which is configured to store the sample speech in a pre-set residual array; and during the process of the generator processing the sample speech, the data output by the specified layer in the generator is stored in the residual array for generating extended speech.

[0092] In some embodiments, the extended speech generation unit 603 can be further configured to, for each convolution layer in the generator, determine whether the cache array corresponding to the convolution layer is empty during the process of processing the sample speech; based on the judgment result, supplement the target number of data at the preset position of the current frame speech data to be processed by the convolution layer, wherein the target number is related to the convolution kernel and the step size of the convolution layer; process the supplemented current frame speech data through the convolution layer; and update the cache data in the corresponding cache array based on the current frame speech data processed by the convolution layer for processing the next frame speech data.

[0093] In some embodiments, the extended speech generation unit 603 can be further configured to, in response to being judged as empty, supplement the front end position of the current frame speech data to be processed by the convolution layer with a target number of preset data; in response to being judged as not empty, supplement the front end position of the current frame speech data to be processed by the convolution layer with a target number of speech data cached at the back end position of the previous frame speech data.

[0094] In some embodiments, the voice band extension device 600 may also include a discriminator unit (not shown in the figure), which is configured to input the extended speech output by the generator into the discriminator of the model, and output the discrimination result of the extended speech; based on the discrimination result and the standard speech, determine the loss function value of the model to update the relevant parameters in the generator and the discriminator.

[0095] In some embodiments, the discriminator may include: a first pooling layer, configured to downsample the extended speech by a first multiple to obtain the first speech; a second pooling layer, configured to downsample the extended speech by a second multiple to obtain the second speech, wherein the second multiple is greater than the first multiple; and three discriminant layers, configured to perform discriminative processing on the extended speech, the first speech, and the second speech, respectively.

[0096] In some embodiments, the data preprocessing unit 601 can be further configured to use the acquired streaming speech data set as standard speech, wherein the sampling rate of the standard speech is 16 khz; downsample the streaming speech data set to obtain sample speech, wherein the sampling rate of the sample speech is 8 khz.

[0097] It is understandable that the units described in the voice band extension device 600 are similar to those described in the reference Figures 1 to 5 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the voice band extension device 600 and the units included therein, and will not be described in detail here.

[0098] Reference below Figure 7 , which shows a structural schematic diagram of an electronic device 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0099] like Figure 7 As shown, the electronic device 700 may include a processing device 701 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0100] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a speaker, a vibrator, etc.; storage devices 708 including, for example, a disk, a hard disk, a memory card, etc.; and communication devices 709. The communication devices 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0101] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0102] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0103] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0104] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: pre-processes the speech data set to obtain sample speech of a first sampling rate and a corresponding standard speech of a second sampling rate, wherein the second sampling rate is greater than the first sampling rate; sets a corresponding cache array for a specified convolution layer in the generator of the model, wherein the cache array is used to cache the previous frame of speech data in the sample speech processed by the convolution layer; inputs the sample speech into the generator, processes the sample speech according to the cache data in the cache array, and outputs the extended speech of the second sampling rate of the sample speech.

[0105] In addition, computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The units described may also be set in a processor, for example, may be described as: a processor includes a data preprocessing unit, a cache setting unit, and an extended speech generation unit. The names of these units do not constitute limitations on the units themselves in some cases, for example, the data preprocessing unit may also be described as a "unit for preprocessing a speech data set."

[0108] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0109] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned voice band extension methods when executed by a processor.

[0110] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A voice band extension method, comprising: Preprocessing the speech data set to obtain sample speech at a first sampling rate and a corresponding standard speech at a second sampling rate, wherein the second sampling rate is greater than the first sampling rate; Setting a corresponding cache array for a specified convolutional layer in a generator of the model, wherein the cache array is used to cache a previous frame of speech data in a sample speech processed by the convolutional layer; The sample speech is input into the generator, and the sample speech is processed according to the buffer data in the buffer array, and the extended speech of the second sampling rate of the sample speech is output.

2. The voice band extension method according to claim 1, wherein: The generator of the model includes: An input layer, configured to perform vectorization processing on the input sample speech; An encoder layer, configured to encode the vector data output by the input layer; A communication convolution layer is configured to process the encoded data output by the encoder layer; A decoder layer, configured to decode the data output by the connected convolutional layer; an output layer, configured to process the data output by the decoder layer and output the result of the generator; and The setting of the corresponding cache array for the specified convolutional layer in the generator of the model includes: For the input layer, encoder layer, connected convolution layer and convolution layer in the decoder layer of the generator, corresponding cache arrays are set.

3. The voice band extension method according to claim 2, wherein: The input layer is a one-dimensional convolutional layer, corresponding to the first bit in the cache array; The encoder layer includes four downsampled encoding blocks, each of which includes three residual convolution layers and one one-dimensional convolution layer, corresponding to the second to seventeenth bits in the cache array; The interconnected convolutional layer includes two one-dimensional convolutional layers corresponding to the eighteenth to nineteenth bits in the cache array; The decoder layer includes four upsampled decoding blocks, each of which includes a transposed convolution layer and three residual convolution layers, wherein the residual convolution layer in the decoder layer corresponds to the 20th to 31st bits in the cache array.

4. The voice band extension method according to claim 1, wherein: The method further comprises: storing the sample speech in a preset residual array; and During the process of the generator processing the sample speech, the data output by the designated layer in the generator is stored in the residual array for generating the extended speech.

5. The voice band extension method according to claim 1, wherein: The processing of the sample speech according to the buffer data in the buffer array includes: For each convolution layer in the generator, during processing of the sample speech, determining whether a cache array corresponding to the convolution layer is empty; According to the judgment result, a target number of data is supplemented at a preset position of the current frame speech data to be processed by the convolution layer, wherein the target number is related to the convolution kernel and the step size of the convolution layer; The supplemented current frame speech data is processed through the convolution layer; According to the current frame of speech data processed by the convolutional layer, the cache data in the corresponding cache array is updated for processing the next frame of speech data.

6. The voice band extension method according to claim 5, wherein: According to the judgment result, the target number of data is supplemented at the preset position of the current frame speech data to be processed by the convolution layer, including: In response to being judged as empty, supplementing a target number of preset data at the front end position of the current frame speech data to be processed by the convolution layer; In response to the judgment that it is not empty, the front end position of the current frame of voice data to be processed by the convolution layer is supplemented with a target number of voice data cached at the back end position of the previous frame of voice data.

7. The voice band extension method according to claim 1, wherein: The method further comprises: Inputting the extended speech output by the generator into the discriminator of the model, and outputting a discrimination result of the extended speech; According to the discrimination result and the standard speech, the loss function value of the model is determined to update relevant parameters in the generator and the discriminator.

8. The voice band extension method according to claim 7, wherein: The discriminator comprises: A first pooling layer is configured to downsample the extended speech by a first multiple to obtain a first speech; A second pooling layer is configured to downsample the extended speech by a second multiple to obtain a second speech, wherein the second multiple is greater than the first multiple; The three discrimination layers are configured to perform discrimination processing on the extended speech, the first speech and the second speech respectively.

9. The voice band extension method according to claim 1, wherein: The preprocessing of the speech data set to obtain a sample speech of a first sampling rate and a corresponding standard speech of a second sampling rate includes: The obtained streaming speech data set is used as the standard speech, wherein the sampling rate of the standard speech is 16 khz; The streaming speech data set is downsampled to obtain sample speech, wherein the sampling rate of the sample speech is 8 kHz.

10. A voice band extension device, comprising: A data preprocessing unit is configured to preprocess the speech data set to obtain a sample speech of a first sampling rate and a corresponding standard speech of a second sampling rate, wherein the second sampling rate is greater than the first sampling rate; A cache setting unit is configured to set a corresponding cache array for a specified convolutional layer in a generator of the model, wherein the cache array is used to cache a previous frame of speech data in a sample speech processed by the convolutional layer; The extended speech generating unit is configured to input the sample speech into the generator, process the sample speech according to the buffer data in the buffer array, and output the extended speech of the sample speech at the second sampling rate.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the voice band extension method as claimed in any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the voice band extension method as claimed in any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the voice band extension method according to any one of claims 1 to 9.

Images