Voice signal reconstruction method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of voice processing, is applied to a voice signal reconstruction scene, and relates to a voice signal reconstruction method, device and equipment and a storage medium thereof. Performing down-sampling processing on different sub-band features by adopting an independent encoder to obtain a low-dimensional feature vector of each sub-band feature; performing quantization processing on the low-dimensional feature vector corresponding to each sub-band feature to obtain a discrete numerical result; and inputting discrete numerical results corresponding to all the sub-band features into a decoder, and performing up-sampling recovery processing to obtain a reconstructed voice waveform corresponding to the original voice signal. According to the method, each sub-band feature is firstly subjected to down-sampling processing, then up-sampling recovery is carried out in combination with all down-sampling quantization results, and audio information in each sub-band feature is fully utilized, so that the high-frequency part of the voice is reconstructed to be more detailed, and the audio synthesis quality of intelligent voice customer service in the financial field is ensured.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and is applied to speech signal reconstruction scenarios, and in particular to a speech signal reconstruction method, apparatus, device, and storage medium thereof. Background Art

[0002] Speech processing is an important branch of artificial intelligence. Speech signal reconstruction technology is an important speech processing technology. It obtains high-quality reconstructed speech signals to provide technical support for speech synthesis and ensure that the synthesized speech has high sound quality.

[0003] Vocoder is a technology that reconstructs speech signals from acoustic features. At present, the mainstream vocoders include: the adversarial generative model represented by hifigan, which works by upsampling the input acoustic features through several layers to generate the final speech signal and align the acoustic features with the speech signal; the model combining upsampling and inverse Fourier transform represented by mb_hifigan, which works by combining upsampling with inverse Fourier transform, and then generating the final speech signal from the output latent variables through inverse Fourier transform. However, the above two reconstruction processing methods cannot fully utilize the audio information in the acoustic features, which affects the reconstruction quality of the audio signal to a certain extent. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a speech signal reconstruction method, device, equipment and storage medium thereof to solve the problem that the existing speech signal reconstruction processing method cannot fully utilize the audio information in the acoustic features, thereby affecting the quality of audio signal reconstruction.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for reconstructing a speech signal, which adopts the following technical solution:

[0006] A method for reconstructing a speech signal comprises the following steps:

[0007] Obtaining the original speech signal;

[0008] The original speech signal is processed by framing, windowing and fast Fourier transform to obtain a Mel-spectrogram corresponding to the original speech signal;

[0009] Identify all sub-band features contained in the Mel-language spectrogram, and count the number of all sub-band features;

[0010] Using independent encoders equal to the number of all sub-band features to perform downsampling processing on different sub-band features respectively, to obtain a low-dimensional feature vector of each sub-band feature after the downsampling processing;

[0011] A vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature, and discrete numerical results corresponding to all sub-band features after quantization are obtained;

[0012] The discrete numerical results corresponding to all sub-band features are respectively input into a preset decoder, and all sub-band features are up-sampled and restored by the decoder to obtain a reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

[0013] Furthermore, before executing the step of identifying all sub-band features contained in the Mel-level spectrogram and counting the number of all sub-band features, the method further includes:

[0014] According to preset segmentation parameters, the original sub-band features contained in the Mel-word spectrogram are segmented to obtain all sub-band features after segmentation based on the original sub-band features, wherein the segmentation parameters include segmentation frequency bands, the original sub-band features refer to equally spaced sub-band features originally contained in the Mel-word spectrogram, and the segmentation operation is performed by a preset bandpass filter group, and each filter in the bandpass filter group segments the original sub-band features according to different segmentation frequency bands.

[0015] Furthermore, before executing the step of using independent encoders equal to the number of all sub-band features to perform downsampling processing on different sub-band features respectively to obtain a low-dimensional feature vector after downsampling processing for each sub-band feature, the method further includes:

[0016] Get the preset downsampling parameters;

[0017] Performing distinguishing numbering processing on all the sub-band features to obtain distinguishing numbering results of all the sub-band features;

[0018] Performing a distinguishing numbering process on all independent encoders to obtain distinguishing numbering results of all independent encoders;

[0019] Allocating independent encoders to all sub-band features according to the difference numbering results of all sub-band features and the difference numbering results of all independent encoders;

[0020] After executing the step of using independent encoders equal in number to all sub-band features to perform downsampling processing on different sub-band features respectively to obtain a low-dimensional feature vector of each sub-band feature after the downsampling processing, the method further includes:

[0021] For each sub-band feature, after downsampling, the low-dimensional feature vector is optimized and outputted using the RVQ algorithm to obtain an optimized output result, wherein the RVQ algorithm is a variant of VQ-VAE, which is mainly used to solve the difference problem between the encoder output and the quantized output;

[0022] According to the distinguished numbering results of all the sub-band features, distinguished numbers are assigned to the optimized output results corresponding to the low-dimensional feature vectors after downsampling of each sub-band feature.

[0023] Furthermore, the step of using a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to all sub-band features after the quantization process specifically includes:

[0024] The FSQ vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature;

[0025] Obtain discrete numerical results corresponding to all sub-band features;

[0026] After executing the step of using a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to all sub-band features after the quantization process, the method further includes:

[0027] The discrete numerical results corresponding to all sub-band features are cached as set or array format data in numerical order, and the distinguishing number results of the corresponding sub-band features are set as the distinguishing numbers of the corresponding cached data.

[0028] Furthermore, before executing the step of inputting the discrete numerical results corresponding to all sub-band features into a preset decoder, the method further includes:

[0029] Splicing all cache data in set form or array format according to the distinguishing numbers of all sub-band features to obtain a target splicing matrix;

[0030] The step of inputting the discrete numerical results corresponding to all sub-band features into a preset decoder specifically includes:

[0031] The target splicing matrix is ​​input into the decoder.

[0032] Furthermore, the step of performing upsampling and recovery processing on all sub-band features through the decoder to obtain a reconstructed speech waveform corresponding to the original speech signal specifically includes:

[0033] Get the preset upsampling parameters;

[0034] According to the upsampling parameters, upsampling is performed on the target splicing matrix to obtain an upsampling result;

[0035] The decoder is used to reconstruct a speech signal on the up-sampling result to obtain a reconstructed speech waveform corresponding to the original speech signal.

[0036] Furthermore, after executing the step of performing up-sampling recovery processing on all sub-band features through the decoder to obtain a reconstructed speech waveform corresponding to the original speech signal, the method further includes:

[0037] Obtaining a speech waveform before reconstruction corresponding to the original speech signal;

[0038] Performing a similarity comparison between the reconstructed speech waveform and the speech waveform before reconstruction to obtain a waveform similarity comparison result;

[0039] If the waveform similarity does not meet the preset similarity threshold, the segmentation parameters, downsampling parameters and upsampling parameters are adjusted, and the encoding and decoding steps are re-executed to obtain the latest reconstructed speech waveform until the waveform similarity meets the preset similarity threshold, and the parameter adjustment is completed;

[0040] If the waveform similarity meets a preset similarity threshold, speech synthesis is performed using the reconstructed speech waveform.

[0041] In order to solve the above technical problems, the embodiment of the present application also provides a speech signal reconstruction device, which adopts the following technical solution:

[0042] A speech signal reconstruction device, comprising:

[0043] An original voice signal acquisition module, used to acquire the original voice signal;

[0044] The original speech signal processing module is used to perform frame segmentation, windowing and fast Fourier transform processing on the original speech signal in order to obtain a Mel spectrogram corresponding to the original speech signal;

[0045] A sub-band feature statistics module is used to identify all sub-band features contained in the Mel-language spectrogram and count the number of all sub-band features;

[0046] A downsampling processing module, used to perform downsampling processing on different sub-band features respectively by using independent encoders equal to the number of all sub-band features, so as to obtain a low-dimensional feature vector of each sub-band feature after the downsampling processing;

[0047] A quantization processing module is used to quantize the low-dimensional feature vector corresponding to each sub-band feature using a vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features after quantization;

[0048] The upsampling recovery module is used to input the discrete numerical results corresponding to all sub-band features into a preset decoder, and perform upsampling recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0050] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech signal reconstruction method when executing the computer-readable instructions.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0052] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech signal reconstruction method as described above.

[0053] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0054] The speech signal reconstruction method described in the embodiment of the present application obtains the original speech signal; performs frame division, windowing and fast Fourier transform processing on the original speech signal in succession to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram, and counts the number of all sub-band features; uses independent encoders equal to the number of all sub-band features to downsample different sub-band features respectively, and obtains a low-dimensional feature vector of each sub-band feature after downsampling; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature, and obtains the discrete numerical results corresponding to all sub-band features after quantization; inputs the discrete numerical results corresponding to all sub-band features into a preset decoder, and performs upsampling and recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal. First, each sub-band feature is downsampled, and then up-sampled and recovered in combination with all down-sampled quantization results, making full use of the audio information in each sub-band feature, so that the high-frequency part of the speech is reconstructed in more detail, and the reconstruction quality of the audio signal is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0057] Figure 2 is a flow chart of an embodiment of a method for reconstructing a speech signal according to the present application;

[0058] Figure 3 It is a flowchart of a specific embodiment of allocating independent encoders in the speech signal reconstruction method described in the present application;

[0059] Figure 4 It is a flowchart of a specific embodiment of optimizing the output of the downsampling processing result in the speech signal reconstruction method described in the present application;

[0060] Figure 5 yes Figure 2 A flowchart of a specific embodiment of performing up-sampling processing in step 206 is shown;

[0061] Figure 6 yes Figure 2 A flowchart of a specific embodiment of performing parameter optimization adjustment after upsampling processing in step 206 is shown;

[0062] Figure 7 is a structural schematic diagram of an embodiment of a speech signal reconstruction device according to the present application;

[0063] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0067] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0068] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0069] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0070] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0071] It should be noted that the speech signal reconstruction method provided in the embodiment of the present application is generally executed by a server, and accordingly, the speech signal reconstruction device is generally arranged in the server.

[0072] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0073] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech signal reconstruction method according to the present application. The speech signal reconstruction method comprises the following steps:

[0074] Step 201: Acquire an original speech signal.

[0075] Specifically, the original speech signal refers to a speech signal directly obtained according to the original speech content of the target pronunciation object, for example, a speech signal directly obtained according to a segment of speech content of the target pronunciation object.

[0076] Specifically, the target pronunciation object is, for example, an artificial agent with a relatively beautiful pronunciation in the financial field, and obtaining the original voice signal is a voice signal directly obtained by obtaining the original voice content of the artificial agent.

[0077] Step 202, the original speech signal is subjected to frame division, windowing and fast Fourier transform processing in sequence to obtain a Mel-spectrogram corresponding to the original speech signal.

[0078] Specifically, the fast Fourier transform, such as FFT (Fast Fourier Transform), is used to transform a signal from the time domain to the frequency domain. The Mel spectrogram refers to a Mel feature graph.

[0079] The Mel-spectrogram is different from other acoustic features in that the spectrogram contains equally spaced original sub-band features.

[0080] The original speech signal corresponding to the artificial agent is processed by framing, windowing and fast Fourier transform in sequence to obtain the Mel spectrogram corresponding to the original speech signal, so as to obtain the reconstructed speech waveform corresponding to the artificial agent later.

[0081] Step 203: identify all sub-band features contained in the Mel-level spectrogram, and count the number of all sub-band features.

[0082] By counting the number of all sub-band features, an equal number of independent encoders can be prepared later.

[0083] Step 204: Use independent encoders equal in number to all sub-band features to perform down-sampling processing on different sub-band features respectively, to obtain a low-dimensional feature vector of each sub-band feature after the down-sampling processing.

[0084] Specifically, the downsampling process, in this embodiment, refers to reducing the sampling rate of a high-resolution audio signal in speech signal processing, thereby reducing the amount of data for storage, transmission or subsequent processing.

[0085] Step 205: Use a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature, and obtain discrete numerical results corresponding to all sub-band features after the quantization process.

[0086] Specifically, the fast Fourier transform, namely FFT transform, is originally a type of discrete Fourier transform. The speech signal obtained is a discrete speech signal. At this time, a vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain a discrete value set corresponding to each sub-band feature for subsequent processing.

[0087] Step 206, input the discrete numerical results corresponding to all sub-band features into a preset decoder, and perform up-sampling recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

[0088] Specifically, the upsampling recovery process refers to the process of converting a low sampling rate audio signal into a high sampling rate audio signal. The sampling rate is increased by inserting additional sample points. Specifically, it inserts new sample points between the original sample points so that there are more sample points between each original sample point. This allows the audio signal to be played at a higher sampling rate, thereby theoretically improving the sound quality.

[0089] In this embodiment, the upsampling recovery process can be understood as that, since the low-dimensional feature vector corresponding to each sub-band feature is quantized to obtain a large number of discrete numerical results, during upsampling recovery, sampling points can be selected from all the discrete numerical results according to the upsampling parameters, thereby ensuring the quality of audio reconstruction.

[0090] Specifically, the speech signal reconstruction method described in the present application can be applied to the financial customer service speech reconstruction scenario, for example: obtaining a customer service speech segment with relatively high-quality sound timbre, and extracting the original speech signal; then using the speech signal reconstruction method to obtain a reconstructed speech waveform; overall, the speech signal reconstruction method is a pre-processing step for actual speech synthesis, and subsequently, when performing speech synthesis in combination with intelligent voice customer service, speech synthesis processing is directly performed through the voice script text and the reconstructed speech waveform to improve the voice timbre quality of the intelligent voice customer service, so that the communication voice of the intelligent voice customer service is closer to the above-mentioned customer service with relatively high-quality sound timbre, ensuring that a better service experience is provided to customers.

[0091] In this embodiment, the original speech signal is obtained; the original speech signal is framed, windowed and fast Fourier transformed to obtain a Mel spectrogram; all sub-band features contained in the Mel spectrogram are identified, and the number of all sub-band features is counted; independent encoders equal to the number of all sub-band features are used to downsample different sub-band features to obtain low-dimensional feature vectors of each sub-band feature after downsampling; the low-dimensional feature vector corresponding to each sub-band feature is quantized by a vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features after quantization; the discrete numerical results corresponding to all sub-band features are input into a preset decoder, and all sub-band features are upsampled and restored by the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal. Each sub-band feature is first downsampled, and then upsampled and restored in combination with all downsampled quantization results, making full use of the audio information in each sub-band feature, so that the high-frequency part of the speech is reconstructed in more detail, and the reconstruction quality of the audio signal is guaranteed.

[0092] In this embodiment, before executing the step of identifying all sub-band features contained in the Mel-word spectrogram and counting the number of all sub-band features, the method also includes: performing segmentation processing on the original sub-band features contained in the Mel-word spectrogram according to preset segmentation parameters, and obtaining all sub-band features after segmentation processing based on the original sub-band features, wherein the segmentation parameters include segmentation frequency bands, the original sub-band features refer to the equally spaced sub-band features originally contained in the Mel-word spectrogram, and the segmentation processing operation is performed by a preset bandpass filter group, and each filter in the bandpass filter group performs segmentation processing on the original sub-band features according to different segmentation frequency bands.

[0093] By performing segmentation again on the basis of the original sub-band features contained in the Mel-level spectrogram, more refined sub-band features can be obtained, ensuring that when the audio signal is subsequently reconstructed, the more refined sub-band features are fully utilized, thereby utilizing more sufficient audio information for audio signal reconstruction.

[0094] Continue to refer Figure 3 In some optional implementations, before step 204, a step of allocating independent encoders is also included. Figure 3 It is a flowchart of a specific embodiment of allocating independent encoders in the speech signal reconstruction method described in the present application, comprising the following steps:

[0095] Step 301, obtaining preset downsampling parameters;

[0096] Step 302, performing a distinguishing numbering process on all the sub-band features to obtain distinguishing numbering results of all the sub-band features;

[0097] Step 303, performing a distinguishing numbering process on all independent encoders to obtain distinguishing numbering results of all independent encoders;

[0098] Step 304: Allocate independent encoders to all sub-band features according to the difference numbering results of all sub-band features and the difference numbering results of all independent encoders.

[0099] By combining the difference numbering results of all sub-band features and the difference numbering results of all independent encoders, all sub-band features are assigned independent encoders, which ensures high controllability of the encoding process, so that each independent encoder only processes sub-band features of a specific frequency band.

[0100] Continue to refer Figure 4 In some optional implementations, after step 204, a step of optimizing and outputting the downsampling processing result is also included. Figure 4 It is a flowchart of a specific embodiment of optimizing the output of the downsampling processing result in the speech signal reconstruction method described in the present application, comprising the following steps:

[0101] Step 401, for each sub-band feature, after downsampling, the low-dimensional feature vector is optimized and outputted using the RVQ algorithm to obtain an optimized output result;

[0102] Among them, the RVQ (Residual Vector Quantization) algorithm is a variant of the VQ-VAE (Vector QuantizedVariational Autoencoder), which is mainly used to solve the difference problem between the encoder output and the quantized output.

[0103] Step 402: according to the distinguishing numbering results of all the sub-band features, assigning distinguishing numbers to the optimized output results corresponding to the low-dimensional feature vectors after downsampling of each sub-band feature.

[0104] By optimizing the output of the downsampling processing results, it is ensured that the output low-dimensional feature vector of each sub-band feature after downsampling processing better meets the quantization requirements and is convenient for subsequent processing.

[0105] In this embodiment, the step of using a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to all sub-band features after quantization specifically includes: using the FSQ vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature; obtaining discrete numerical results corresponding to all sub-band features.

[0106] Among them, the FSQ (Finite Scalar Quantization) vector quantization algorithm is an optimization solution used to replace the vector quantization in VQ-VAE (Vector Quantized Variational Autoencoder). It does not require additional auxiliary loss functions to pull the codeword vector and the unquantized representation vector, which simplifies the training process and improves the coding utilization.

[0107] In this embodiment, after executing the step of using a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to all sub-band features after quantization, the method also includes: caching the discrete numerical results corresponding to all sub-band features in a set form or array format data in numerical order, and setting the distinguishing number result of the corresponding sub-band feature as the distinguishing number of the corresponding cached data.

[0108] The low-dimensional feature vector corresponding to each sub-band feature is quantized through a vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features after quantization, and the difference number results of the sub-band features are set as the difference number of the corresponding cache data, so as to facilitate the call of the discrete numerical results before subsequent decoding.

[0109] In this embodiment, before executing the step of inputting the discrete numerical results corresponding to all sub-band features into a preset decoder, the method also includes: splicing all cache data in set form or array format according to the different numbers of all sub-band features to obtain a target splicing matrix.

[0110] In this embodiment, the step of inputting the discrete numerical results corresponding to all sub-band features into a preset decoder specifically includes: inputting the target splicing matrix into the decoder.

[0111] By splicing all cached data in set or array format according to the different numbers of all sub-band features, a target splicing matrix is ​​obtained, and the target splicing matrix is ​​input into the decoder, that is, the discrete numerical results are spliced ​​according to the different frequency bands corresponding to the sub-band features to ensure that more sampling points can be used when the decoder is used for subsequent decoding, so that the high-frequency part of the speech is reconstructed in more detail, and the reconstruction quality of the audio signal is guaranteed. The discrete numerical results are spliced ​​according to the different frequency bands corresponding to the sub-band features and applied to the pre-processing of speech synthesis in the financial field, which ensures the reconstruction quality of the audio signal corresponding to the artificial seat, thereby improving the communication voice quality of the subsequent intelligent voice customer service.

[0112] Continue to refer Figure 5 , Figure 5 yes Figure 2 The flowchart of a specific embodiment of performing upsampling processing in step 206 shown includes the following steps:

[0113] Step 501, obtaining preset upsampling parameters;

[0114] The upsampling parameters include sampling rate, bit depth and number of channels.

[0115] Step 502, performing upsampling processing on the target splicing matrix according to the upsampling parameters to obtain an upsampling result;

[0116] Step 503: Reconstruct the speech signal of the up-sampling result by using the decoder to obtain a reconstructed speech waveform corresponding to the original speech signal.

[0117] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of performing parameter optimization adjustment after upsampling processing in step 206 shown includes the following steps:

[0118] Step 601, obtaining a speech waveform before reconstruction corresponding to the original speech signal;

[0119] Step 602, performing a similarity comparison between the reconstructed speech waveform and the speech waveform before reconstruction to obtain a waveform similarity comparison result;

[0120] Step 603, if the waveform similarity does not meet the preset similarity threshold, the segmentation parameter, downsampling parameter and upsampling parameter are adjusted, and the encoding and decoding steps are re-executed to obtain the latest reconstructed speech waveform until the waveform similarity meets the preset similarity threshold, and the parameter adjustment is completed;

[0121] Specifically, the encoding and decoding steps are re-executed, that is, after adjusting the segmentation parameters, downsampling parameters and upsampling parameters, the independent encoders in steps 204 to 206 are re-executed for downsampling. Since the segmentation parameters change, the number of independent encoders also changes accordingly, and the downsampling parameters and upsampling parameters also change again, so it is equivalent to reconstructing the original speech signal.

[0122] Step 604: If the waveform similarity meets a preset similarity threshold, speech synthesis is performed using the reconstructed speech waveform.

[0123] The upsampling method is adopted to sample from the discrete numerical results corresponding to all sub-band features, and the audio signal is reconstructed. The segmentation parameters, downsampling parameters and upsampling parameters are adjusted in combination with the parameter optimization adjustment method, which fully guarantees the reconstruction quality of the audio signal.

[0124] The present application obtains the original speech signal; performs frame division, windowing and fast Fourier transform processing on the original speech signal in succession to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram, and counts the number of all sub-band features; uses independent encoders equal to the number of all sub-band features to downsample different sub-band features respectively, and obtains a low-dimensional feature vector of each sub-band feature after downsampling; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature, and obtains discrete numerical results corresponding to all sub-band features after quantization; inputs the discrete numerical results corresponding to all sub-band features into a preset decoder, and performs upsampling and recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal. First, each sub-band feature is downsampled, and then up-sampled and recovered in combination with all down-sampled quantization results, making full use of the audio information in each sub-band feature, so that the high-frequency part of the speech is reconstructed in more detail, and the reconstruction quality of the audio signal is guaranteed. In general, the speech signal reconstruction method is a pre-processing step for actual speech synthesis. Subsequently, when speech synthesis is performed in combination with intelligent voice customer service, speech synthesis processing is performed directly through the voice script text and the reconstructed speech waveform to improve the voice timbre quality of the intelligent voice customer service, making the communication voice of the intelligent voice customer service closer to the above-mentioned customer service with better sound and timbre, ensuring a better service experience for customers.

[0125] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0126] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0127] In the embodiment of the present application, the original speech signal is obtained; the original speech signal is framed, windowed and fast Fourier transformed successively to obtain a Mel spectrogram; all sub-band features contained in the Mel spectrogram are identified, and the number of all sub-band features is counted; independent encoders equal to the number of all sub-band features are used to downsample different sub-band features respectively to obtain a low-dimensional feature vector of each sub-band feature after downsampling; the low-dimensional feature vector corresponding to each sub-band feature is quantized by a vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features after quantization; the discrete numerical results corresponding to all sub-band features are input into a preset decoder, and all sub-band features are upsampled and restored by the decoder to obtain a reconstructed speech waveform corresponding to the original speech signal. Each sub-band feature is first downsampled, and then upsampled and restored in combination with all downsampled quantization results, making full use of the audio information in each sub-band feature, so that the high-frequency part of the speech is reconstructed in more detail, ensuring the reconstruction quality of the audio signal.

[0128] Further references Figure 7 , as a response to the above Figure 2 The present application provides an embodiment of a speech signal reconstruction device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0129] like Figure 7 As shown, the speech signal reconstruction device 700 described in this embodiment includes: an original speech signal acquisition module 701, an original speech signal processing module 702, a sub-band feature statistics module 703, a downsampling processing module 704, a quantization processing module 705 and an upsampling recovery module 706. Among them:

[0130] The original speech signal acquisition module 701 is used to acquire the original speech signal;

[0131] The original speech signal processing module 702 is used to perform frame segmentation, windowing and fast Fourier transform processing on the original speech signal in sequence to obtain a Mel spectrogram corresponding to the original speech signal;

[0132] The sub-band feature statistics module 703 is used to identify all sub-band features contained in the Mel-language spectrogram and count the number of all sub-band features;

[0133] A downsampling processing module 704 is used to perform downsampling processing on different sub-band features respectively using independent encoders equal to the number of all sub-band features to obtain a low-dimensional feature vector of each sub-band feature after the downsampling processing;

[0134] The quantization processing module 705 is used to quantize the low-dimensional feature vector corresponding to each sub-band feature using a vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features after quantization;

[0135] The upsampling recovery module 706 is used to input the discrete numerical results corresponding to all sub-band features into a preset decoder, and perform upsampling recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

[0136] The present application obtains the original speech signal; performs frame division, windowing and fast Fourier transform processing on the original speech signal in succession to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram, and counts the number of all sub-band features; uses independent encoders equal to the number of all sub-band features to downsample different sub-band features respectively, and obtains a low-dimensional feature vector of each sub-band feature after downsampling; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature, and obtains discrete numerical results corresponding to all sub-band features after quantization; inputs the discrete numerical results corresponding to all sub-band features into a preset decoder, and performs upsampling and recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal. First, each sub-band feature is downsampled, and then up-sampled and recovered in combination with all down-sampled quantization results, making full use of the audio information in each sub-band feature, so that the high-frequency part of the speech is reconstructed in more detail, and the reconstruction quality of the audio signal is guaranteed.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0138] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0139] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0140] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c which are interconnected through a system bus. It should be noted that: Figure 8 Only a computer device 8 having a component memory 8a, a processor 8b, and a network interface 8c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0141] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0142] The memory 8a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 8a can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 8a can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 8a is generally used to store the operating system and various application software installed on the computer device 8, such as a computer-readable instruction of a speech signal reconstruction method, etc. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or are to be output.

[0143] The processor 8b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 8b is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to run the computer-readable instructions or process data stored in the memory 8a, such as computer-readable instructions for running the speech signal reconstruction method.

[0144] The network interface 8c may include a wireless network interface or a wired network interface. The network interface 8c is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0145] The computer device proposed in this embodiment belongs to the field of speech processing technology and is applied to the reconstruction scenario of speech signals. The present application obtains the original speech signal; performs frame division, windowing and fast Fourier transform processing on the original speech signal in succession to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram, and counts the number of all sub-band features; uses independent encoders equal to the number of all sub-band features to downsample different sub-band features respectively, and obtains a low-dimensional feature vector after downsampling of each sub-band feature; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature, and obtains the discrete numerical results corresponding to all sub-band features after quantization; inputs the discrete numerical results corresponding to all sub-band features into a preset decoder, and performs upsampling and recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal. First, each sub-band feature is downsampled, and then up-sampled and recovered in combination with all down-sampled quantization results, making full use of the audio information in each sub-band feature, so that the high-frequency part of the speech is reconstructed in more detail, and the reconstruction quality of the audio signal is guaranteed.

[0146] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the speech signal reconstruction method as described above.

[0147] The computer-readable storage medium proposed in this embodiment belongs to the field of speech processing technology and is applied to the reconstruction scenario of speech signals. The present application obtains the original speech signal; performs frame division, windowing and fast Fourier transform processing on the original speech signal in succession to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram, and counts the number of all sub-band features; uses independent encoders equal to the number of all sub-band features to downsample different sub-band features respectively, and obtains a low-dimensional feature vector after downsampling of each sub-band feature; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature, and obtains the discrete numerical results corresponding to all sub-band features after quantization; inputs the discrete numerical results corresponding to all sub-band features into a preset decoder, and performs upsampling and recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal. First, each sub-band feature is downsampled, and then up-sampled and recovered in combination with all down-sampled quantization results, making full use of the audio information in each sub-band feature, so that the high-frequency part of the speech is reconstructed in more detail, and the reconstruction quality of the audio signal is guaranteed.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech signal reconstruction method, characterized in that: The steps include: Obtaining the original speech signal; The original speech signal is subjected to frame division, windowing and fast Fourier transform processing in sequence to obtain a Mel-spectrogram corresponding to the original speech signal; Identify all sub-band features contained in the Mel-language spectrogram, and count the number of all sub-band features; Using independent encoders equal to the number of all sub-band features to downsample different sub-band features respectively, to obtain a low-dimensional feature vector of each sub-band feature after the downsampling process; A vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature, and the discrete numerical results corresponding to all sub-band features after quantization are obtained; The discrete numerical results corresponding to all sub-band features are respectively input into a preset decoder, and all sub-band features are up-sampled and restored by the decoder to obtain a reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

2. The speech signal reconstruction method according to claim 1, characterized in that: Before executing the step of identifying all sub-band features contained in the Mel-level spectrogram and counting the number of all sub-band features, the method further includes: According to preset segmentation parameters, the original sub-band features contained in the Mel-word spectrogram are segmented to obtain all sub-band features after segmentation based on the original sub-band features, wherein the segmentation parameters include segmentation frequency bands, the original sub-band features refer to equally spaced sub-band features originally contained in the Mel-word spectrogram, and the segmentation operation is performed by a preset bandpass filter group, and each filter in the bandpass filter group segments the original sub-band features according to different segmentation frequency bands.

3. The method for reconstructing a speech signal according to claim 2, characterized in that: Before executing the step of using independent encoders equal in number to all sub-band features to perform down-sampling processing on different sub-band features respectively to obtain a low-dimensional feature vector of each sub-band feature after the down-sampling processing, the method further includes: Get the preset downsampling parameters; Performing distinguishing numbering processing on all the sub-band features to obtain distinguishing numbering results of all the sub-band features; Performing distinguishing number processing on all independent encoders to obtain distinguishing number results of all independent encoders; Allocating independent encoders to all sub-band features according to the difference numbering results of all sub-band features and the difference numbering results of all independent encoders; After executing the step of using independent encoders equal in number to all sub-band features to perform downsampling processing on different sub-band features respectively to obtain a low-dimensional feature vector of each sub-band feature after the downsampling processing, the method further includes: For each sub-band feature, after downsampling, the low-dimensional feature vector is optimized and outputted using the RVQ algorithm to obtain an optimized output result, wherein the RVQ algorithm is a variant of VQ-VAE, which is mainly used to solve the difference problem between the encoder output and the quantized output; According to the distinguished numbering results of all the sub-band features, distinguished numbers are assigned to the optimized output results corresponding to the low-dimensional feature vectors after downsampling of each sub-band feature.

4. The method for reconstructing a speech signal according to claim 3, characterized in that: The step of using a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to all sub-band features after the quantization process specifically includes: The FSQ vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature; Obtain discrete numerical results corresponding to all sub-band features; After executing the step of using a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to all sub-band features after the quantization process, the method further includes: The discrete numerical results corresponding to all sub-band features are cached as set or array format data in numerical order, and the distinguishing number results of the corresponding sub-band features are set as the distinguishing numbers of the corresponding cached data.

5. The method for reconstructing a speech signal according to claim 3, characterized in that: Before executing the step of inputting the discrete numerical results corresponding to all sub-band features into a preset decoder, the method further includes: According to the difference numbers of all sub-band features, all cache data in set form or array format are spliced ​​to obtain a target splicing matrix; The step of inputting the discrete numerical results corresponding to all sub-band features into a preset decoder specifically includes: The target splicing matrix is ​​input into the decoder.

6. The method for reconstructing a speech signal according to claim 5, characterized in that: The step of performing up-sampling recovery processing on all sub-band features through the decoder to obtain a reconstructed speech waveform corresponding to the original speech signal specifically includes: Get the preset upsampling parameters; According to the upsampling parameters, upsampling is performed on the target splicing matrix to obtain an upsampling result; The decoder is used to reconstruct the speech signal of the up-sampling result to obtain a reconstructed speech waveform corresponding to the original speech signal.

7. The method for reconstructing a speech signal according to claim 6, characterized in that: After executing the step of performing up-sampling recovery processing on all sub-band features through the decoder to obtain a reconstructed speech waveform corresponding to the original speech signal, the method further includes: Obtaining the speech waveform before reconstruction corresponding to the original speech signal; Performing a similarity comparison between the reconstructed speech waveform and the speech waveform before reconstruction to obtain a waveform similarity comparison result; If the waveform similarity does not meet the preset similarity threshold, the segmentation parameters, downsampling parameters and upsampling parameters are adjusted, and the encoding and decoding steps are re-executed to obtain the latest reconstructed speech waveform until the waveform similarity meets the preset similarity threshold, and the parameter adjustment is completed; If the waveform similarity meets a preset similarity threshold, speech synthesis is performed using the reconstructed speech waveform.

8. A speech signal reconstruction device, characterized in that: include: An original voice signal acquisition module, used to acquire the original voice signal; The original speech signal processing module is used to perform frame segmentation, windowing and fast Fourier transform processing on the original speech signal in order to obtain a Mel spectrogram corresponding to the original speech signal; A sub-band feature statistics module is used to identify all sub-band features contained in the Mel-language spectrogram and count the number of all sub-band features; A downsampling processing module, used to perform downsampling processing on different sub-band features respectively by using independent encoders equal to the number of all sub-band features, so as to obtain a low-dimensional feature vector of each sub-band feature after the downsampling processing; A quantization processing module is used to quantize the low-dimensional feature vector corresponding to each sub-band feature using a vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features after quantization; The upsampling recovery module is used to input the discrete numerical results corresponding to all sub-band features into a preset decoder, and perform upsampling recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech signal reconstruction method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech signal reconstruction method according to any one of claims 1 to 7 are implemented.

Images