A voice signal reconstruction method, device, equipment and storage medium thereof

Abstract

The embodiment of the application belongs to the technical field of speech processing, is applied to a speech signal reconstruction scene, and relates to a speech signal reconstruction method, device and equipment and a storage medium thereof. All subband features contained in a mel spectrogram are identified. Independent encoders are used to respectively perform downsampling processing on different subband features, so as to obtain a low-dimensional feature vector of each subband feature. The low-dimensional feature vector corresponding to each subband feature is quantized to obtain a discrete numerical result. The discrete numerical results corresponding to all subband features are input into a decoder for upsampling recovery processing, so as to obtain a reconstructed speech waveform corresponding to an original speech signal. The downsampling processing is first performed on each subband feature, and then the upsampling recovery is performed in combination with all downsampling quantization results, so that the audio information in each subband feature is fully utilized, the speech high-frequency part is reconstructed in a more detailed manner, and the audio synthesis quality of the intelligent speech customer service in the financial field is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and is applied to the scene of speech signal reconstruction, and particularly relates to a speech signal reconstruction method and device, equipment and a storage medium thereof. BACKGROUND

[0002] Speech processing is an important branch field in the field of artificial intelligence, wherein, the speech signal reconstruction technology is an important speech processing technology, and high-quality reconstructed speech signals are obtained so as to provide technical support for speech synthesis and ensure that the synthesized speech has high sound quality.

[0003] A vocoder is a technology for reconstructing a speech signal from acoustic features. At present, mainstream vocoders include an adversarial generative model represented by hifigan, the principle of which is to generate a final speech signal by upsampling the input acoustic features through several layers to complete the alignment of acoustic features and speech signals; and a model combining upsampling and inverse Fourier transform represented by mb_hifigan, the principle of which is to combine upsampling and inverse Fourier transform, and then generate a final speech signal by inverse Fourier transform from the output hidden variable. However, the above two reconstruction processing methods cannot fully utilize the audio information in the acoustic features, which to some extent affects the reconstruction quality of the audio signal. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a speech signal reconstruction method, device, equipment and storage medium, so as to solve the problem that the existing speech signal reconstruction processing method cannot fully utilize the audio information in the acoustic features, affecting the audio signal reconstruction quality.

[0005] In order to solve the above technical problems, the embodiments of the present application provide a speech signal reconstruction method, which adopts the following technical scheme:

[0006] A speech signal reconstruction method comprises the following steps:

[0007] Obtaining an original speech signal;

[0008] Performing frame, windowing and fast Fourier transform processing on the original speech signal in sequence to obtain a mel spectrogram corresponding to the original speech signal;

[0009] Identifying all subband features contained in the mel spectrogram, and counting the number of all subband features;

[0010] Using an independent encoder equivalent to the number of all subband features to perform downsampling processing on different subband features respectively to obtain a low-dimensional feature vector of each subband feature after downsampling processing;

[0011] The vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature, and discrete numerical results corresponding to all sub-band features after quantization are obtained.

[0012] The discrete numerical results corresponding to all sub-band features are input into a preset decoder, and the decoder is used for up-sampling recovery processing of all sub-band features, to obtain a reconstructed speech waveform corresponding to the original speech signal, thereby completing reconstruction of the original speech signal.

[0013] Further, before the step of identifying all sub-band features contained in the mel spectrogram and counting the number of all sub-band features, the method further comprises:

[0014] According to a preset segmentation parameter, the original sub-band features contained in the mel spectrogram are segmented to obtain all sub-band features after segmentation processing based on the original sub-band features, wherein the segmentation parameter includes a segmentation frequency band, the original sub-band feature refers to the originally contained equidistant sub-band feature in the mel spectrogram, and the segmentation processing operation is completed by a preset band-pass filter set. Each filter in the band-pass filter set segments the original sub-band feature according to different segmentation frequency bands.

[0015] Further, before the step of using an independent encoder equal to the number of all sub-band features to perform down-sampling processing on different sub-band features to obtain a low-dimensional feature vector of each sub-band feature after down-sampling processing, the method further comprises:

[0016] Obtaining a preset down-sampling parameter;

[0017] Differently numbering all sub-band features to obtain a differentiated numbering result of all sub-band features;

[0018] Differently numbering all independent encoders to obtain a differentiated numbering result of all independent encoders;

[0019] According to the differentiated numbering result of all sub-band features and the differentiated numbering result of all independent encoders, the independent encoders are allocated to all sub-band features;

[0020] After the step of using an independent encoder equal to the number of all sub-band features to perform down-sampling processing on different sub-band features to obtain a low-dimensional feature vector of each sub-band feature after down-sampling processing, the method further comprises:

[0021] For each sub-band feature, the low-dimensional feature vector after down-sampling processing is optimized and output by using the RVQ algorithm to obtain the optimized output result, wherein the RVQ algorithm is a variant of VQ-VAE and is mainly used to solve the difference between the encoder output and the quantization output.

[0022] According to the difference number results of all sub-band features, the optimized output results corresponding to the low-dimensional feature vectors of each sub-band feature after down-sampling processing are assigned with difference numbers.

[0023] Further, the step of quantizing the low-dimensional feature vector corresponding to each sub-band feature by using the vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features respectively, specifically includes:

[0024] The low-dimensional feature vector corresponding to each sub-band feature is quantized by using the FSQ vector quantization algorithm;

[0025] Discrete numerical results corresponding to all sub-band features respectively are obtained.

[0026] After the step of quantizing the low-dimensional feature vector corresponding to each sub-band feature by using the vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features respectively, the method further includes:

[0027] The discrete numerical results corresponding to all sub-band features respectively are cached in a set form or an array format according to the numerical order, and the difference number results of the corresponding sub-band features are set as the difference numbers of the corresponding cached data.

[0028] Further, before the step of inputting the discrete numerical results corresponding to all sub-band features respectively into the preset decoder, the method further includes:

[0029] All set form or array format cached data are spliced according to the difference numbers of all sub-band features to obtain a target splicing matrix;

[0030] The step of inputting the discrete numerical results corresponding to all sub-band features respectively into the preset decoder, specifically includes:

[0031] The target splicing matrix is input into the decoder.

[0032] Further, the step of performing up-sampling recovery processing on all sub-band features by the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, specifically includes:

[0033] An up-sampling parameter is obtained in advance;

[0034] According to the up-sampling parameter, the target stitching matrix is up-sampled to obtain an up-sampling result;

[0035] The up-sampling result is reconstructed into a speech waveform corresponding to the original speech signal by using the decoder.

[0036] Further, after the step of recovering all sub-band features by up-sampling through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, the method further comprises:

[0037] obtaining a speech waveform before reconstruction corresponding to the original speech signal;

[0038] comparing the reconstructed speech waveform with the speech waveform before reconstruction to obtain a waveform similarity comparison result;

[0039] If the waveform similarity does not satisfy a preset similarity threshold, adjusting the segmentation parameter, the down-sampling parameter and the up-sampling parameter, and re-executing the encoding and decoding steps to obtain a latest reconstructed speech waveform until the waveform similarity satisfies the preset similarity threshold, and completing parameter adjustment;

[0040] If the waveform similarity satisfies the preset similarity threshold, performing speech synthesis by using the reconstructed speech waveform.

[0041] To solve the above technical problems, the embodiment of the present application further provides a speech signal reconstruction device, which adopts the technical scheme as follows:

[0042] A speech signal reconstruction device comprises:

[0043] An original speech signal acquisition module is configured to acquire an original speech signal.

[0044] An original speech signal processing module is configured to sequentially perform frame segmentation, windowing and fast Fourier transform processing on the original speech signal to obtain a Mel spectrogram corresponding to the original speech signal.

[0045] A sub-band feature statistics module is configured to identify all sub-band features contained in the Mel spectrogram and count the number of the all sub-band features.

[0046] A down-sampling processing module is configured to use independent encoders in an amount equal to the number of the all sub-band features to respectively perform down-sampling processing on different sub-band features to obtain low-dimensional feature vectors of each sub-band feature after down-sampling processing.

[0047] The quantization module is used to quantize the low-dimensional feature vector corresponding to each sub-band feature using a vector quantization algorithm, so as to obtain the discrete numerical results corresponding to all sub-band features after quantization.

[0048] The upsampling recovery module is used to input the discrete numerical results corresponding to all sub-band features into a preset decoder, and to perform upsampling recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

[0049] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0050] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech signal reconstruction method described above.

[0051] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0052] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the speech signal reconstruction method described above.

[0053] Compared with the prior art, the embodiments of this application have the following main advantages:

[0054] The speech signal reconstruction method described in this application involves: acquiring the original speech signal; performing frame segmentation, windowing, and fast Fourier transform processing on the original speech signal to obtain a Mel spectrogram; identifying all sub-band features contained in the Mel spectrogram and counting the number of all sub-band features; using an independent encoder with an equal number of sub-band features to downsample different sub-band features to obtain a low-dimensional feature vector for each sub-band feature after downsampling; using a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to each sub-band feature after quantization; inputting the discrete numerical results corresponding to each sub-band feature into a preset decoder, and using the decoder to upsample and restore all sub-band features to obtain the reconstructed speech waveform corresponding to the original speech signal. By first downsampling each sub-band feature and then combining all downsampled and quantized results for upsampling and restoration, the method fully utilizes the audio information in each sub-band feature, resulting in more detailed reconstruction of the high-frequency part of the speech and ensuring the reconstruction quality of the audio signal. Attached Figure Description

[0055] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0057] Figure 2 This is a flowchart of an embodiment of the speech signal reconstruction method according to this application;

[0058] Figure 3 This is a flowchart of a specific embodiment of the speech signal reconstruction method described in this application, which involves allocating independent encoders;

[0059] Figure 4 This is a flowchart of a specific embodiment of the speech signal reconstruction method described in this application, which optimizes the output of the downsampling processing result;

[0060] Figure 5 yes Figure 2 A flowchart of a specific embodiment of the upsampling process in step 206 is shown;

[0061] Figure 6 yes Figure 2 The flowchart shows a specific embodiment of parameter optimization and adjustment after upsampling in step 206.

[0062] Figure 7 This is a schematic diagram of one embodiment of the speech signal reconstruction apparatus according to this application;

[0063] Figure 8 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0065] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0067] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0068] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0069] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers.

[0070] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0071] It should be noted that the speech signal reconstruction method provided in this application embodiment is generally executed by a server, and correspondingly, the speech signal reconstruction device is generally located in the server.

[0072] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0073] Continue to refer to Figure 2 A flowchart of an embodiment of the speech signal reconstruction method according to this application is shown. The speech signal reconstruction method includes the following steps:

[0074] Step 201: Obtain the original speech signal.

[0075] Specifically, the original speech signal refers to the speech signal directly obtained from the original speech content of the target speech object, for example, the speech signal directly obtained from a segment of speech content of the target speech object.

[0076] Specifically, the target speech object is, for example, a human agent in the financial field with a pleasant voice, and the original speech signal is obtained directly from the original speech content of the aforementioned human agent.

[0077] Step 202: The original speech signal is processed by framing, windowing and fast Fourier transform to obtain the Mel speech spectrogram corresponding to the original speech signal.

[0078] Specifically, the Fast Fourier Transform (FFT) is used to transform a signal from the time domain to the frequency domain. The Mel spectrogram refers to the Mel feature map.

[0079] The Mel spectrogram differs from other acoustic features in that it consists of equally spaced original subbands.

[0080] By sequentially performing frame segmentation, windowing, and fast Fourier transform processing on the original speech signal corresponding to the aforementioned human agent, the Mel speech spectrogram corresponding to the original speech signal is obtained, so as to facilitate the subsequent acquisition of the reconstructed speech waveform corresponding to the human agent.

[0081] Step 203: Identify all subband features contained in the Mel language spectrogram and count the number of all subband features.

[0082] By counting the number of features in all sub-bands, it is possible to prepare an equal number of independent encoders in the future.

[0083] Step 204: Use an independent encoder with an equal number of sub-band features to perform downsampling processing on different sub-band features to obtain a low-dimensional feature vector of each sub-band feature after downsampling processing.

[0084] Specifically, in this embodiment, the downsampling process refers to reducing the sampling rate of a high-resolution audio signal in speech signal processing, thereby reducing the amount of data to facilitate storage, transmission, or subsequent processing.

[0085] Step 205: Use vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to all sub-band features after quantization.

[0086] Specifically, the Fast Fourier Transform (FFT) is a type of Discrete Fourier Transform. The resulting speech signal is a discrete speech signal. At this point, a vector quantization algorithm is used to quantize the low-dimensional feature vectors corresponding to each sub-band feature to obtain a discrete numerical set corresponding to each sub-band feature, which is then used for subsequent processing.

[0087] Step 206: Input the discrete numerical results corresponding to all sub-band features into a preset decoder, and perform upsampling recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, thus completing the reconstruction of the original speech signal.

[0088] Specifically, the upsampling restoration process refers to the process of converting a low-sampling-rate audio signal into a high-sampling-rate audio signal. This is achieved by inserting additional sample points to increase the sampling rate. Specifically, it inserts new sample points between existing sample points, resulting in more sample points between each existing sample point. This allows the audio signal to be played at a higher sampling rate, theoretically improving sound quality.

[0089] In this embodiment, the upsampling restoration process can be understood as follows: since the low-dimensional feature vectors corresponding to each sub-band feature are quantized to obtain a large number of discrete numerical results, during upsampling restoration, sampling points can be selected from all the discrete numerical results according to the upsampling parameters, thus ensuring the quality of audio reconstruction.

[0090] Specifically, the speech signal reconstruction method described in this application can be applied to the speech reconstruction scenario of financial customer service. For example, it can be used to obtain a customer service speech segment with a relatively high-quality voice tone and extract the original speech signal; then, the speech signal reconstruction method is used to obtain the reconstructed speech waveform. Overall, the speech signal reconstruction method is a preprocessing step for actual speech synthesis. Subsequently, when combining intelligent voice customer service for speech synthesis, speech synthesis processing is performed directly through the speech script text and the reconstructed speech waveform to improve the voice tone quality of intelligent voice customer service, making the communication voice of intelligent voice customer service closer to the aforementioned customer service with a relatively high-quality voice tone, thus ensuring a better service experience for customers.

[0091] In this embodiment, the original speech signal is acquired; the original speech signal is then processed sequentially by framing, windowing, and Fast Fourier Transform to obtain a Mel spectrogram; all sub-band features contained in the Mel spectrogram are identified, and the number of all sub-band features is counted; an independent encoder with an equal number of sub-band features is used to downsample different sub-band features to obtain a low-dimensional feature vector for each sub-band feature after downsampling; a vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to each sub-band feature after quantization; the discrete numerical results corresponding to each sub-band feature are input into a preset decoder, and the decoder performs upsampling recovery processing on all sub-band features to obtain the reconstructed speech waveform corresponding to the original speech signal. By first downsampling each sub-band feature and then combining all downsampling and quantization results for upsampling recovery, the audio information in each sub-band feature is fully utilized, resulting in more detailed reconstruction of the high-frequency part of the speech and ensuring the reconstruction quality of the audio signal.

[0092] In this embodiment, before performing the steps of identifying all sub-band features contained in the Mel spectrogram and counting the number of all sub-band features, the method further includes: segmenting the original sub-band features contained in the Mel spectrogram according to preset segmentation parameters to obtain all sub-band features after segmentation based on the original sub-band features. The segmentation parameters include segmentation frequency bands, the original sub-band features refer to the equally spaced sub-band features initially contained in the Mel spectrogram, and the segmentation operation is performed by a preset bandpass filter bank. Each filter in the bandpass filter bank segments the original sub-band features according to different segmentation frequency bands.

[0093] By further segmenting the original sub-band features contained in the Mel spectrogram, more refined sub-band features are obtained, ensuring that the more refined sub-band features are fully utilized during subsequent audio signal reconstruction, thereby making full use of more audio information for audio signal reconstruction.

[0094] Continue to refer to Figure 3 In some optional implementations, a step of assigning an independent encoder is included before step 204. Figure 3 This is a flowchart of a specific embodiment of the speech signal reconstruction method described in this application, which involves assigning independent encoders, and includes the following steps:

[0095] Step 301: Obtain the pre-set downsampling parameters;

[0096] Step 302: Perform a distinguishing numbering process on all the sub-band features to obtain the distinguishing numbering results for all the sub-band features;

[0097] Step 303: Perform a distinguishing numbering process on all independent encoders to obtain the distinguishing numbering results for all independent encoders;

[0098] Step 304: Based on the differentiation numbering results of all sub-band features and the differentiation numbering results of all independent encoders, assign independent encoders to all sub-band features.

[0099] By combining the differentiation numbering results of all sub-band features with the differentiation numbering results of all independent encoders, all sub-band features are assigned to independent encoders, ensuring high controllability of the encoding process and enabling each independent encoder to process only sub-band features of a specific frequency band.

[0100] Continue to refer to Figure 4 In some optional implementations, step 204 is followed by a step of optimizing the output of the downsampling result. Figure 4 This is a flowchart of a specific embodiment of the speech signal reconstruction method described in this application, which optimizes the output of the downsampling processing result, and includes the following steps:

[0101] Step 401: For each sub-band feature, after downsampling, the low-dimensional feature vector is optimized using the RVQ algorithm to obtain the optimized output result.

[0102] Among them, the RVQ (Residual Vector Quantization) algorithm is a variant of VQ-VAE (Vector Quantized Variational Autoencoder), mainly used to solve the problem of the difference between encoder output and quantization output.

[0103] Step 402: Based on the difference numbering results of all sub-band features, assign a difference number to the optimized output result corresponding to the low-dimensional feature vector after downsampling of each sub-band feature.

[0104] By optimizing the output of the downsampling process, we can ensure that the low-dimensional feature vectors of each sub-band feature after downsampling are more in line with the quantization requirements, which will facilitate subsequent processing.

[0105] In this embodiment, the step of using a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature and obtaining discrete numerical results corresponding to all sub-band features after quantization specifically includes: using the FSQ vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature; and obtaining discrete numerical results corresponding to all sub-band features.

[0106] Among them, the FSQ (Finite Scalar Quantization) vector quantization algorithm is an optimization scheme to replace the vector quantization in VQ-VAE (Vector Quantized Variational Autoencoder). It does not require an additional auxiliary loss function to stretch the codeword vector and the unquantized representation vector, which simplifies the training process and improves the encoding utilization.

[0107] In this embodiment, after performing the step of quantizing the low-dimensional feature vector corresponding to each sub-band feature using the vector quantization algorithm to obtain the discrete numerical results corresponding to all sub-band features after quantization, the method further includes: caching the discrete numerical results corresponding to all sub-band features into set or array format data according to the numerical order, and setting the difference number result of the corresponding sub-band feature as the difference number of the corresponding cached data.

[0108] The low-dimensional feature vector corresponding to each sub-band feature is quantized using a vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features after quantization. The difference number of the sub-band feature is set as the difference number of the corresponding cached data, which facilitates the retrieval of the discrete numerical results before subsequent decoding.

[0109] In this embodiment, before performing the step of inputting the discrete numerical results corresponding to all sub-band features into a preset decoder, the method further includes: concatenating all cached data in set or array format according to the difference number of all sub-band features to obtain a target concatenation matrix.

[0110] In this embodiment, the step of inputting the discrete numerical results corresponding to all sub-band features into a preset decoder specifically includes: inputting the target splicing matrix into the decoder.

[0111] By concatenating all cached data in set or array format according to the distinguishing numbers of all sub-band features, a target concatenation matrix is ​​obtained. This target concatenation matrix is ​​then input into the decoder, which concatenates the discrete numerical results according to the different frequency bands corresponding to the sub-band features. This ensures that more sampling points can be used during subsequent decoding, resulting in more detailed reconstruction of the high-frequency part of the speech and guaranteeing the reconstruction quality of the audio signal. Applying this concatenation of discrete numerical results according to the different frequency bands corresponding to the sub-band features to the pre-processing of speech synthesis in the financial field ensures the reconstruction quality of the audio signal corresponding to human agents, thereby improving the communication quality of subsequent intelligent voice customer service.

[0112] Continue to refer to Figure 5 , Figure 5 yes Figure 2 A flowchart of a specific embodiment of the upsampling process in step 206 is shown, including the following steps:

[0113] Step 501: Obtain the pre-set upsampling parameters;

[0114] The upsampling parameters include sampling rate, bit depth, and number of channels.

[0115] Step 502: Based on the upsampling parameters, perform upsampling processing on the target splicing matrix to obtain the upsampling result;

[0116] Step 503: Use the decoder to reconstruct the speech signal from the upsampled result to obtain the reconstructed speech waveform corresponding to the original speech signal.

[0117] Continue to refer to Figure 6 , Figure 6 yes Figure 2 The flowchart shown in step 206 illustrates a specific embodiment of parameter optimization and adjustment after upsampling, including the following steps:

[0118] Step 601: Obtain the unreconstructed speech waveform corresponding to the original speech signal;

[0119] Step 602: Compare the similarity between the reconstructed speech waveform and the unreconstructed speech waveform to obtain the waveform similarity comparison result;

[0120] Step 603: If the waveform similarity does not meet the preset similarity threshold, adjust the segmentation parameters, downsampling parameters, and upsampling parameters, and re-execute the encoding and decoding steps to obtain the latest reconstructed speech waveform until the waveform similarity meets the preset similarity threshold, thus completing the parameter adjustment.

[0121] Specifically, the re-execution of the encoding and decoding steps means that after adjusting the segmentation parameters, downsampling parameters, and upsampling parameters, the independent encoders in steps 204 to 206 are re-executed for downsampling. Since the segmentation parameters have changed, the number of independent encoders has also changed accordingly, and the downsampling parameters and upsampling parameters have also changed again. Therefore, it is equivalent to reconstructing the original speech signal.

[0122] Step 604: If the waveform similarity meets the preset similarity threshold, then the reconstructed speech waveform is used for speech synthesis.

[0123] An upsampling method is used to sample discrete numerical results corresponding to all sub-band features and reconstruct the audio signal. In addition, the segmentation parameters, downsampling parameters and upsampling parameters are adjusted by combining parameter optimization adjustment methods to fully ensure the reconstruction quality of the audio signal.

[0124] This application obtains the original speech signal; performs frame segmentation, windowing, and Fast Fourier Transform on the original speech signal to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram and counts the number of all sub-band features; uses an independent encoder with an equal number of sub-band features to downsample different sub-band features to obtain a low-dimensional feature vector for each sub-band feature after downsampling; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to each sub-band feature after quantization; inputs the discrete numerical results corresponding to each sub-band feature into a preset decoder, and uses the decoder to upsample and restore all sub-band features to obtain the reconstructed speech waveform corresponding to the original speech signal. By first downsampling each sub-band feature and then combining all downsampled and quantized results for upsampling and restoration, the audio information in each sub-band feature is fully utilized, resulting in more detailed reconstruction of the high-frequency part of the speech and ensuring the reconstruction quality of the audio signal. Overall, this speech signal reconstruction method is a preprocessing step for actual speech synthesis. Subsequently, when combining with intelligent voice customer service for speech synthesis, speech synthesis processing is performed directly through the speech script text and the reconstructed speech waveform to improve the voice quality of intelligent voice customer service. This makes the communication voice of intelligent voice customer service closer to that of a customer service representative with a high-quality voice, ensuring a better service experience for customers.

[0125] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0126] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0127] In this embodiment, the original speech signal is acquired; the original speech signal is then processed sequentially by framing, windowing, and Fast Fourier Transform to obtain a Mel spectrogram; all sub-band features contained in the Mel spectrogram are identified, and the number of all sub-band features is counted; an independent encoder with an equal number of sub-band features is used to downsample different sub-band features to obtain a low-dimensional feature vector for each sub-band feature after downsampling; a vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to each sub-band feature after quantization; the discrete numerical results corresponding to each sub-band feature are input into a preset decoder, and the decoder performs upsampling recovery processing on all sub-band features to obtain the reconstructed speech waveform corresponding to the original speech signal. By first downsampling each sub-band feature and then combining all downsampling and quantization results for upsampling recovery, the audio information in each sub-band feature is fully utilized, resulting in more detailed reconstruction of the high-frequency part of the speech and ensuring the reconstruction quality of the audio signal.

[0128] Further reference Figure 7 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a speech signal reconstruction apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0129] like Figure 7 As shown, the speech signal reconstruction device 700 described in this embodiment includes: an original speech signal acquisition module 701, an original speech signal processing module 702, a sub-band feature statistics module 703, a downsampling processing module 704, a quantization processing module 705, and an upsampling recovery module 706. Wherein:

[0130] The raw speech signal acquisition module 701 is used to acquire the raw speech signal;

[0131] The original speech signal processing module 702 is used to perform frame segmentation, windowing and fast Fourier transform processing on the original speech signal in sequence to obtain the Mel speech spectrogram corresponding to the original speech signal;

[0132] The sub-band feature statistics module 703 is used to identify all sub-band features contained in the Mel language spectrogram and count the number of all sub-band features;

[0133] The downsampling processing module 704 is used to perform downsampling processing on different sub-band features using an independent encoder with an equal number of sub-band features to obtain a low-dimensional feature vector of each sub-band feature after downsampling.

[0134] The quantization processing module 705 is used to quantize the low-dimensional feature vector corresponding to each sub-band feature using a vector quantization algorithm to obtain discrete numerical results corresponding to all sub-band features after quantization.

[0135] The upsampling recovery module 706 is used to input the discrete numerical results corresponding to all sub-band features into a preset decoder, and to perform upsampling recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

[0136] This application obtains the original speech signal; performs frame segmentation, windowing, and Fast Fourier Transform on the original speech signal to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram and counts the number of all sub-band features; uses an independent encoder with an equal number of sub-band features to downsample different sub-band features to obtain a low-dimensional feature vector for each sub-band feature after downsampling; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature to obtain discrete numerical results corresponding to each sub-band feature after quantization; inputs the discrete numerical results corresponding to each sub-band feature into a preset decoder, and uses the decoder to upsample and restore all sub-band features to obtain the reconstructed speech waveform corresponding to the original speech signal. By first downsampling each sub-band feature and then combining all downsampled and quantized results for upsampling and restoration, the audio information in each sub-band feature is fully utilized, resulting in more detailed reconstruction of the high-frequency part of the speech and ensuring the reconstruction quality of the audio signal.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0138] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0139] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0140] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c that are interconnected via a system bus. It should be noted that... Figure 8 Only a computer device 8 with component memory 8a, processor 8b, and network interface 8c is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0141] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0142] The memory 8a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 8a may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 8a may also be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the memory 8a may include both the internal storage unit and its external storage device of the computer device 8. In this embodiment, the memory 8a is typically used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions for a speech signal reconstruction method. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or will be output.

[0143] In some embodiments, the processor 8b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 8b is typically used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to execute computer-readable instructions stored in the memory 8a or to process data, such as executing computer-readable instructions for the speech signal reconstruction method.

[0144] The network interface 8c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 8 and other electronic devices.

[0145] The computer device proposed in this embodiment belongs to the field of speech processing technology and is applied in speech signal reconstruction scenarios. This application acquires the original speech signal; performs framing, windowing, and fast Fourier transform processing on the original speech signal to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram and counts the number of all sub-band features; uses an independent encoder with an equal number of sub-band features to downsample different sub-band features, obtaining a low-dimensional feature vector for each sub-band feature after downsampling; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature, obtaining discrete numerical results corresponding to each sub-band feature after quantization; inputs the discrete numerical results corresponding to each sub-band feature into a preset decoder, and uses the decoder to upsample and recover all sub-band features to obtain the reconstructed speech waveform corresponding to the original speech signal. By first downsampling each sub-band feature and then combining all downsampled and quantized results for upsampling and recovery, the audio information in each sub-band feature is fully utilized, resulting in more detailed reconstruction of the high-frequency part of the speech and ensuring the reconstruction quality of the audio signal.

[0146] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the speech signal reconstruction method described above.

[0147] The computer-readable storage medium proposed in this embodiment belongs to the field of speech processing technology and is applied in speech signal reconstruction scenarios. This application acquires the original speech signal; performs framing, windowing, and fast Fourier transform processing on the original speech signal to obtain a Mel spectrogram; identifies all sub-band features contained in the Mel spectrogram and counts the number of all sub-band features; uses an independent encoder with an equal number of sub-band features to downsample different sub-band features, obtaining a low-dimensional feature vector for each sub-band feature after downsampling; uses a vector quantization algorithm to quantize the low-dimensional feature vector corresponding to each sub-band feature, obtaining discrete numerical results corresponding to each sub-band feature after quantization; inputs the discrete numerical results corresponding to each sub-band feature into a preset decoder, and uses the decoder to upsample and recover all sub-band features to obtain the reconstructed speech waveform corresponding to the original speech signal. By first downsampling each sub-band feature and then combining all downsampled and quantized results for upsampling and recovery, the audio information in each sub-band feature is fully utilized, resulting in more detailed reconstruction of the high-frequency part of the speech and ensuring the reconstruction quality of the audio signal.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0149] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for reconstructing speech signals, characterized in that, Includes the following steps: Acquire the raw speech signal; The original speech signal is sequentially processed by framing, windowing, and fast Fourier transform to obtain the Mel spectrogram corresponding to the original speech signal; Identify all subband features contained in the Mel language spectrogram and count the number of all subband features; An independent encoder, with an equal number of sub-band features to all sub-band features, is used to downsample different sub-band features to obtain a low-dimensional feature vector for each sub-band feature after downsampling. The FSQ vector quantization algorithm is used to quantize the low-dimensional feature vector corresponding to each sub-band feature, and the discrete numerical results corresponding to all sub-band features after quantization are obtained. The FSQ vector quantization algorithm does not require the introduction of a loss function to stretch the codeword vector and the unquantized representation vector. For the discrete numerical results corresponding to all sub-band features, cache them as set or array data in numerical order, and set the difference number of the corresponding sub-band feature as the difference number of the corresponding cached data. All cached data in set or array format are concatenated according to the distinguishing numbers of all sub-band features to obtain the target concatenation matrix; The discrete numerical results corresponding to all sub-band features are input into a preset decoder, and the decoder performs upsampling recovery processing on all sub-band features to obtain the reconstructed speech waveform corresponding to the original speech signal, thus completing the reconstruction of the original speech signal. Specifically, the step of inputting the discrete numerical results corresponding to all sub-band features into the preset decoder includes: inputting the target splicing matrix into the decoder.

2. The speech signal reconstruction method according to claim 1, characterized in that, Before performing the steps of identifying all subband features contained in the Mel language spectrogram and counting the number of all subband features, the method further includes: According to preset segmentation parameters, the original sub-band features contained in the Mel spectrogram are segmented to obtain all sub-band features after segmentation based on the original sub-band features. The segmentation parameters include segmentation frequency bands, and the original sub-band features refer to the equally spaced sub-band features initially contained in the Mel spectrogram. The segmentation operation is performed by a preset bandpass filter bank, and each filter in the bandpass filter bank segments the original sub-band features according to different segmentation frequency bands.

3. The speech signal reconstruction method according to claim 2, characterized in that, Before performing the step of downsampling different sub-band features using an independent encoder of equal number to the number of all sub-band features to obtain a low-dimensional feature vector for each sub-band feature after downsampling, the method further includes: Obtain the pre-set downsampling parameters; All sub-band features are distinguished and numbered to obtain the distinguished numbering results of all sub-band features; Perform a distinguishable numbering process on all independent encoders to obtain the distinguishable numbering results for all independent encoders; Based on the differentiation numbering results of all sub-band features and the differentiation numbering results of all independent encoders, independent encoders are assigned to all sub-band features; After performing the step of downsampling different sub-band features using an independent encoder of equal number to the number of all sub-band features to obtain a low-dimensional feature vector for each sub-band feature after downsampling, the method further includes: For each subband feature, after downsampling, the low-dimensional feature vector is optimized using the RVQ algorithm to obtain the optimized output result. The RVQ algorithm is a variant of VQ-VAE, mainly used to solve the difference between encoder output and quantization output. Based on the distinguishing numbering results of all sub-band features, the optimized output result corresponding to the low-dimensional feature vector after downsampling of each sub-band feature is assigned a distinguishing number.

4. The speech signal reconstruction method according to claim 1, characterized in that, The step of upsampling and restoring all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal specifically includes: Obtain the pre-set upsampling parameters; Based on the upsampling parameters, the target splicing matrix is ​​upsampled to obtain the upsampling result; The decoder is used to reconstruct the speech signal from the upsampled result to obtain the reconstructed speech waveform corresponding to the original speech signal.

5. The speech signal reconstruction method according to claim 4, characterized in that, After performing the step of upsampling and recovering all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, the method further includes: Obtain the unreconstructed speech waveform corresponding to the original speech signal; The similarity between the reconstructed speech waveform and the unreconstructed speech waveform is compared to obtain the waveform similarity comparison result; If the waveform similarity does not meet the preset similarity threshold, the segmentation parameters, downsampling parameters, and upsampling parameters are adjusted, and the encoding and decoding steps are re-executed to obtain the latest reconstructed speech waveform until the waveform similarity meets the preset similarity threshold, thus completing the parameter adjustment. If the waveform similarity meets the preset similarity threshold, then the reconstructed speech waveform is used for speech synthesis.

6. A speech signal reconstruction device, characterized in that, The speech signal reconstruction apparatus implements the steps of the speech signal reconstruction method as described in any one of claims 1 to 5, wherein the speech signal reconstruction apparatus comprises: The raw speech signal acquisition module is used to acquire the raw speech signal; The original speech signal processing module is used to perform frame segmentation, windowing and fast Fourier transform processing on the original speech signal in sequence to obtain the Mel speech spectrogram corresponding to the original speech signal; The sub-band feature statistics module is used to identify all sub-band features contained in the Mel language spectrogram and count the number of all sub-band features. The downsampling processing module is used to perform downsampling processing on different sub-band features using an independent encoder with an equal number of sub-band features to obtain a low-dimensional feature vector of each sub-band feature after downsampling. The quantization module is used to quantize the low-dimensional feature vector corresponding to each sub-band feature using a vector quantization algorithm, so as to obtain the discrete numerical results corresponding to all sub-band features after quantization. The upsampling recovery module is used to input the discrete numerical results corresponding to all sub-band features into a preset decoder, and to perform upsampling recovery processing on all sub-band features through the decoder to obtain the reconstructed speech waveform corresponding to the original speech signal, thereby completing the reconstruction of the original speech signal.

7. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech signal reconstruction method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech signal reconstruction method as described in any one of claims 1 to 5.

Images