Voice processing method and device, computer equipment and storage medium

Abstract

The invention discloses a voice processing method and device, computer equipment and a storage medium, and relates to the technical field of voice processing, and the method comprises the steps: selecting a segment of a first preset frame from a target voice as a first Mel spectrum picture segment, and inputting the first Mel spectrum picture segment to a reference encoder, obtaining a target global feature output by the reference encoder; selecting a segment of a second preset frame from the target voice as a second Mel spectrum picture segment, and inputting the second Mel spectrum picture segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second Mel spectrum picture segment output by the encoding module; and inputting the target global feature and the target content representation into a decoding module to obtain a reconstructed Mel spectrum picture segment output by the decoding module. The method can be applied to intelligent voice customer service systems of financial science and technology, medical health, old-age care and the like, and the voice processing efficiency and robustness are improved.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech processing method, apparatus, computer equipment and storage medium. Background Art

[0002] In recent years, speech processing technology has made significant progress and has been widely applied in key areas such as intelligent customer service in finance, healthcare, and elderly care. With the introduction of large language models (LLMs) into speech processing, text-to-speech (TTS) technology has achieved a qualitative leap. Existing large language model-text-to-speech (LLM-TTS) methods typically treat speech processing as a next-word prediction task, relying on encoding and decoding modules for speech tokenization and waveform reconstruction.

[0003] In practice, multi-codebook speech encoding and decoding typically requires splitting the speech into two parts: the processed speech and the reference speech, and encoding and decoding them separately. This approach increases the computational load, resulting in low processing efficiency and poor stability and robustness across diverse scenarios and data. Therefore, improving speech processing efficiency and robustness has become a pressing technical challenge. Summary of the Invention

[0004] The embodiments of the present invention provide a speech processing method, apparatus, computer equipment and storage medium to solve the technical problem of low efficiency and robustness of existing speech processing.

[0005] In a first aspect, a speech processing method is provided, comprising: Selecting a segment of a first preset frame from the target speech as a first mel-spectrogram segment, and inputting the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder, wherein the target global feature includes a timbre feature and an acoustic environment feature; Selecting a segment of a second preset frame from the target speech as a second mel-spectrogram segment, and inputting the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module; The target global feature and the target content representation are input into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that fuses the target global feature and the target content representation.

[0006] In a second aspect, a speech processing device is provided, comprising: a first output module, configured to select a segment of a first preset frame from the target speech as a first mel-spectrogram segment, and input the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder, wherein the target global feature includes a timbre feature and an acoustic environment feature; a second output module, configured to select a segment of a second preset frame from the target speech as a second mel-spectrogram segment, and input the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module; a third output module, configured to input the target global feature and the target content representation into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that fuses the target global feature and the target content representation.

[0007] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech processing method described in the first aspect when executing the computer program.

[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech processing method described in the first aspect is implemented.

[0009] In one solution implemented by the above-mentioned speech processing method, apparatus, computer device, and storage medium, a segment of a first preset frame is selected from the target speech as a first mel-spectrogram segment, and the first mel-spectrogram segment is input into a reference encoder to obtain target global features output by the reference encoder, wherein the target global features include timbre features and acoustic environment features; a segment of a second preset frame is selected from the target speech as a second mel-spectrogram segment, and the second mel-spectrogram segment and the target global features are input into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module; the target global features and the target content representation are input into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that integrates the target global features and the target content representation. In this embodiment, by sharing the encoding and decoding processes for different preset frame segments of the same target speech, only a common codebook is required during the quantization process, effectively solving the technical problems of low efficiency and robustness in existing speech processing. That is, the selected first Mel-spectrogram segment is input into the reference encoder to obtain the target global features including timbre and acoustic environment features, and then combined with the selected second Mel-spectrogram segment, it is input into the encoding module to obtain the target content representation, and finally the reconstructed Mel-spectrogram segment is obtained through the decoding module. This process avoids the complex process of using multiple codebooks to process different speech separately, and only uses one common codebook and shared encoding and decoding process, which greatly reduces the consumption of computing resources and improves speech processing efficiency. At the same time, the simplified process also reduces the instability factors caused by complex processing and enhances robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0011] Figure 1 is a schematic diagram of an application environment of a speech processing method according to an embodiment of the present invention; Figure 2 is a flow chart of a speech processing method according to an embodiment of the present invention; Figure 3 is an overall flow chart of a speech processing method according to an embodiment of the present invention; Figure 4 is a flow chart of step S10 in one embodiment of the present invention; Figure 5 is a flow chart of step S20 in one embodiment of the present invention; Figure 6 is a flow chart of step S30 in one embodiment of the present invention; Figure 7 is a flow chart of step S301 in one embodiment of the present invention; Figure 8 is another flow chart of a speech processing method according to one embodiment of the present invention; Figure 9 is a schematic diagram of a speech processing device according to an embodiment of the present invention; Figure 10 FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0013] The speech processing method provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the speech processing method is applied in a speech processing system, which includes Figure 1 The client and server shown communicate over a network to improve speech processing efficiency and robustness. Clients include, but are not limited to, various personal computers, laptops, smartphones, and tablets. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0014] It should be understood that this speech processing method can be applied to scenarios such as speech conversion and speech enhancement. For example, in a speech conversion scenario, the target speech can include a source speech and a reference speech. A target content representation can be extracted from the source speech, and target global features can be extracted from the reference speech. The target content representation and target global features are then input into a decoding module to generate converted speech segments (i.e., reconstructed mel-spectrogram segments). Based on these converted speech segments, a converted speech waveform can be generated. For example, in the financial sector, bank customer service phone calls can be converted into various voice styles to enhance the user experience. In a speech enhancement scenario, target content representations (e.g., content representations of noisy speech) and target global features (e.g., global features of noise-free speech) are extracted from the target speech. The target content representation and target global features are then input into a decoding module to generate enhanced speech segments (i.e., reconstructed mel-spectrogram segments). Based on these enhanced speech segments, an enhanced speech waveform can be generated. For example, in the financial sector, bank customer service phone calls can be enhanced to achieve clearer speech to enhance the user experience.

[0015] A speech processing method provided by the present invention will be described in detail below using specific embodiments.

[0016] First, as Figure 2 and 3 As shown, a speech processing method is provided, which is applied in Figure 1 The server in the example is used as an example, and the steps are as follows: S10. Select a segment of a first preset frame from the target speech as a first mel-spectrogram segment, and input the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder, wherein the target global feature includes a timbre feature and an acoustic environment feature.

[0017] In this embodiment, the target speech may be the same speech segment including the source speech and / or the reference speech, or may be the same speech segment including two different segments, or other same speech segments.

[0018] The source speech refers to the speech part that includes core content information. For example, in a speech conversion task, if you want to convert a male voice into a female voice, then this male voice is the source speech. The speech content contained in it, such as "I want to learn about financial product A," is the content that needs to be converted into content representation, that is, the content that needs to be represented by a vector.

[0019] Reference speech refers to speech features that provide a standard for processing the source speech. For example, in a voice conversion task, if the timbre of the source speech is to be converted to that of a specific speaker, the speaker's voice is the reference speech.

[0020] The first preset frame segment refers to a preset set of continuous audio frames having a specific number.

[0021] A reference encoder is a component or model that extracts target global features such as timbre and acoustic environment characteristics from speech clips. For example, it extracts features such as Mel-Frequency Cepstral Coefficients (MFCCs) from speech clips to characterize the acoustic properties of speech. Reference encoders can be based on convolutional neural networks (CNNs), attention mechanisms, variational autoencoders (VAEs), or hybrid architectures, without limitation here.

[0022] Target global features can include timbre features, acoustic environment features, and other features. Timbre features refer to the sound quality, intonation, and speaker characteristics of the speech, such as the voice styles of different speakers. Acoustic environment features refer to the background noise and / or sound field environment of the speech signal, such as indoor and outdoor ambient noise.

[0023] As an example, a segment of a predetermined length (i.e., a segment of the first preset frame) is selected from the target speech. This segment length can be set according to actual needs. In other words, the segment of the first preset frame can be set according to actual needs. For example, a speech segment of 600 frames in length is selected. During the selection process, a short-time Fourier transform (STFT) can be performed on the target speech. Then, based on a mel-spectrogram processing method, the selected speech segment is converted into a corresponding first mel-spectrogram segment (s1). Next, this first mel-spectrogram segment is input into a reference encoder, which can extract the target global features of the speech from the first mel-spectrogram segment. Through the processing of this embodiment, accurate global feature information is provided for subsequent content encoding and speech reconstruction, thereby improving the speech processing effect.

[0024] S20. Select a segment of a second preset frame from the target speech as a second mel-spectrogram segment, and input the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module.

[0025] In this embodiment, the segment of the second preset frame also refers to a preset set of continuous audio frames having a specific number.

[0026] A coding module refers to a functional unit or model for encoding speech.

[0027] The target content representation refers to the vector representation corresponding to the text content extracted from the second mel-spectrogram segment. For example, the text content "The weather is nice today" is extracted from the second mel-spectrogram segment and encoded into a vector. This vector is the target content representation.

[0028] As a consistent example, another segment of a predetermined length (i.e., a segment of a second preset frame) is selected from the target speech signal. The length of this segment can also be set according to actual needs. That is, the segment of the second preset frame can be set according to actual needs. For example, a speech segment with a length of 200 frames is selected. In order to ensure that the selected segment can effectively represent the speech content, the short-time Fourier transform (STFT) technology is also used in combination with the Mel spectrum conversion method to obtain a second Mel spectrum image segment (s2). Then, the selected second Mel spectrum image segment is input into the encoding module, which can extract feature information related to the speech content from the second Mel spectrum image segment and convert it into the corresponding target content representation to ensure that the converted or enhanced speech can accurately express the input language content.

[0029] S30. Input the target global feature and the target content representation into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that fuses the target global feature and the target content representation.

[0030] In this embodiment, the decoding module refers to a functional unit or model for decoding speech.

[0031] As an example, after processing steps S10 and S20, the target global features and target content representation are obtained. Next, the target global features and target content representation are input to the decoding module, which generates a new mel-spectrogram segment (i.e., reconstructed mel-spectrogram segment (s2')) based on the target global features and target content representation. For example, in the financial field, the target global features may set the voice to a professional, calm, and steady male voice, and the target content representation may be a speech segment about "how to open online banking." The decoding module fuses the corresponding information of the target global features and the target content representation to generate a mel-spectrogram segment expressing "how to open online banking" in a professional, calm, and steady male voice—i.e., the reconstructed mel-spectrogram segment.

[0032] In a solution provided by an embodiment of the present invention, by sharing the encoding and decoding processes for different preset frame segments of the same target speech, only one common codebook is needed in the quantization process, which effectively solves the technical problem of low efficiency and robustness of existing speech processing. That is, the selected first mel spectrum image segment is input into the reference encoder to obtain the target global features including timbre and acoustic environment features, and then combined with the selected second mel spectrum image segment, it is input into the encoding module to obtain the target content representation, and finally the reconstructed mel spectrum image segment is obtained through the decoding module. This process avoids the complex process of using multiple codebooks to process different speech separately, and only uses one common codebook and a common encoding and decoding process, which greatly reduces the consumption of computing resources and improves speech processing efficiency. At the same time, the simplified process also reduces the instability factors caused by complex processing and enhances robustness. It should be understood that the first mel spectrum image segment and the second mel spectrum image segment of the present application are different segments in the same target speech. For example, Figure 3 Different before and after clips are shown.

[0033] In one embodiment, if Figure 4 As shown, in step S10, the reference encoder includes multiple layers of two-dimensional convolutional layers and gated recurrent unit layers; That is, inputting the first mel-spectrogram segment into a reference encoder to obtain the target global features output by the reference encoder includes the following steps: S101: Input the first mel-spectrogram segment into multiple layers of the two-dimensional convolutional layers until a target feature is obtained from the output of the last two-dimensional convolutional layer. In the process of processing the first mel-spectrogram segment by each two-dimensional convolutional layer, the output of the previous two-dimensional convolutional layer serves as the input of the next two-dimensional convolutional layer. S102: Input the target feature into the gated recurrent unit layer to obtain the target global feature output by the gated recurrent unit layer.

[0034] In this embodiment, the reference encoder includes multiple layers of two-dimensional convolutional layers and gated recurrent unit (GRU) layers, for example, six layers of two-dimensional convolutional layers and one layer of gated recurrent unit layer.

[0035] As an example, after the reference encoder receives the first mel spectrum image segment as input, it is sequentially input into multiple layers of two-dimensional convolutional layers. In the process of processing the first mel spectrum image segment by each layer of two-dimensional convolutional layer, the first layer of two-dimensional convolutional layer first performs convolution processing on the first mel spectrum image segment as input, and performs nonlinear transformation with the corresponding activation function, such as ReLU or LeakyReLU activation function, to obtain the corresponding output result. The output result is passed to the next layer of two-dimensional convolutional layer for further feature extraction until the last layer of two-dimensional convolutional layer is processed to obtain the corresponding target feature. Next, the obtained target feature is input into the gated recurrent unit layer. The function of the gated recurrent unit layer is to further convert the target features extracted by the multiple layers of two-dimensional convolutional layers into time-invariant global features, and finally output the target global features.

[0036] In one embodiment, if Figure 5 As shown, in step S20, the encoding module includes a convolutional neural network encoder, a first resampling module and a first bidirectional long short-term memory network module, wherein the convolutional neural network encoder is connected to the first resampling module, and the first resampling module is connected to the first bidirectional long short-term memory network module; That is, inputting the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module includes the following steps: S201. Input the second mel-spectrogram segment and the target global feature into the convolutional neural network encoder to obtain a potential content representation corresponding to the second mel-spectrogram segment output by the convolutional neural network encoder; S202: Input the latent content representation into the first resampling module to obtain a sampled latent content representation output by the first resampling module; S203: Input the sampled potential content representation into the first bidirectional long short-term memory network module to obtain the target content representation output by the first bidirectional long short-term memory network module.

[0037] In this embodiment, the encoding module includes a convolutional neural network encoder (e.g., a Conformer encoder), a first resampling module, and a first bidirectional long short-term memory network (BLSTM or Bi-LSTM) module. The convolutional neural network encoder is connected to the first resampling module, which is in turn connected to the first bidirectional long short-term memory network module.

[0038] The potential content representation refers to the content representation corresponding to the second mel-spectrogram segment with the target global feature removed, that is, the content representation obtained by subtracting the target global feature output by the reference encoder from the second mel-spectrogram segment.

[0039] As an example, after obtaining the second Mel spectrum image segment, first, the second Mel spectrum image segment is input into the convolutional neural network encoder, wherein the convolutional neural network encoder includes multiple convolution layers and pooling layers, and each convolution layer is equipped with a corresponding convolution kernel and activation function; the convolution layer performs a convolution operation by sliding the convolution kernel on the second Mel spectrum image segment to extract local features, and the activation function performs a nonlinear transformation on the convolution result; the pooling layer is used to reduce the feature dimension and finally obtain the potential content representation output by the convolutional neural network encoder; then, the potential content representation is input into the first sampling module to obtain the sampled potential content representation output by the first sampling module. Specifically, the first sampling module can obtain the sampled potential content representation based on the preset resampling module formula (1) and output it. Wherein, formula (1) is as follows: R Formula (1) in, Characterize the latent content representation, represents the downsampling operation, Up represents the upsampling operation, and R Characterize the latent content representation after sampling.

[0040] Furthermore, the sampled content representation is input into a first bidirectional long short-term memory network module, which is composed of a forward long short-term memory network and a reverse long short-term memory network. The first bidirectional long short-term memory network module processes the sampled potential content representation in chronological order and reverse chronological order, respectively. Through the forward and reverse temporal modeling processes, the temporal information in the sampled content representation is captured, thereby outputting the target content representation.

[0041] In one embodiment, if Figure 6 As shown, in step S30, the decoding module includes a second bidirectional long short-term memory network module, a second resampling module and a convolutional neural network decoder, wherein the second bidirectional long short-term memory network module is connected to the second resampling module, and the second resampling module is connected to the convolutional neural network decoder; That is, the step of inputting the target global feature and the target content representation into a decoding module to obtain a reconstructed mel spectrum image segment output by the decoding module includes the following steps: S301. Input the target content representation into the second bidirectional long short-term memory network module to obtain a temporal encoding result output by the second bidirectional long short-term memory network module, where the temporal encoding result is a captured temporal feature of the target content representation.

[0042] As an example, after obtaining the target content representation, such as Figure 7 As shown, first, step S3011 is executed, namely: S3011. Input the target content representation into a vector quantizer to obtain a discrete speech unit sequence output by the vector quantizer.

[0043] In this embodiment, the vector quantizer is connected to the first bidirectional long short-term memory network module and the second bidirectional long short-term memory network module respectively.

[0044] The vector quantizer maps continuous speech features (such as the target content representation of the second mel-spectrogram segment) into a finite sequence of discrete speech units. This process converts the continuous features into a sequence of discrete symbols. This process is achieved by performing lookups and quantization operations in a predefined codebook, thereby achieving a discretized representation of the speech content.

[0045] Step S3012 is further executed, namely: S3012. Input the discrete speech unit sequence into the second bidirectional long short-term memory network module to obtain the temporal coding result output by the second bidirectional long short-term memory network module.

[0046] As an example, the second bidirectional LSTM module can model the input discrete speech unit sequence using both forward and backward propagation. The forward propagation processes the discrete speech unit sequence from the beginning to the end, while the backward propagation processes the discrete speech unit sequence from the end to the beginning. Through this bidirectional modeling, the second bidirectional LSTM module can more comprehensively capture the temporal features of the input discrete speech unit sequence, thereby outputting a temporal encoding result.

[0047] Further perform the following steps S302 to S303: S302: Input the temporal encoding result into the second resampling module to obtain a reconstructed target content representation output by the second resampling module; S303. Input the target global feature and the reconstructed target content representation into the convolutional neural network decoder to obtain the reconstructed mel spectrum image segment generated by the convolutional neural network decoder.

[0048] As an example, after obtaining the temporal coding results, the temporal coding results are further input into the resampling module, so that the second resampling module generates a reconstructed target content representation based on the temporal coding results. Next, the target global features and the reconstructed target content representation are input into the convolutional neural network decoder. The convolutional neural network decoder performs a series of inverse convolution operations on the target global features and the reconstructed target content representation to ultimately generate a reconstructed mel-spectrogram segment.

[0049] In one embodiment, if Figure 8 As shown, after step S30, that is, after inputting the target global feature and the target content representation into the decoding module to obtain the reconstructed mel spectrum image segment output by the decoding module, the following steps are included: S40, inputting the reconstructed mel spectrum image segment into a discriminator to obtain a discrimination result output by the discriminator; S50: Calculate the reconstruction loss between the second mel-spectrogram segment and the reconstructed mel-spectrogram segment; S60: Obtain a representation difference between the second mel-spectrogram segment before and after quantization, where the representation before quantization is the target content representation output by the encoding module, and the representation after quantization is the discrete speech unit sequence output by the vector quantizer, and calculate an update loss between the target content representation and the discrete speech unit sequence; S70 . Determine whether to adjust target parameters of the encoding module and the decoding module according to the discrimination result, the reconstruction loss, the representation difference, and the update loss.

[0050] In this embodiment, the discriminator refers to a functional unit or model used to evaluate the similarity between the original mel-spectrogram segment and the reconstructed mel-spectrogram segment. It can also be understood as a functional unit or model used to determine whether the generated speech is similar to the real speech. For example, the discriminator determines whether the second mel-spectrogram segment (s2) and the reconstructed mel-spectrogram segment (s2') are real or fake.

[0051] Reconstruction loss is a loss function that measures the difference between the reconstructed mel-spectrogram segment and the original mel-spectrogram segment. It can also be understood as a loss function used to evaluate the difference between the generated speech and the real speech. For example, a loss function measures the difference between the second mel-spectrogram segment (s2) and the reconstructed mel-spectrogram segment (s2').

[0052] As an example, after the reconstructed mel-spectrogram segment is output by the decoding module, the reconstructed mel-spectrogram segment is first input into the discriminator for further processing. The discriminator evaluates the reconstructed mel-spectrogram segment and outputs a discrimination result. Next, the reconstruction loss between the second mel-spectrogram segment and the reconstructed mel-spectrogram segment is calculated. Then, the representation difference before and after quantization of the second mel-spectrogram segment is obtained, and the update loss between the target content representation and the discrete speech unit sequence is calculated, where the representation before quantization is the target content representation output by the encoding module, and the representation after quantization is the discrete speech unit sequence output by the vector quantizer. After the above steps are completed, based on the discrimination result, reconstruction loss, representation difference, and update loss, it is determined whether the target parameters of the encoding module and the decoding module need to be adjusted to further optimize the reconstructed mel-spectrogram segment, thereby improving the quality, accuracy, and robustness of the converted or enhanced speech, and ensuring that the generated speech is more natural and clear.

[0053] In one embodiment, step S70, that is, determining whether to adjust target parameters of the encoding module and the decoding module based on the discrimination result, the reconstruction loss, the representation difference, and the update loss, includes the following steps: S71. Obtain a total loss of the reconstructed mel-spectrogram segment according to the discrimination result, the reconstruction loss, the representation difference, and the update loss; S72: Determine whether to adjust target parameters of the encoding module and the decoding module according to the total loss.

[0054] As an example, the corresponding total loss can be calculated according to formula (2). Specifically, formula (2) is as follows: Formula (2) in, Characterize the total loss, Representation reconstruction loss, Characterize the differences, Representation update loss, Characterize the discrimination results.

[0055] When the total loss is greater than or equal to a preset threshold, the target parameters of the encoding and decoding modules are adjusted. When the total loss is less than the preset threshold, the constructed mel-spectrogram segments are input into the vocoder to obtain the target speech waveform generated by the vocoder based on the reconstructed mel-spectrogram segments. This process ensures the accuracy of speech reconstruction, ensuring that the resulting speech waveform better meets actual needs.

[0056] In one embodiment, after step S30, that is, after inputting the target global feature and the target content representation into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, the following steps are included: S80: Input the reconstructed mel-spectrogram segment into a vocoder to obtain a target speech waveform generated by the vocoder according to the reconstructed mel-spectrogram segment.

[0057] In this embodiment, a vocoder refers to a functional unit that generates a target speech waveform based on an input mel-spectrogram segment. The vocoder may be a Hi-Fi vocoder, a GAN vocoder, or other type of vocoder, which is not limited here.

[0058] As an example, after step S30, the decoding module has output a reconstructed mel-spectrogram segment. Next, the reconstructed mel-spectrogram segment is input into the vocoder, and the vocoder generates a corresponding target speech waveform based on the input reconstructed mel-spectrogram segment. For example, the goal is to convert a female voice into a male voice. After processing in step S30, the decoding module generates a reconstructed mel-spectrogram segment that matches the male voice feature. Next, step S80 inputs the reconstructed mel-spectrogram segment into the vocoder, and the vocoder generates a waveform based on the reconstructed mel-spectrogram segment. This waveform corresponds to the target male voice, and is ultimately output as a clear and natural male voice, for example, the audio related to "How to open an online bank" spoken by a male voice. The goal of this process is to convert the reconstructed mel-spectrogram segment into a real and natural speech waveform through the vocoder, thereby completing tasks such as speech conversion or enhancement.

[0059] Taking the financial sector as an example, in the intelligent customer service system for financial services, a customer calls to inquire about information such as the annualized rate of return of a financial product. The customer's original voice is relatively rapid and has a dialect accent. After processing in the previous steps, the decoding module outputs a reconstructed mel-spectrogram segment that matches the professional and clear Mandarin voice features. Subsequently, step S80 inputs this reconstructed mel-spectrogram segment into the vocoder. Based on the reconstructed mel-spectrogram segment, the vocoder generates a corresponding clear and standard Mandarin speech waveform, for example, a speech waveform of "What is the annualized rate of return of this financial product?" spoken in clear and standard Mandarin. This process, with the help of the vocoder, converts the reconstructed mel-spectrogram segment into a clear and standard Mandarin speech waveform, effectively achieving voice conversion, ensuring that the financial customer service can accurately understand the customer's needs, and improving the efficiency and quality of communication between the financial customer service and the customer.

[0060] It should be understood that step S80 may generate the corresponding speech waveform directly after obtaining the reconstructed mel-spectrogram segment, or may generate the corresponding speech waveform after executing step S72, which is not limited here.

[0061] In summary, the following is a complete example based on the description of the embodiment of steps S10 to S80 of the present invention: As an example, in the financial sector, the speech processing method described in the first aspect can be used in speech enhancement scenarios. For example, assume that a customer calls a bank customer service hotline in a noisy environment, and the speech is accompanied by background noise. First, a first mel-spectrogram segment (s1) is selected from the target speech. This first mel-spectrogram segment includes spectral information of the customer's speech and background noise. Then, the first mel-spectrogram segment is input into a reference encoder, which extracts target global features. These target global features include the timbre characteristics and acoustic environment characteristics of the customer's speech. Next, a second mel-spectrogram segment (s2) is selected from the target speech and input into an encoding module along with the extracted target global features. The encoding module converts the second mel-spectrogram segment into a target content representation, which includes the actual content and semantic information in the second mel-spectrogram segment. Next, the target global features and target content representation are input into a decoding module, which generates a reconstructed mel-spectrogram segment (s2'). This reconstructed mel-spectrogram segment retains the customer's speech content and timbre characteristics, but has the background noise eliminated or significantly reduced. Finally, the reconstructed mel-spectrogram segments are fed into the vocoder, which generates the corresponding target speech waveform based on the reconstructed mel-spectrograms. The resulting speech waveform is clear and natural, effectively removing background noise, ensuring that the customer's voice request can be accurately recognized and processed by the automatic speech recognition system. This speech enhancement technology allows banks to provide more accurate and clear automated voice services, especially in noisy environments, ensuring that customer voice requests can be accurately recognized and responded to by the system.

[0062] As another example, in the medical field, the speech processing method described in the first aspect can also be used in speech conversion scenarios. For example, in a telemedicine consultation, expert A is from the United States and communicates in English, while the patient is in China and speaks Chinese. When the patient describes his symptoms to expert A, his Chinese speech serves as the target speech. First, a first mel-spectrogram segment is selected from the patient's Chinese speech. This first mel-spectrogram segment includes the patient's speech information. Next, the first mel-spectrogram segment is input into a reference encoder to extract target global features. These target global features include the patient's timbre characteristics and the characteristics of the audio environment during the consultation. Subsequently, a second mel-spectrogram segment is selected from the target speech and input into an encoding module along with the extracted target global features. The encoding module converts the second mel-spectrogram segment into a corresponding target content representation, which contains the actual content and semantic information of the patient's description of the symptoms. The target global features, target content representation, and pre-prepared standard American English pronunciation reference features are then input into a decoding module, which converts the Chinese content into a reconstructed mel-spectrogram segment expressing the American English pronunciation conventions. Finally, the vocoder generates the corresponding English speech waveform based on the reconstructed mel-spectrogram segment, thereby effectively overcoming language communication barriers, allowing remote medical consultations to proceed smoothly, and improving the accuracy and efficiency of diagnosis.

[0063] It should be noted that the above-mentioned voice processing method can also be extended to other fields such as finance, medical care or education, and the above-mentioned is only an example and does not constitute a limitation of the present invention.

[0064] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0065] In a second aspect, a speech processing device is provided, which corresponds one-to-one to the speech processing method in the above embodiment. Figure 9 As shown, the speech processing device includes a first output module 101, a second output module 102 and a third output module 103. The functional modules are described in detail as follows: A first output module 101 is configured to select a segment of a first preset frame from the target speech as a first mel-spectrogram segment, and input the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder, wherein the target global feature includes a timbre feature and an acoustic environment feature; A second output module 102 is configured to select a segment of a second preset frame from the target speech as a second mel-spectrogram segment, and input the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module; The third output module 103 is configured to input the target global feature and the target content representation into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that integrates the target global feature and the target content representation.

[0066] In one embodiment, the reference encoder includes multiple layers of two-dimensional convolutional layers and gated recurrent unit layers; The first output module 101 is specifically configured to: Inputting the first mel-spectrogram segment into multiple layers of the two-dimensional convolutional layers until a target feature is output by the last two-dimensional convolutional layer, wherein, in the process of each two-dimensional convolutional layer processing the first mel-spectrogram segment, the output of the previous two-dimensional convolutional layer serves as the input of the next two-dimensional convolutional layer; The target feature is input into the gated recurrent unit layer to obtain the target global feature output by the gated recurrent unit layer.

[0067] In one embodiment, the encoding module includes a convolutional neural network encoder, a first resampling module, and a first bidirectional long short-term memory network module, wherein the convolutional neural network encoder is connected to the first resampling module, and the first resampling module is connected to the first bidirectional long short-term memory network module; The second output module 102 is specifically configured to: Inputting the second mel-spectrogram segment and the target global feature into the convolutional neural network encoder to obtain a potential content representation corresponding to the second mel-spectrogram segment output by the convolutional neural network encoder; Inputting the latent content representation into the first resampling module to obtain a sampled latent content representation output by the first resampling module; The sampled potential content representation is input into the first bidirectional long short-term memory network module to obtain the target content representation output by the first bidirectional long short-term memory network module.

[0068] In one embodiment, the decoding module includes a second bidirectional long short-term memory network module, a second resampling module and a convolutional neural network decoder, wherein the second bidirectional long short-term memory network module is connected to the second resampling module, and the second resampling module is connected to the convolutional neural network decoder; The second output module 103 is specifically configured to: Inputting the target content representation into the second bidirectional long short-term memory network module to obtain a temporal encoding result output by the second bidirectional long short-term memory network module, wherein the temporal encoding result is a captured temporal feature of the target content representation; Inputting the temporal encoding result into the second resampling module to obtain a reconstructed target content representation output by the second resampling module; The target global feature and the reconstructed target content representation are input into the convolutional neural network decoder to obtain the reconstructed mel spectrum image segment generated by the convolutional neural network decoder.

[0069] In one embodiment, the second output module 103 is further configured to: Inputting the target content representation into a vector quantizer to obtain a discrete speech unit sequence output by the vector quantizer; The discrete speech unit sequence is input into the second bidirectional long short-term memory network module to obtain the temporal encoding result output by the second bidirectional long short-term memory network module.

[0070] In one embodiment, the speech processing apparatus further includes: A fourth output module 104 is configured to input the reconstructed mel-spectrogram segment into a discriminator to obtain a discrimination result output by the discriminator; A calculation module 105 is configured to calculate a reconstruction loss between the second mel-spectrogram segment and the reconstructed mel-spectrogram segment; an acquisition module 106 configured to obtain a difference in representation before and after quantization of the second mel-spectrogram segment, where the representation before quantization is the target content representation output by the encoding module and the representation after quantization is the discrete speech unit sequence output by the vector quantizer, and calculate an update loss between the target content representation and the discrete speech unit sequence; The determination module 107 is configured to determine whether to adjust target parameters of the encoding module and the decoding module according to the discrimination result, the reconstruction loss, the representation difference, and the update loss.

[0071] In one embodiment, the speech processing apparatus further includes: The generating module 108 is configured to input the reconstructed mel-spectrogram segment into a vocoder to obtain a target speech waveform generated by the vocoder according to the reconstructed mel-spectrogram segment.

[0072] For the specific definition of the speech processing device, please refer to the definition of the speech processing method above and will not be repeated here. Each module in the above-mentioned speech processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the corresponding operations of each of the above modules.

[0073] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 10 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required for executing the speech processing method described in the first aspect above. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the speech processing method described in the first aspect above is implemented.

[0074] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed: Selecting a segment of a first preset frame from the target speech as a first mel-spectrogram segment, and inputting the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder, wherein the target global feature includes a timbre feature and an acoustic environment feature; Selecting a segment of a second preset frame from the target speech as a second mel-spectrogram segment, and inputting the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module; The target global feature and the target content representation are input into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that fuses the target global feature and the target content representation.

[0075] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Selecting a segment of a first preset frame from the target speech as a first mel-spectrogram segment, and inputting the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder, wherein the target global feature includes a timbre feature and an acoustic environment feature; Selecting a segment of a second preset frame from the target speech as a second mel-spectrogram segment, and inputting the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module; The target global feature and the target content representation are input into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that fuses the target global feature and the target content representation.

[0076] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0077] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0078] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A speech processing method, characterized in that: include: Selecting a segment of a first preset frame from the target speech as a first mel-spectrogram segment, and inputting the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder, wherein the target global feature includes a timbre feature and an acoustic environment feature; Selecting a segment of a second preset frame from the target speech as a second mel-spectrogram segment, and inputting the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module; The target global feature and the target content representation are input into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that fuses the target global feature and the target content representation.

2. The speech processing method according to claim 1, wherein: The reference encoder includes multiple layers of two-dimensional convolutional layers and gated recurrent unit layers; Inputting the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder includes: Inputting the first mel-spectrogram segment into multiple layers of the two-dimensional convolutional layers until a target feature is output by the last two-dimensional convolutional layer, wherein, in the process of each two-dimensional convolutional layer processing the first mel-spectrogram segment, the output of the previous two-dimensional convolutional layer serves as the input of the next two-dimensional convolutional layer; The target feature is input into the gated recurrent unit layer to obtain the target global feature output by the gated recurrent unit layer.

3. The speech processing method according to claim 1, wherein: The encoding module includes a convolutional neural network encoder, a first resampling module and a first bidirectional long short-term memory network module, wherein the convolutional neural network encoder is connected to the first resampling module, and the first resampling module is connected to the first bidirectional long short-term memory network module; Inputting the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module includes: Inputting the second mel-spectrogram segment and the target global feature into the convolutional neural network encoder to obtain a potential content representation corresponding to the second mel-spectrogram segment output by the convolutional neural network encoder; Inputting the latent content representation into the first resampling module to obtain a sampled latent content representation output by the first resampling module; The sampled potential content representation is input into the first bidirectional long short-term memory network module to obtain the target content representation output by the first bidirectional long short-term memory network module.

4. The speech processing method according to claim 1, wherein: The decoding module includes a second bidirectional long short-term memory network module, a second resampling module and a convolutional neural network decoder, wherein the second bidirectional long short-term memory network module is connected to the second resampling module, and the second resampling module is connected to the convolutional neural network decoder; The step of inputting the target global feature and the target content representation into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module includes: Inputting the target content representation into the second bidirectional long short-term memory network module to obtain a temporal encoding result output by the second bidirectional long short-term memory network module, wherein the temporal encoding result is a captured temporal feature of the target content representation; Inputting the temporal encoding result into the second resampling module to obtain a reconstructed target content representation output by the second resampling module; The target global feature and the reconstructed target content representation are input into the convolutional neural network decoder to obtain the reconstructed mel spectrum image segment generated by the convolutional neural network decoder.

5. The speech processing method according to claim 4, wherein: Inputting the target content representation into the second bidirectional long short-term memory network module to obtain a temporal encoding result output by the second bidirectional long short-term memory network module includes: Inputting the target content representation into a vector quantizer to obtain a discrete speech unit sequence output by the vector quantizer; The discrete speech unit sequence is input into the second bidirectional long short-term memory network module to obtain the temporal encoding result output by the second bidirectional long short-term memory network module.

6. The speech processing method according to claim 5, wherein: After inputting the target global feature and the target content representation into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, the method further includes: Inputting the reconstructed Mel-spectrogram segment into a discriminator to obtain a discrimination result output by the discriminator; Calculating a reconstruction loss between the second mel-spectrogram segment and the reconstructed mel-spectrogram segment; Obtaining a representation difference between the second mel-spectrogram segment before and after quantization, where the representation before quantization is the target content representation output by the encoding module and the representation after quantization is the discrete speech unit sequence output by the vector quantizer, and calculating an update loss between the target content representation and the discrete speech unit sequence; Determining whether to adjust target parameters of the encoding module and the decoding module is performed according to the discrimination result, the reconstruction loss, the representation difference, and the update loss.

7. The speech processing method according to any one of claims 1 to 6, characterized in that: After inputting the target global feature and the target content representation into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, the method further includes: The reconstructed mel-spectrogram segment is input into a vocoder to obtain a target speech waveform generated by the vocoder according to the reconstructed mel-spectrogram segment.

8. A speech processing device, characterized in that: include: a first output module, configured to select a segment of a first preset frame from the target speech as a first mel-spectrogram segment, and input the first mel-spectrogram segment into a reference encoder to obtain a target global feature output by the reference encoder, wherein the target global feature includes a timbre feature and an acoustic environment feature; a second output module, configured to select a segment of a second preset frame from the target speech as a second mel-spectrogram segment, and input the second mel-spectrogram segment and the target global feature into an encoding module to obtain a target content representation corresponding to the second mel-spectrogram segment output by the encoding module; a third output module, configured to input the target global feature and the target content representation into a decoding module to obtain a reconstructed mel-spectrogram segment output by the decoding module, wherein the reconstructed mel-spectrogram segment is a mel-spectrogram segment that fuses the target global feature and the target content representation.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the speech processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech processing method according to any one of claims 1 to 7 is implemented.

Images