A method, device, equipment and medium for separating speech and background sound based on Codec

Abstract

The present invention discloses a method, device, equipment and medium for separating speech and background sound based on Codec, and relates to the technical field of speech noise separation. The method includes a separation model construction process, a model training process and an audio separation process. By constructing and training a codec-based encoding and decoding separation model, including an encoder, a separation module, two audio representation quantizers and a decoder, audio separation in the feature representation space is achieved. The present invention provides a method, device, equipment and medium for separating speech and background sound based on Codec, which takes Codec technology as the core. By separating speech and background sound in the representation space, the computational burden and separation error caused by direct processing at the waveform or spectrum level are avoided, and the problem that traditional separation methods are difficult to balance separation performance and audio quality under low bit rate compression conditions is successfully solved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech noise separation, and in particular to a method, device, equipment and medium for separating speech and background sound based on a Codec. Background Art

[0002] In the fields of speech processing and audio signal processing, separating speech from background sound has long been a crucial research topic. In daily life, humans are frequently exposed to mixed audio, which includes both human voices (foreground sound) and environmental noise, background music, and other components (background sound). Efficiently and accurately separating human voices from background sound is crucial for many applications, such as voice communication, speech recognition, hearing assistance devices, music production, and audio enhancement. In remote video conferencing and voice calls, clear human voice extraction effectively improves speech intelligibility and communication quality; in hearing assistance devices, removing background noise helps the hearing-impaired hear speech more accurately; and in music production and film and television post-processing, the ability to accurately separate human voices from background sound facilitates mixing and noise reduction. However, traditional speech enhancement and separation methods still face challenges when working with complex background environments, such as the variability of background noise, the diversity of speakers, and limited computing resources.

[0003] In recent years, with the development of deep learning technology, neural network-based speech processing methods have made significant progress. Currently, speech separation primarily relies on deep learning methods such as time-frequency masking (TF-masking), self-supervised learning, and end-to-end speech separation models. These methods have achieved promising results at high bitrates, with performance in tasks such as speech separation and speech enhancement continuously improving. However, in low-bitrate compression and transmission scenarios, traditional neural network methods often suffer from degraded separation performance due to information loss. Therefore, existing methods struggle to balance speech quality and separation performance under resource-constrained conditions. Furthermore, existing neural network speech separation models are primarily constructed and evaluated at high bitrates and lack adaptability to compressed speech features, limiting their practicality at low bitrates. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a Codec-based speech and background sound separation method, device, equipment and medium. With Codec technology as the core, by separating speech and background sound in the representation space, the computational burden and separation error caused by direct processing at the waveform or spectrum level are avoided, and the problem that traditional separation methods are difficult to strike a balance between separation performance and audio quality under low bit rate compression conditions is successfully solved.

[0005] In a first aspect, the present invention provides a method for separating speech and background sound based on a Codec, comprising:

[0006] Separation model construction process: Construct a codec-based codec separation model, including an encoder, a separation module, two audio representation quantizers and a decoder; the encoder is used to encode audio into a continuous audio representation; the separation module is used to separate the continuous audio representation into a hidden representation containing speech and a hidden representation containing background sound, and the two audio representation quantizers are used to discretize the hidden representation containing speech and the hidden representation containing background sound, respectively, and compress them to a low bit rate level to obtain a compressed discrete speech representation and a compressed discrete background sound representation; the decoder is used to restore the compressed discrete speech representation and the compressed discrete background sound representation to the separated speech signal and background sound signal, thereby realizing audio separation in the feature representation space;

[0007] Model training process: Randomly select pure speech samples and superimpose background sound to generate mixed input audio. After audio separation using the codec-based codec separation model, the speech signal and background sound signal are obtained, and the speech signal and background sound signal are mixed to obtain the predicted mixed audio. The first and second discriminant modules are used to evaluate the model's reconstruction quality of the speech and background sound after separation, respectively. The third discriminant module is also used to evaluate the reconstruction quality of the predicted mixed audio. The codec-based codec separation model is trained until completion.

[0008] Audio separation process: Use the trained Codec-based codec separation model to separate the target audio and quantitatively evaluate the voice quality of the model separation output.

[0009] Furthermore, the separation module includes two groups of nonlinear layers, and each group of nonlinear layers includes a plurality of nonlinear layers.

[0010] Furthermore, the process of generating mixed input audio is as follows: obtaining an audio signal containing a single or multiple voices and an audio signal containing only background sound , through volume normalization and signal superposition, it is mixed into an audio signal of voice and background sound, that is, mixed input audio. The formula is as follows: ; ;

[0011] Where T is the number of audio sampling points.

[0012] Furthermore, both audio representation quantizers use FSQ quantizers, and the quantization process is as follows: discretizing the hidden representation containing speech and the hidden representation containing background sound, respectively, and compressing them to a low bit rate level, converting each scalar contained in the X tensor to a range from 0 to L-1, so that the total number of vectors composed of scalars is finite, as shown in the following formula: ;

[0013] in, is the rounding function, , L is the discretization level, d is the dimension of each vector in X, then is the vocabulary size of the discretized vector.

[0014] Furthermore, when the input audio sampling rate is 24k, the quantization vocabulary of the FSQ quantizer is set to [8,8,8,8,5,5,5,5,5,5,5,5,5,5,5], and the latent representation is compressed to a bit rate level of 3 kbps.

[0015] Furthermore, the first discrimination module, the second discrimination module and the third discrimination module respectively include a multi-scale Fourier discriminator and a multi-resolution short-time Fourier transform discriminator.

[0016] Furthermore, the perceptual speech quality evaluation index is used to quantitatively evaluate the speech quality of the model separation output.

[0017] In a second aspect, the present invention provides a speech and background sound separation device of a Codec, including a codec-based codec separation model, the codec-based codec separation model including an encoder, a separation module, two audio representation quantizers and a decoder; the encoder is used to encode the audio into a continuous audio representation; the separation module is used to separate the continuous audio representation into a hidden representation containing speech and a hidden representation containing background sound, the two audio representation quantizers are used to discretize the hidden representation containing speech and the hidden representation containing background sound, and compress them to a low bit rate level to obtain a compressed discrete speech representation and a compressed discrete background sound representation; the decoder is used to restore the compressed discrete speech representation and the compressed discrete background sound representation to the separated speech signal and background sound signal, thereby realizing audio separation in the feature representation space.

[0018] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the program is executed by a processor.

[0020] The technical solutions provided in the embodiments of the present invention have at least the following technical effects:

[0021] This paper proposes a method for separating speech and background sound in low-bitrate environments. This method extends the end-to-end neural audio codec framework, Codec, to speech separation tasks for the first time. This method not only retains the modeling advantages of neural codec models at high compression ratios, but also achieves collaborative modeling of compression and separation tasks through systematic optimization of internal modules. This significantly improves the separation quality and robustness at low bitrates, resolving the problem that traditional separation methods struggle to balance separation performance and audio quality under low-bitrate compression conditions. Compared to existing methods, this invention has at least the following advantages:

[0022] 1. Starting from the model architecture design, it innovatively solves the problem of easy confusion between speech and background sound and limited feature expression under low bit rate conditions, providing a more practical solution for application scenarios such as remote communication, speech recognition preprocessing, and hearing aid design.

[0023] 2. At a low bitrate of 3kbps, although the PESQ of the separated speech is 1.853, slightly lower than the single audio reconstruction result (1.901) at the same bitrate, the speech quality remains good compared to the complexity of the task, indicating that this method is still practical and robust in resource-constrained scenarios.

[0024] 3. Targeted structural improvements have been made to the Codec model for separation scenarios under low bit rates, expanding the application boundaries of the Codec model in the field of audio processing.

[0025] 4. It has important theoretical significance and broad application prospects in the integration of audio compression and multi-task speech processing, and has significant innovation and promotion value.

[0026] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] Figure 1 This is a schematic diagram of the overall framework of the Codec in the prior art;

[0029] Figure 2 is a flow chart of the method in embodiment 1 of the present invention;

[0030] Figure 3 Schematic diagram of the structure of the codec-based encoding and decoding separation model in Example 1 of the present invention;

[0031] Figure 4Schematic diagram of the model training process in Example 1 of the present invention;

[0032] Figure 5 Schematic diagram of exploring the optimal model architecture in the first embodiment of the present invention;

[0033] Figure 6 Schematic diagram of the FSQ quantization principle in the first embodiment of the present invention;

[0034] Figure 7 This is a schematic structural diagram of an electronic device in a third embodiment of the present invention;

[0035] Figure 8 Schematic diagram of the structure of the medium in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present invention provide a Codec-based method, apparatus, device, and medium for separating speech and background sound. With Codec technology as the core, this method separates speech and background sound in a representational space, avoiding the computational burden and separation errors caused by direct processing at the waveform or spectrum level. This successfully addresses the difficulty of traditional separation methods in balancing separation performance and audio quality under low-bitrate compression conditions.

[0037] The technical solution in the embodiment of the present invention has the following general ideas:

[0038] The Codec model focuses on end-to-end neural audio coding and decoding technology, which can achieve high-fidelity audio reconstruction at extremely low bit rates. The existing Codec system is mainly used for audio compression and reconstruction, and its input and output are the same audio. The system is usually composed of four modules, such as Figure 1 As shown in Figure 1, the codec system comprises: 1) an audio encoder, which extracts audio features and encodes them into a high-dimensional representation space while simultaneously downsampling the audio; 2) a quantizer, which discretizes the high-dimensional representation to generate a discrete representation consisting of a finite number of vectors; 3) an audio decoder, which restores the discrete representation to the original audio, achieving compression and reconstruction; and 4) a discriminator, which typically consists of multiple sub-discriminators and assesses the quality of audio reconstruction based on multiple dimensions, including temporal sampling points, spectrograms, and perceptual evaluation of speech quality (PESQ). During model training, the design and performance of any module can impact the overall system performance, and the selection of each module is closely related to the final performance. Current codec systems are limited to audio compression and reconstruction. While they can achieve efficient compression and restoration, no research has systematically explored the impact of these modules on audio reconstruction performance at low bitrates, nor the specific impact of internal modules (especially the quantizer of the audio representation) on model performance. Furthermore, their potential for speech and background sound separation has not been explored.

[0039] Against this backdrop, this paper proposes a method for speech and background sound separation in low-bitrate environments, introducing the end-to-end neural audio codec framework, Codec, for the first time. By constructing a unified evaluation framework and combining it with a series of modular ablation experiments, this paper systematically explores the impact of the combination of key modules—the encoder, quantizer, decoder, and discriminator—on system performance, identifying the optimal architecture for audio compression and separation. This approach not only retains the modeling advantages of neural codec models at high compression ratios, but also achieves collaborative modeling of compression and separation tasks through systematic optimization of internal modules (such as the encoder architecture, discriminator design, and audio quantizer), significantly improving separation quality and robustness at low bitrates. The speech and background sound separation method based on Codec technology, proposed in this embodiment of the present invention, aims to achieve low-bitrate compression while simultaneously enhancing efficient speech and background sound separation capabilities. Addressing the limitations of existing audio codec technologies in speech separation, this paper systematically explores the applicability of various Codec frameworks for speech and background sound separation and thoroughly examines the specific impact of different modules, particularly the audio quantization module, on separation performance. On this basis, the present invention proposes an optimized design of a Codec-based codec separation model. Through the coordinated improvement of the encoder structure, discriminator design, and quantizer model, it can still achieve good speech and background sound separation performance under low bit rate compression conditions. This invention is the first work to systematically study the impact of various Codec modules (including quantizer, discriminator, and encoder architecture) on the performance of speech and background sound separation. It verifies its scalability in multi-task audio processing and makes targeted structural improvements for this new task scenario, expanding the application boundaries of the Codec model in the field of audio processing. In addition to choosing to implement it based on the Codec model architecture, it can also be implemented based on other speech codecs (such as Opus, AMR, etc.).

[0040] Compared to existing methods, this invention innovatively addresses the issues of speech and background noise confusion and limited feature expression at low bit rates through model architecture design. This provides a more practical solution for applications such as telecommunications, speech recognition preprocessing, and hearing aid design. Therefore, this solution has significant theoretical significance and broad application prospects in the integration of audio compression and multi-task speech processing, demonstrating significant innovation and potential for widespread adoption.

[0041] Example 1

[0042] This embodiment provides a method for separating speech and background sound based on Codec. Figure 2 Shown, including:

[0043] S1, Separation model construction process: Construct a codec separation model based on Codec, such as Figure 3 As shown, it includes an encoder, a separation module, two audio representation quantizers and a decoder; the encoder is used to encode audio into a continuous audio representation; the separation module is used to separate the continuous audio representation into a hidden representation containing speech and a hidden representation containing background sound, and the two audio representation quantizers are used to discretize the hidden representation containing speech and the hidden representation containing background sound, and compress them to a low bit rate level to obtain a compressed discrete speech representation and a compressed discrete background sound representation; the decoder is used to restore the compressed discrete speech representation and the compressed discrete background sound representation to the separated speech signal and background sound signal, thereby realizing audio separation in the feature representation space.

[0044] S2, model training process: Figure 4 As shown, a pure speech sample is randomly selected and superimposed with background sound to generate a mixed input audio; after audio separation is performed through the Codec-based codec separation model, a speech signal and a background sound signal are obtained, and the speech signal and the background sound signal are mixed to obtain a predicted mixed audio; the first discriminant module and the second discriminant module are used to respectively evaluate the reconstruction quality of the model for the speech and background sound after separation, and the third discriminant module is used to evaluate the reconstruction quality of the predicted mixed audio, thereby training the Codec-based codec separation model until completion.

[0045] S3. Audio separation process: Use the trained Codec-based codec separation model to separate the target audio, and use the perceptual speech quality evaluation index (PESQ) to quantitatively evaluate the speech quality of the model separation output.

[0046] During model training, the design and performance of any module can impact the overall system performance, and the selection of each module is closely related to the final performance. Currently, no research has systematically explored the impact of each of these modules on the performance of audio reconstruction tasks under low bitrate conditions. In one specific embodiment, we systematically explored the impact of different module designs on audio compression and reconstruction performance under different neural encoder architectures (such as Encodec and HILCodec). Specifically, the research content includes: comparative experiments on multiple discriminator combinations, such as the combination of Multi-Scale Feature-Based Discriminator (MFBD) and Multi-Scale Short-Time Fourier Transform Discriminator (MSTFTD), or the combination of Multi-Period Discriminator (MFD) and Multi-Scale Period Discriminator (MPD), as well as the combined use of the four discriminators; different types of quantizer designs, including vector residual quantizer (RVQ), residual RVQ, fully connected structure quantizer (FSQ), binary quantizer (BSQ), low-frequency quantizer (LFQ) and residual low-frequency quantizer (Residual LFQ); and configuration combinations with different codebook sizes and number of codebooks.

[0047] The above research was conducted in an ultra-low bitrate compression scenario of 3kbps. The trade-off between audio compression rate and reconstruction fidelity of various designs was systematically evaluated, further providing optimization solutions for low bitrate speech processing applications. Figure 5 .

[0048] The final selected Codec-based codec separation model is as follows Figure 3 As shown, based on the HILCodec model architecture, it includes four main parts: an encoder, a separation module, two audio representation quantizers, and a decoder. The separation module includes two sets of nonlinear layers, each of which includes multiple nonlinear layers of different dimensions, specifically a 64-dimensional nonlinear layer, a 32-dimensional nonlinear layer, and a 16-dimensional nonlinear layer. The first, second, and third discriminant modules include a multi-scale Fourier discriminator (MFBD) and a multi-resolution short-time Fourier transform discriminator (MSTFTD), respectively. The audio representation quantizer uses a fully connected structured quantizer (FSQ).

[0049] The specific steps of the model training process are as follows:

[0050] 1. Use multiple open-source datasets to train the model, including: 1) speech datasets: DNS-Challenge, LibriTTS, and VCTK; 2) noise datasets: DNS-Challenge noise dataset; 3) music and other sound datasets: MTG-Jamendo and AudioSet (audio_event). The process of generating mixed input audio for training is as follows: randomly obtain an audio signal containing a single or multiple person's voices. and an audio signal containing only background sound , through volume normalization and signal superposition, it is mixed into an audio signal of voice and background sound, that is, mixed input audio. The formula is as follows: ; ;

[0051] Where T is the number of audio sampling points.

[0052] 2. Encode the audio into a continuous audio representation through the encoder , where N is the characterization length. The formula is as follows: ;

[0053] in, For the audio encoder function.

[0054] 3. The output of the audio encoder passes through a separation module Sep(x) consisting of multiple nonlinear layers, and the output contains the hidden representation of the speech and a hidden representation containing background sound , the formula is as follows: ;

[0055] 4. The hidden representations containing speech and background sound are discretized by FSQ quantizer and compressed to a bit rate of 3 kbps. FSQ calculates each scalar so that the total number of vectors composed of scalars is finite. The principle of FSQ quantizer is as follows: Figure 6 The specific formula is as follows: ;

[0056] in, is the rounding function, , L is the discretization level, d is the dimension of each vector in X, then is the vocabulary size of the discretized vector.

[0057] FSQ converts each scalar contained in the X tensor to a range from 0 to L-1. This embodiment is performed under the condition that the input audio sampling rate is 24k, and the quantization vocabulary of the FSQ quantizer is set to [8,8,8,8,5,5,5,5,5,5,5,5,5,5,5,5].

[0058] 5. The compressed discrete audio representation is fed into the decoder Decoder(x), which further restores it to the separated multi-speaker speech signal and background sound signal, thereby achieving audio separation in the feature representation space. The formula is as follows: .

[0059] The MFBD (Multi-Scale Fourier Discriminator) and MSTFTD (Multi-Resolution Short-Time Fourier Transform Discriminator) discriminant modules are used to evaluate the quality of the model after separation and reconstruction of audio in the representation space. This discriminator combination is highly effective in evaluating the performance of audio reconstruction tasks under low bitrate conditions. Specifically, the first and second discriminant modules evaluate the model's reconstruction quality of the speech and background sound after separation, respectively. The third discriminant module also determines the reconstruction quality of the entire mixed audio (comparing the predicted mixed audio with the mixed input audio). This improves the separation effect while minimizing information loss.

[0060] Finally, in the audio separation process, there is no need to use the discriminant module, and the PESQ (perceptual speech quality) indicator can be used to quantitatively evaluate the speech quality of the model separation output.

[0061] The method of this embodiment is centered on Codec technology. By separating speech and background sound in the representation space, it successfully solves the problem that traditional separation methods have difficulty balancing separation performance and audio quality under low-bitrate compression conditions. Specifically, the Codec model can effectively extract a compact, high-information-density intermediate representation of speech and background sound, enabling the separation module to accurately distinguish different sound sources in the compressed representation, significantly reducing the risk of information loss. At the same time, this solution avoids the computational burden and separation errors brought about by direct processing at the waveform or spectral level. At a low bitrate of 3kbps, although the PESQ of the separated speech is 1.853, slightly lower than the single-audio reconstruction result (1.901) at the same bitrate, the speech quality remains good compared to the complexity of the task, indicating that this method is still practical and robust in resource-constrained scenarios.

[0062] Based on the same inventive concept, this application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0063] Example 2

[0064] In this embodiment, a device for separating speech and background sound based on Codec is provided, including a codec separation model based on Codec (which can be Figure 3 The model architecture shown in the figure) is shown in the figure), and the Codec-based codec separation model includes an encoder, a separation module, two audio representation quantizers and a decoder; the encoder is used to encode the audio into a continuous audio representation; the separation module is used to separate the continuous audio representation into a hidden representation containing speech and a hidden representation containing background sound, and the two audio representation quantizers are used to discretize the hidden representation containing speech and the hidden representation containing background sound, respectively, and compress them to a low bit rate level to obtain a compressed discrete speech representation and a compressed discrete background sound representation; the decoder is used to restore the compressed discrete speech representation and the compressed discrete background sound representation to the separated speech signal and background sound signal, thereby realizing audio separation in the feature representation space.

[0065] Since the device described in the second embodiment of the present invention is used to implement the method of the first embodiment of the present invention, those skilled in the art will be able to understand the specific structure and variations of the device based on the method described in the first embodiment of the present invention, and therefore will not be described in detail here. All devices used in the method of the first embodiment of the present invention fall within the scope of protection of the present invention.

[0066] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0067] Example 3

[0068] This embodiment provides an electronic device, such as Figure 7 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0069] Since the electronic device described in this embodiment is the device used to implement the method in Example 1 of this application, based on the method described in Example 1 of this application, those skilled in the art will be able to understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. As long as the device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection to be provided by this application.

[0070] Based on the same inventive concept, this application provides a storage medium corresponding to Example 1, see Example 4 for details.

[0071] Example 4

[0072] This embodiment provides a computer-readable storage medium, such as Figure 8 As shown, a computer program is stored thereon, and when the computer program is executed by a processor, any implementation method in Example 1 can be implemented.

[0073] Since the computer-readable storage medium described in this embodiment is the computer-readable storage medium used to implement the method in Example 1 of this application, based on the method described in Example 1 of this application, those skilled in the art will be able to understand the specific implementation and various variations of the computer-readable storage medium of this embodiment, so how the computer-readable storage medium implements the method in the embodiment of this application will not be described in detail here. As long as those skilled in the art implement the computer-readable storage medium used in the method in the embodiment of this application, it falls within the scope of protection to be provided by this application.

[0074] The present invention proposes a method for separating speech and background sound in low-bitrate environments. For the first time, it extends the end-to-end neural audio codec framework Codec to the speech separation task. It not only retains the modeling advantages of the neural codec model under high compression ratios, but also realizes the collaborative modeling of compression and separation tasks through systematic optimization of internal modules, significantly improving the separation quality and robustness under low bit rates. Compared with existing methods, the present invention innovatively solves the problems of easy confusion between speech and background sound and limited feature expression under low bitrate conditions based on model architecture design, providing a more practical solution for application scenarios such as remote communication, speech recognition preprocessing, and hearing aid design. This solution has important theoretical significance and broad application prospects in the integration of audio compression and multi-task speech processing, and has significant innovation and promotion value. The technical solution of the present invention mainly relates to a speech and background sound separation system under low bitrate conditions. After the technology of this solution matures, the voice component service of this solution can be called on power-related systems.

[0075] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0077] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0079] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for separating speech and background sound based on Codec, characterized in that: include: Separation model construction process: Build a codec-based codec separation model, including an encoder, a separation module, two audio representation quantizers, and a decoder; The encoder is used to encode audio into a continuous audio representation; the separation module is used to separate the continuous audio representation into a hidden representation containing speech and a hidden representation containing background sound. The two audio representation quantizers are used to discretize the hidden representation containing speech and the hidden representation containing background sound, respectively, and compress them to a low bit rate level to obtain a compressed discrete speech representation and a compressed discrete background sound representation; the decoder is used to restore the compressed discrete speech representation and the compressed discrete background sound representation to the separated speech signal and background sound signal, thereby achieving audio separation in the feature representation space; Model training process: Randomly select pure speech samples and superimpose background sound to generate mixed input audio. After audio separation using the codec-based codec separation model, the speech signal and background sound signal are obtained, and the speech signal and background sound signal are mixed to obtain the predicted mixed audio. The first and second discriminant modules are used to evaluate the model's reconstruction quality of the speech and background sound after separation, respectively. The third discriminant module is also used to evaluate the reconstruction quality of the predicted mixed audio. The codec-based codec separation model is trained until completion. Audio separation process: Use the trained Codec-based codec separation model to separate the target audio and quantitatively evaluate the voice quality of the model separation output.

2. The method according to claim 1, wherein: The separation module includes two groups of nonlinear layers, each group of nonlinear layers includes multiple nonlinear layers of different dimensions, and the two groups of nonlinear layers are used to extract speech signals and background sound signals respectively.

3. The method according to claim 1, wherein: The process of generating mixed input audio is as follows: Get an audio signal containing a single or multiple voices and an audio signal containing only background sound , through volume normalization and signal superposition, it is mixed into an audio signal of voice and background sound, that is, mixed input audio. The formula is as follows: ; ； Where T is the number of audio sampling points.

4. The method according to claim 1, wherein Both audio representation quantizers use FSQ quantizers. The quantization process is as follows: discretize the hidden representation containing speech and the hidden representation containing background sound, compress them to a low bit rate level, and convert each scalar contained in the X tensor to a range of 0 to L-1, so that the total number of vectors composed of scalars is finite. The formula is as follows: ; in, is the rounding function, , L is the discretization level, d is the dimension of each vector in X, then is the vocabulary size of the discretized vector.

5. The method according to claim 4, characterized in that: When the input audio sampling rate is 24k, the quantization vocabulary of the FSQ quantizer is set to [8,8,8,8,5,5,5,5,5,5,5,5,5,5,5], and the latent representation is compressed to a bit rate of 3 kbps.

6. The method according to claim 1, wherein: The first discrimination module, the second discrimination module and the third discrimination module respectively include a multi-scale Fourier discriminator and a multi-resolution short-time Fourier transform discriminator.

7. The method according to claim 1, wherein: The perceptual speech quality evaluation index is used to quantitatively evaluate the speech quality of the model separation output.

8. A device for separating speech and background sound of a codec, characterized by: The invention comprises a codec-based codec separation model, which comprises an encoder, a separation module, two audio representation quantizers and a decoder; the encoder is used to encode the audio into a continuous audio representation; the separation module is used to separate the continuous audio representation into a hidden representation containing speech and a hidden representation containing background sound, and the two audio representation quantizers are used to discretize the hidden representation containing speech and the hidden representation containing background sound, and compress them to a low bit rate level to obtain a compressed discrete speech representation and a compressed discrete background sound representation; the decoder is used to restore the compressed discrete speech representation and the compressed discrete background sound representation to the separated speech signal and background sound signal, thereby realizing audio separation in the feature representation space.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images