Method and apparatus for speech enhancement

Abstract

The invention relates to a method for speech enhancement, a method for training a neural network model, a device for speech enhancement, electronic equipment and a computer readable storage medium. According to the invention, the coding vector is decoded based on the coding vector and the decoding auxiliary vector, and the decoding auxiliary vector fuses the task prompt information indicating the voice enhancement task type and the semantics of the enhancement prompt information capable of realizing voice enhancement at the signal level. Due to the flexibility of decoding the auxiliary vector, the embodiments of the present disclosure can be adapted to more types of speech enhancement tasks. In some embodiments of the disclosure, the decoding auxiliary vector can be flexibly applied to connection of an encoder-decoder network, so that the capability suitable for processing various speech enhancement tasks is obtained through the decoding auxiliary vector in encoding-decoding, and a speech enhancement system is more efficient and flexible when coping with complex speech data.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence services, and more specifically to a method for speech enhancement, an apparatus for speech enhancement, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In real-time communication and complex acoustic environments, acoustic echo and ambient noise significantly impact the intelligibility and overall sound quality of voice data. Acoustic echo primarily arises from sound propagating from the speaker to the microphone and being recorded again. Acoustic echo and ambient noise, such as traffic noise and air conditioning noise, interfere with the intelligibility of voice data, reducing its readability and negatively impacting the performance of communication systems and user experience.

[0003] Traditionally, digital signal processing methods are used to enhance speech data. While effective in some cases, these methods are performance-limited when acquiring speech data in complex acoustic scenarios. With the rapid development of deep learning technology, combining deep neural networks has been considered to achieve higher-performance speech enhancement algorithms. However, such speech enhancement algorithms still lack flexibility and cannot adapt to various speech enhancement task requirements. Furthermore, the aforementioned methods may consume excessive computational resources, making them difficult to implement directly in industry.

[0004] Therefore, speech enhancement technology still needs further optimization and improvement. Summary of the Invention

[0005] This disclosure provides a method for speech enhancement, a method for training a neural network, an apparatus for speech enhancement, an electronic device, and a computer-readable storage medium.

[0006] This disclosure provides a method for speech enhancement, the method comprising: determining multi-frame feature data based on time-frequency data corresponding to speech data; generating multiple encoding vectors based on feature data of at least one frame of the multi-frame feature data; generating nested prompt information based on task prompt information and enhancement prompt information; adjusting at least one encoding vector among the multiple encoding vectors using the nested prompt information to generate at least one decoding auxiliary vector; generating a decoding vector based on at least one encoding vector and the at least one decoding auxiliary vector; and generating speech enhancement data based on the decoding vector.

[0007] This disclosure provides a method for training a neural network model, the neural network model including: a frequency band segmentation module, an encoder-decoder module, a nested information enhancement module, and a frequency band merging module. The method includes: determining multi-frame feature data based on time-frequency data corresponding to sample speech data using the frequency band segmentation module; generating multiple encoding vectors using an encoder in the encoder-decoder module based on feature data from at least one frame of the multi-frame feature data; generating nested prompt information using a nested information enhancement module based on task prompt information and enhanced prompt information; adjusting at least one encoding vector among the multiple encoding vectors using the nested prompt information to generate at least one decoding auxiliary vector; generating a decoding vector using a decoder in the encoder-decoder module based on at least one encoding vector and the at least one decoding auxiliary vector; generating speech enhancement data using the frequency band merging module based on the decoding vector; and training one or more modules in the neural network model using the speech enhancement data and the enhancement data corresponding to the sample speech data.

[0008] This disclosure provides an apparatus for speech enhancement, comprising: a frequency band segmentation module configured to: determine the representation of text data in a latent space based on text data; an encoder configured to: generate multiple encoding vectors based on feature data from at least one frame of multi-frame feature data; a nested information enhancement module configured to: generate nested prompt information based on task prompt information and enhancement prompt information, and adjust at least one encoding vector among the multiple encoding vectors using the nested prompt information to generate at least one decoding auxiliary vector; a decoder configured to: generate a decoding vector based on at least one encoding vector among the multiple encoding vectors and the at least one decoding auxiliary vector; and a frequency band merging module configured to: generate speech enhancement data based on the decoding vector.

[0009] This disclosure provides an electronic device, including: one or more processors; and one or more memories, wherein the memories store a computer-executable program, and when the processor executes the computer-executable program, the above-described method is performed.

[0010] This disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the above-described method.

[0011] According to another aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the methods provided in the foregoing aspects or various alternative implementations of the foregoing aspects.

[0012] This disclosure uses encoding vectors and decoding auxiliary vectors to decode encoding vectors. The decoding auxiliary vectors integrate task prompts indicating the type of speech enhancement task and enhancement prompts that enable speech enhancement at the signal level. Due to the flexibility of the decoding auxiliary vectors, this disclosure can adapt to more types of speech enhancement tasks, significantly improving the flexibility of speech enhancement technology. In some embodiments of this disclosure, the decoding auxiliary vectors can be flexibly applied to the connection of encoder-decoder networks to obtain the ability to handle various speech enhancement tasks during encoding-decoding, making the speech enhancement system more efficient and flexible when dealing with complex speech data. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the following description are merely exemplary embodiments of this disclosure.

[0014] Figure 1 This is an example schematic diagram illustrating a scenario according to an embodiment of the present disclosure.

[0015] Figure 2 This is yet another schematic diagram illustrating a scenario according to an embodiment of the present disclosure.

[0016] Figure 3 This is a schematic diagram illustrating the architecture of a neural network according to an embodiment of the present disclosure.

[0017] Figure 4 This is a schematic diagram illustrating a frequency band segmentation module according to an embodiment of the present disclosure.

[0018] Figure 5 This is a schematic diagram illustrating a variable frame rate layer in the encoder of an encoder-decoder module according to an embodiment of the present disclosure.

[0019] Figure 6 This is a schematic diagram illustrating a nested information enhancement module according to an embodiment of the present disclosure.

[0020] Figure 7 This is a schematic diagram illustrating a band combining module according to an embodiment of the present disclosure.

[0021] Figure 8 A flowchart of a method for speech enhancement according to an embodiment of the present disclosure is shown.

[0022] Figure 9 A flowchart illustrating a method for training a neural network model according to an embodiment of the present disclosure is shown.

[0023] Figure 10 This is a comparison chart showing test results according to embodiments of the present disclosure.

[0024] Figure 11 This is a schematic diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0025] Figure 12 This is a schematic diagram illustrating the architecture of an exemplary computing device according to embodiments of the present disclosure.

[0026] Figure 13 This is a schematic diagram illustrating a storage medium according to an embodiment of the present disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0028] In this specification and accompanying drawings, operations and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these operations and elements are omitted. Furthermore, in the description of this disclosure, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0029] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0030] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0031] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interaction systems, and mechatronics. Pre-trained models, also known as large-scale models or foundational models, can be fine-tuned and widely applied to downstream tasks across various AI fields. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning. With the advancement of AI research, its applications are expanding into numerous fields, such as smart homes, wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role.

[0032] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models are the latest development in deep learning, integrating all of these techniques.

[0033] Optionally, the models used in embodiments of this disclosure as described below can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs, where neurons are arranged in different layers. Generally, a neural network model includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Nodes are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at nodes in the input layer of the neural network is propagated to nodes in the output layer via any of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and this disclosure does not limit this.

[0034] The solutions provided in this disclosure involve technologies such as artificial intelligence and / or machine learning, which are specifically illustrated through the following embodiments.

[0035] First, refer to Figure 1 The present disclosure describes application scenarios of methods and apparatuses for speech enhancement according to embodiments of the present disclosure. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and a plurality of terminals 120 are schematically illustrated.

[0036] The neural network model of this disclosure can be integrated into various electronic devices, for example. Figure 1 The neural network model can be integrated into any electronic device in server 110 and multiple terminals 120. For example, the neural network model can be integrated into terminal 120. Terminal 120 can be a mobile phone, tablet, laptop, desktop computer, personal computer (PC), smart speaker, or smartwatch, but is not limited to these. Alternatively, the neural network model can also be integrated into server 110. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations.

[0037] It is understood that the apparatus for inference using the neural network model of the embodiments of this disclosure can be a terminal, a server, or a system composed of a terminal and a server. The speech enhancement method of the embodiments of this disclosure can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0038] The artificial intelligence model provided in this disclosure can also relate to artificial intelligence cloud services in the field of cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, network, and other resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites, drug research websites, and many more portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require strong system support, which needs to be realized through cloud computing.

[0039] It is worth noting that both the terminal 120 and server 110 according to the embodiments of this disclosure adhere to data protection principles, respect users' data rights, and safeguard users' data security and privacy. The terminal 120 and server 110 according to the embodiments of this disclosure will clearly inform users of the purpose, method, and scope of collecting, using, storing, transmitting, and deleting user data, and obtain users' consent. The terminal 120 and server 110 according to the embodiments of this disclosure will take reasonable technical and management measures to prevent user data from being leaked, tampered with, damaged, or lost. The providers of the terminal 120 and server 110 according to the embodiments of this disclosure will regularly review and update user data, and promptly delete expired or useless data. Furthermore, cloud service providers using the embodiments of this disclosure respect users' rights to data access, correction, deletion, withdrawal of consent, complaints, and claims, and provide convenient channels and procedures to enable users to effectively exercise these rights.

[0040] Furthermore, the process of data analysis using artificial intelligence technology in terminal 120 or server 110 is conducted based on the principles of legality, rationality, and transparency. The data collected and processed by the artificial intelligence model according to embodiments of this disclosure is relevant, necessary, and appropriate for the predictive purpose, and does not contain any personally identifiable or sensitive information. The neural network model according to embodiments of this disclosure employs appropriate techniques and organizational measures to protect the security and integrity of data, preventing unauthorized access, use, or disclosure.

[0041] The artificial intelligence-based neural network model according to embodiments of this disclosure will comply with relevant data protection regulations and ethical principles. This neural network model is trained on a large amount of anonymized and de-identified data, and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable, and will not cause any misleading or discriminatory results. The artificial intelligence model is designed solely to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. Furthermore, the neural network model will be regularly reviewed and updated to adapt to changes in the data environment and legal regulations.

[0042] Speech enhancement (SE) is a key research area in communication and audio processing. In complex acoustic environments, speech enhancement is crucial for improving the quality and clarity of speech signals. In real-world environments, speech signals are frequently affected by environmental noise, echoes, reverberation, far-field effects, and distortion caused by communication system transmission. To recover the target speech from this mixture of noise, speech enhancement techniques attempt to remove interference and enhance degraded speech.

[0043] Systems employing speech enhancement technology can be broadly categorized into two types: the first type is time-domain based methods, which directly process waveforms in the time domain; the second type is frequency-domain based methods, which focus more on spectrum estimation and enhancement.

[0044] In recent years, speech enhancement techniques combined with deep learning have been proposed. Both of these methods can be integrated with deep learning models. Specifically, deep learning models can be used to predict masking vectors and use them as denoising filters, or to estimate the spectrum distribution through regression methods. Schemes have been proposed to optimize the real and imaginary parts of the spectrogram through complex-valued operations to implicitly optimize phase information, thereby improving the quality of speech signal reconstruction.

[0045] To address more challenging acoustic scenarios such as low signal-to-noise ratio and high reverberation, industry and academia have proposed multi-stage fitting methods that divide the speech enhancement process into several stages, from coarse to fine. For example, in these methods, Taylor approximation theory can be used to stack networks to achieve amplitude preprocessing and fine recovery of amplitude and phase.

[0046] Beyond basic denoising and dereverberation tasks, bandwidth extension (BWE) has become an important research direction. This area focuses on predicting high-frequency components of low-resolution audio to meet the bandwidth requirements of different application scenarios. Furthermore, generative techniques such as Generative Adversarial Networks (GANs) and diffusion models are also being applied to reconstruct clean speech signals.

[0047] While the above-mentioned solutions may achieve good speech enhancement results on different tasks, they may struggle to meet the needs of practical applications. Specifically, existing solutions are based on fixed neural network architectures and are typically only adaptable to a single task. For example, solutions suitable for denoising tasks often perform poorly for echo removal tasks, and echo removal tasks also struggle to achieve good performance in denoising tasks. It is evident that traditional speech enhancement solutions lack flexibility and cannot simultaneously adapt to the diverse needs of various speech enhancement tasks. Furthermore, the aforementioned solutions may consume excessive computational resources, making them difficult to implement directly in industry.

[0048] To address this, this disclosure proposes a robust and flexible speech enhancement solution. Optionally, this disclosure improves the architecture of traditional neural network models used for speech enhancement. Compared to traditional neural network-based speech enhancement schemes, the improved neural network model allows for coupling of the encoder-decoder module and the nested information enhancement module during training and inference to enhance speech data.

[0049] Therefore, the improved neural network model can process the time-frequency data corresponding to the input speech data in three separate stages to achieve higher performance.

[0050] Optionally, in the first stage, the time-frequency data corresponding to the speech data can be segmented in both the time and frequency domains using a frequency band segmentation module to obtain multi-frame feature data. Then, in the second stage, a combination of an encoder-decoder module and a nested information enhancement module can be used to generate a decoding vector based on the multi-frame feature data. In the third stage, a frequency band merging module can be used to generate speech enhancement data based on the decoding vector.

[0051] Compared to traditional speech enhancement techniques, the improved neural network model according to embodiments of this disclosure combines an encoder-decoder module with a nested information enhancement module, utilizing decoding auxiliary vectors to decode the encoded vectors. These decoding auxiliary vectors integrate task prompts indicating the type of speech enhancement task with semantic enhancement prompts that enable speech enhancement at the signal level. Due to the flexibility of the decoding auxiliary vectors, embodiments of this disclosure can adapt to more types of speech enhancement tasks, significantly improving the flexibility of speech enhancement technology. For example, in some embodiments of this disclosure, decoding auxiliary vectors can be flexibly applied to the connections of the encoder-decoder network to obtain the ability to handle various speech enhancement tasks during the encoding-decoding process, making the speech enhancement system more efficient and flexible when dealing with complex speech data.

[0052] This disclosure provides a method for speech enhancement to utilize an improved neural network model for inference. The method includes: determining multiple frames of feature data based on time-frequency data corresponding to the speech data; generating multiple encoding vectors based on the feature data of at least one frame of the multiple frames of feature data; generating nested prompt information based on task prompt information and enhancement prompt information; adjusting at least one encoding vector among the multiple encoding vectors using the nested prompt information to generate at least one decoding auxiliary vector; generating a decoding vector based on the at least one encoding vector and the at least one decoding auxiliary vector; and generating speech enhancement data based on the decoding vector.

[0053] Optionally, during the inference process using the improved neural network, embodiments of this disclosure generate nested prompt information based on task prompt information and enhanced prompt information. Thus, the nested information enhancement module according to embodiments of this disclosure can specifically generate nested prompt information based on the speech enhancement task indicated by the task prompt information, and generate specific decoding auxiliary vectors based on the nested prompt information. Therefore, embodiments of this disclosure are no longer limited by traditional speech enhancement techniques, but can flexibly adjust the speech data to be generated directly through prompt information.

[0054] For example, if the task prompt indicates that the current speech enhancement task is both noise reduction and acoustic echo removal, then nested prompts based on this task prompt can generate more targeted decoding auxiliary vectors for the encoded vector to assist the decoder. In this way, the improved neural network can perform both noise reduction and acoustic echo removal tasks on the same speech data at once, and obtain speech enhancement data with noise and acoustic echo removed in one go.

[0055] Compared to traditional neural networks used for speech enhancement, the improved neural network has fewer parameters and can perform multiple tasks at once, giving it a significant advantage in terms of computational cost and making it easier to deploy in industry.

[0056] Furthermore, this disclosure also provides a method for training the improved neural network model. The neural network includes a frequency band segmentation module, an encoder-decoder module, a nested information enhancement module, and a frequency band merging module. The method includes: determining multiple frames of feature data using the frequency band segmentation module based on time-frequency data corresponding to sample speech data; generating multiple encoding vectors using the encoder in the encoder-decoder module based on feature data from at least one frame of the multiple frames of feature data; generating nested prompt information using the nested information enhancement module based on task prompt information and enhanced prompt information; adjusting at least one encoding vector among the multiple encoding vectors using the nested prompt information to generate at least one decoding auxiliary vector; generating a decoding vector using the decoder in the encoder-decoder module based on at least one encoding vector and the at least one decoding auxiliary vector; generating speech enhancement data using the frequency band merging module based on the decoding vector; and training one or more modules in the neural network model using the speech enhancement data and the enhancement data corresponding to the sample speech data.

[0057] The following reference Figures 2 to 8 Methods for speech enhancement, methods for training neural networks, and apparatus according to embodiments of the present disclosure are described.

[0058] Figure 2 This is a schematic diagram illustrating an application scenario of a neural network according to an embodiment of the present disclosure.

[0059] The neural network according to embodiments of this disclosure can be applied to speech enhancement of sound captured in complex acoustic scenes. For example, suppose a user is using a smartphone to make a voice call or record voice notes on a busy street. The actual voice signal emitted by the user can be recorded as... L represents the total number of sample points obtained by sampling the speech signal s in the time domain. The user is surrounded by noisy traffic, a cacophony of voices, and various background noises. The user's smartphone is equipped with a high-quality microphone capable of capturing all these sounds, including the user's voice, possible acoustic echoes, and background noise. These sounds mix together to form a speech signal x. x can be considered as the actual speech signal s produced by the user after various degradations and interferences. Mathematically, the relationship between the speech signal x and the speech signal s can be described by formula (1).

[0060]

[0061] in, The function represents the degradation function. Optionally, the task of the neural network according to embodiments of this disclosure is to reconstruct the speech signal s from the speech signal x. Mathematically, this process can be described by formula (2).

[0062]

[0063] in, This represents the recovery function. This represents the recovered signal. One of the tasks of speech enhancement is to find the optimal recovery function. In traditional speech enhancement techniques, the powerful nonlinear mapping capabilities of deep neural networks (DNNs) are considered to attempt to find the optimal recovery function. While DNN-based algorithms perform well in handling single noises or a few specific degradations, their generalization ability is poor. To enable neural network models to generalize to a wider range of real-world applications, embodiments of this disclosure attempt to understand and simulate various types of degradation at the operational level. This includes the combined effects of different factors such as noise, reverberation, and bandwidth limitations, to achieve a more robust and flexible speech enhancement system capable of adapting to varying real-world conditions.

[0064] For example, the task of this disclosure embodiment is to attempt to find such a recovery function. It may be a linear or non-linear superposition of the following four speech enhancement tasks. It is worth noting that the following four speech enhancement tasks are merely examples. In real-world acoustic scenarios, there may be more or fewer speech enhancement tasks, which will not be listed here.

[0065] The first type of speech enhancement task is the traditional denoising task. In general, denoising tasks... The goal is to remove additive interference. (For example, background noise). For this task, mathematically, the relationship between speech signal x and speech signal s can be derived from formula (3).

[0066] x=s+n (3)

[0067] The second type of speech enhancement task is the dereverberation task. Due to spatial influences, the original speech signal may undergo multiple reflections in various spaces, resulting in a time difference between the speech signal reaching the microphone and the subsequent reverberation, which can severely affect the quality and intelligibility of the speech. The interference in the dereverberation task is usually in the form of convolution and can be modeled as a convolution between the speech signal s and the spatial impulse response (RIR) h. For this task, mathematically, the relationship between the speech signal x and the speech signal s can be expressed by formula (4).

[0068] x=s*n (4)

[0069] The third type of speech enhancement task is bandwidth expansion. In speech transmission and communication, limitations of acquisition equipment and channel conditions often result in low sampling rates and limited bandwidth. For example, if the bandwidth used by the communication equipment is limited, high-frequency components may often be lost in the speech data acquired by the microphone. The bandwidth expansion task can be modeled as a process of resampling and anti-aliasing low-pass filtering. Mathematically, this task can be addressed using Equation (5), which shows the relationship between the speech signal x and the speech signal s.

[0070] x = Resample(s*h, o, u) (5)

[0071] Where o and u represent the original sampling rate and the sampling rate under bandwidth-limited conditions, respectively.

[0072] The fourth speech enhancement task is the near effect removal task. The near effect refers to a change in audio characteristics that occurs when a sound source is very close to a microphone (such as a microphone). When the near effect occurs, because the sound source is very close to the microphone, the wavelength of low-frequency sound waves is longer, generating more pressure on the microphone diaphragm, resulting in an enhancement of low-frequency components. The near effect is often difficult to model, therefore it is necessary to try to find the optimal recovery function with the help of neural networks.

[0073] The neural network of this disclosure can directly extract a clear speech signal s from a speech signal x. Based on task prompts, the neural network of this disclosure can simultaneously suppress various interferences such as background noise, echo, and reverberation, significantly improving the intelligibility and naturalness of speech. Whether in telephone calls, meeting minutes, or voice recognition for intelligent assistants, it ensures high-quality transmission and recording of speech signals. The application of this disclosure improves the user experience, enabling clear and accurate voice communication even in the most challenging and complex sound environments.

[0074] Figure 3 A schematic diagram of the architecture of a neural network according to an embodiment of the present disclosure is shown. Figure 4 A schematic diagram of a frequency band segmentation module according to an embodiment of the present disclosure is shown. Figure 5 A schematic diagram of a variable frame rate layer in an encoder in an encoder-decoder module according to an embodiment of the present disclosure is shown. Figure 6 A schematic diagram of a nested information enhancement module according to an embodiment of the present disclosure is shown. Figure 7 A schematic diagram of a band combining module according to an embodiment of the present disclosure is shown.

[0075] like Figure 3As shown, the neural network model according to the embodiments of this disclosure mainly consists of four or more decoupled modules: a frequency band segmentation module, an encoder-decoder module, a nested information enhancement module, and a frequency band merging module.

[0076] Before inputting the voice signal x into the frequency band segmentation module, the voice signal x needs to be converted into time-frequency data X = {X...} r X i This process can be implemented using a Short-Time Fourier Transform (STFT) operation. For example, the time-frequency data corresponding to the speech data is a complex representation obtained by transforming the speech signal x in the time domain. The frequency information in each time frame (or frame) contains the real part X. r And the imaginary part X i Time-frequency data uses this format to represent the frequency components at different time points, forming a matrix where rows represent time, columns represent frequency, and element values ​​are complex values ​​corresponding to the time and frequency points. Processing speech data in the time-frequency domain can improve the accuracy and richness of speech data processing.

[0077] In the first stage, the time-frequency data corresponding to the speech data can be segmented in both the time and frequency domains using a frequency band segmentation module to obtain multi-frame feature data. For example... Figure 4 As shown, the band splitting module performs a band splitting operation, which divides the time-frequency data X in the complex spectrum corresponding to a time frame into Q data X corresponding to frequency bands along the frequency axis. q , where the subscript q represents the segmented frequency band, q∈{1,...,Q}.

[0078] Each segment yields a feature X q The data is further fed into a normalization layer and a fully connected layer. The normalization layer helps stabilize the learning process, making the model insensitive to scale variations in the input data. The fully connected layer is used to extract and transform feature representations, converting the segmented features X... q Mapping to a higher-level feature data H q Next, the compressed feature data H q They are merged into a single feature data H. Mathematically, this process can be described by formulas (6) to (8), where Merge indicates that cascading is performed along the frequency axis.

[0079] {H1, ..., H Q} = BandSplit(X) (6)

[0080] H q =FC q (Norm(H q (7)

[0081] H = Merge(H1, ..., H) Q (8)

[0082] In the second stage, a combination of an encoder-decoder module and a nested information enhancement module can be used to generate decoding vectors based on multi-frame feature data. For example... Figure 3 As shown, the encoder-decoder module adopts an architecture similar to UNet (RecurrentUNet), but a different dimensionality reduction method is designed in each encoder layer compared to the traditional UNet. In this disclosure, this special encoder-decoder module is named Recurrent UNet.

[0083] For example, such as Figure 3 As shown, Recurrent UNet is a UNet structure that combines the characteristics of Recurrent Neural Networks (RNNs). It enables efficient processing of time-series data (i.e., multi-frame feature data) within an encoder-decoder framework.

[0084] The encoder in the recursive UNet aims to progressively extract features from multi-frame feature data. Specifically, the encoder can generate multiple encoded vectors based on feature data H from at least one frame of the multi-frame feature data. Each encoder layer outputs one encoded vector. Except for the first encoder layer, the input of each encoder layer is the encoded vector output by the previous encoder layer. The input of the first encoder layer is the feature data from the at least one frame. The encoded vector output by the last encoder layer is input to the first decoder layer in the decoder.

[0085] The encoder layers in the encoder follow a principle of fine-to-coarse, progressively reducing the down-sampling resolution in each encoder layer while maintaining the dimensionality of the output data. The decoder performs the opposite operation, moving from coarse to fine, gradually restoring the temporal resolution through multi-level causal time up-sampling while refining the feature representation. The decoder utilizes upsampling and transpose operations to recover the temporal dimension of the signal. In the recursive UNet, information is directly passed between the encoder and decoder via skip connections. This design allows the decoder to access high-resolution features from the encoder, thereby better reconstructing the details of the signal. Furthermore, each module of the recursive UNet employs a dual-path structure, where an RNN is used to mine temporal relationships, and a multilayer perceptron (MLP)-based band shuffler is used for global band modeling, further enhancing the model's ability to understand and process signals. Of course, this disclosure is not limited to this.

[0086] For example, each encoder layer can be Figure 5 The variable frame rate layer is shown. Variable frame rate layers can process features at different time scales, thus more effectively capturing dynamic changes in time-series data. Each variable frame rate layer may optionally include four modules: a downsampling module, a temporal mixing module, a frequency mixing module, and an upsampling module. However, this disclosure is not limited thereto.

[0087] The downsampling module downsamples the input vector to the encoder layer based on the downsampling rate corresponding to the encoder. The downsampling module is the starting point of the variable frame rate layer; it achieves temporal compression of the signal through causal convolution to compress the input encoded vector Z or a frame of feature data Z to Z0. d Specifically, mathematically, this process can be represented by formula (9).

[0088] Z d =Down-Sample(Z) (9)

[0089] The temporal mixing module generates a hidden vector that incorporates temporal information based on the downsampled input vector. The temporal mixing module can be a recurrent neural network (RNN) module, capable of capturing long-term dependencies in a time series and performing in-depth analysis of the signal's temporal dynamics. The RNN module may include: a normalization layer (Norm), a gated recurrent unit (GRU), and a fully connected layer (FC) arranged in sequence. With the assistance of the normalization layer and the fully connected layer, the temporal mixing module can effectively extract key information from the time series and pass it to the subsequent frequency-domain mixing module. This design not only enhances the network's sensitivity to time variations but also improves its understanding of the overall temporal structure of the signal. Specifically, mathematically, this process can be represented by formula (10).

[0090] Z t =Timemixing(Z d )+Z d (10)

[0091] The frequency domain mixing module upsamples the hidden vectors of both time-domain and frequency-domain information based on the upsampling rate corresponding to the encoder layer. Implemented using GateMLP, the frequency domain mixing module enables information exchange between different frequency sub-bands and global frequency band modeling. This module utilizes channel projection and gating mechanisms to rearrange and combine the frequency band characteristics of the signal, thereby achieving information fusion between frequency bands.

[0092] For example, such as Figure 5 As shown, the frequency domain mixing module includes a first channel projection module, a first normalization layer, a gating unit, and a second channel projection module arranged in sequence. The gating unit is the core of the frequency domain mixing module. It divides the feature map into multiple parts along the channel dimension through the splitheads mechanism, providing flexibility for the interaction between frequency bands. Each segment is processed independently, enhancing the model's ability to distinguish different frequency sub-bands. The gating unit also includes normalization and band projection, enabling the gating module to model the frequency band features along the time axis. The second channel projection module can then apply one-dimensional convolution again to further adjust and optimize the feature representation. Specifically, mathematically, this process can be represented by formula (11).

[0093] Z b =Bandmixing(Z t )+Z t (11)

[0094] In the frequency band mixing module, the hidden features are first expanded by doubling the channel dimension using one-dimensional convolution in the first channel projection module. The gating unit then divides the expanded feature representation into multiple parts along the channel dimension using a head segmentation mechanism. Each segmented feature is then subjected to frequency band projection along the time axis, enabling information exchange between frequency bands. The features after frequency band projection are then fused with another part using a multiplication gating mechanism. This mechanism allows the model to dynamically adjust the information flow between different frequency bands, enhancing the understanding of the signal's spectral characteristics. Finally, in the final channel projection stage, the feature map is reshaped and one-dimensional convolution is applied again to further refine the feature representation, providing richer and more structured feature information for the next layer of the network.

[0095] The main purpose of upsampling layers is to improve the spatial or temporal resolution of signals or feature maps to recover or add detail information. Specifically, mathematically, this process can be represented by formula (12).

[0096] Z u =Up-Sample(Z) t (12)

[0097] The upsampling and downsampling rates are different for different variable frame rate layers. Suppose the downsampling module of the first variable frame rate layer is set to a high downsampling rate, such as 1024. This means that 1024 data points need to be collected for one frame of feature data. The sampling rate of the second downsampling module will be lower than that of the first variable frame rate layer, for example, 512. This means that only 512 data points need to be collected for the encoded vector output by the first variable frame rate layer. And so on, the downsampling rate of the last variable frame rate layer can even be reduced to 64. This disclosure is not limited thereto.

[0098] It's worth noting that the upsampling module of each variable frame rate layer uses the same upsampling rate as the downsampling module. Referring to the example above, the upsampling rate of the first variable frame rate layer's upsampling module is also 1024. This ensures that the encoded vector output by the first variable frame rate layer has the same dimension as the feature vector of a frame.

[0099] Therefore, in each variable frame rate layer, compression is performed along the time axis using a causal one-dimensional convolutional layer, where the kernel size and stride are set to the same values ​​as the compression ratio λ to maintain causality. This means that if the input encoded vector Z or a frame of feature data Z is represented as After downsampling, it is converted to As for the upsampling layer, each variable frame rate layer uses interpolation operations and one-dimensional pointwise convolutional layers to predict future frames while maintaining causality, ensuring the logic of time series analysis and the accuracy of prediction.

[0100] like Figure 3 As shown, in the recursive UNet, the encoder and decoder directly pass information through skip connections. This connection mechanism enables rapid feature fusion by directly linking the output of each level in the encoder to the same or corresponding level in the decoder. For example, high-frequency details captured in the early stages of the recursive UNet can be directly input to the later stages of the decoder via skip connections, thus preserving these details during reconstruction. This design allows the decoder to access high-resolution features from the encoder, thereby better reconstructing the details of the signal. For instance, the output of the first encoder layer is connected to the last decoder layer. Assuming the encoder consists of N encoder layers and the decoder consists of N decoder layers, then the output of the i-th encoder layer, after being enhanced by nested information enhancement modules, becomes a decoding auxiliary vector that is input to the Ni-th decoder layer. Of course, this disclosure is not limited thereto.

[0101] The nested information enhancement module (PiP) combines each encoded vector with nested cue information to generate a decoding auxiliary vector. The nested information enhancement module can be inserted as a plug-and-play component into skip connections of a recursive UNet. The nested cue information based on task cue information and enhancement cue information can improve controllability according to embodiments of this disclosure. The task cue information includes information characterizing a specific task type, while the enhancement cue information includes information for signal-level enhancement.

[0102] For example, the nested information enhancement module receives the output Z of the i-th encoder. u As input, and based on learnable nested cue information P, the task type is identified, thereby outputting the corresponding latent features. Used to guide the decoder in performing decoding tasks. The nested information enhancement module acts as an adaptive, lightweight component, capable of encoding degradation context information across multiple scales in the recursive UNet. The entire forward process can be represented by Equation (13).

[0103]

[0104] Where, P = {p1, ..., p} n p B}, where p n This indicates a task prompt message, where n represents the nth task type (e.g., reference). Figure 2 (Description of denoising tasks, denoising tasks, etc.), p BThis indicates enhanced information. Nested information enhancement modules pass and enhance information across different layers of the recursive UNet in this way, thereby improving overall recovery performance. This design not only increases the model's flexibility but also enhances its ability to handle various complex degradation scenarios.

[0105] More specifically, task prompts are provided as input text and instructions to the nested information enhancement module to guide it in generating a specific type of nested prompt message P. Task prompt message p n It can be a one-dimensional embedding vector to represent and embed the corresponding task-specific information, and all task prompt information p n set It can cover N tasks. Specifically, these learnable task cues enable the model to adapt to parameters more effectively and clearly express different concepts for different speech enhancement task types.

[0106] Although task cues can be used to perceive the overall speech enhancement task type, the input feature maps still need to be recalibrated to achieve signal-level enhancement. To address this, enhancement cues p adapted to the latent feature maps are introduced. B This allows for more detailed restoration and calibration. Specifically, an `nn.Parameter()` layer is used to generate basic enhancement cues, which are then used as training parameters to be optimized during speech enhancement. It's worth noting that the parameter space for the enhancement cues is shared across different tasks. `nn.Parameter()` is a class in the PyTorch deep learning framework used to define learnable parameters that are optimized during model training. In neural networks, parameters typically include weights and biases, which are the core components of network layers. Using `nn.Parameter()`, a tensor can be registered as a parameter of the model, making it an optimization target. Of course, this disclosure is not limited to this.

[0107] like Figure 6 As shown, nested prompt information can be generated based on task prompt information and enhanced prompt information through affine transformation. This process can be described as follows: generating modulation features for scaling and modulation features for translation based on the task prompt information; modulating the enhanced prompt information using the modulation feature γ for scaling to generate modulated enhanced prompt information; and generating the nested prompt information based on the modulated enhanced prompt information and the modulation feature for translation. Of course, this disclosure is not limited thereto.

[0108] Specifically, the i-th task prompt message p iFirst, preliminary processing is performed using multiple linear layers and trigonometric function Tanh layers. The task prompt information p after preliminary processing is then generated. i Two 1D convolutional layers and a Tanh function are fed into the system to generate modulation features γ for scaling and β for translation, enabling conditional fusion of task and augmented cue information. The scaling feature γ and the augmented cue information are element-wise multiplied, and then added to the translation feature β to form the modulated nested cue information P. Through this conditional feature transformation, the modulated nested cue information P can simultaneously possess good task representation and feature detail expression capabilities. Figure 6 The process can be represented by formula (14) to formula (16).

[0109] γ=Tanh(Conv1d(p))(14)

[0110] β=Tanh(Conv1d(p))(15)

[0111]

[0112] Optionally, in the process of generating the decoding auxiliary vector corresponding to the encoder layer, the optional embodiments of this disclosure may perform the following operations: projecting the encoded vector output by the encoder layer onto the query matrix Q; projecting the nested prompt information onto the key matrix K and the value matrix V; and determining the decoding auxiliary vector corresponding to the encoder layer based on the nonlinear transformation (CAtt(Q, K, V)) of the query matrix Q, the key matrix K, and the value matrix V.

[0113] Specifically, in the process of adjusting at least one of the plurality of encoded vectors using the nested cue information to generate at least one decoding auxiliary vector, a cross-attention layer is used to fuse the nested cue information and an encoded vector output by an encoder layer. In this way, optional embodiments of this disclosure can further utilize the nested cue information to control features related to the speech enhancement task. Specifically, the encoded vector output by any encoder layer is projected onto a query matrix Q, and the nested cue information is projected onto a key matrix K and a value matrix V. These projection processes can be performed using a learnable linear projection l. Q l K l V The process can be represented by formulas (17) to (20).

[0114] Q = l Q (Z i (17)

[0115] K = l K (P) (18)

[0116] V = l V (P) (19)

[0117]

[0118] Where d is the latent projection dimension of the key matrix and query matrix. The output of cross-attention is a weighted average of the value matrix V, where the weights involved in this weighted averaging process are the attention map. This attention map can be derived by calculating the correlation between Q and K. Furthermore, multi-head attention is used in parallel to improve its performance, and the results are then passed through a learned linear layer to obtain the final latent features. Therefore, the nested information enhancement module can simultaneously consider information between different tasks and between tasks and features, systematically learn the interactions and dependencies between them, thereby achieving generalization for various speech enhancement tasks and performing appropriate enhancement operations.

[0119] In the third stage, a frequency band combining module can be used to generate speech enhancement data based on the decoding vector. For example... Figure 7 As shown, the band combining module can perform three operations: subband segmentation, data processing, and combining. Of course, the band combining module can also perform more operations. Specifically, the band combining module takes the output of the decoder in the encoder-decoder module—the decoded vector U—as input, and uses the subband segmentation operation to decompose the decoded vector U into mask features {U1, ..., U2} corresponding to multiple subbands. Q Mask features {U1, ..., U} Q} is capable of restoring speech signal x to speech signal on each sub-band. The mask. The mask feature U corresponding to each sub-band. q It is fed into a normalization layer. After normalization, the mask feature U corresponding to each sub-band is... q The input then enters a multilayer perceptron (MLP) layer. The MLP layer, through its nonlinear transformation capabilities, models the features of each sub-band and estimates the corresponding complex-valued time-frequency mask G. q Here, the subscript q represents each subband from 1 to Q, where Q is the total number of subbands.

[0120] Each G q Containing the gain information of the corresponding sub-bands in the time-frequency plane, these masks are stacked and merged along the frequency axis to form a unified TF mask G. The dimension of G matches the time-frequency representation of the original input signal X, ensuring that it can be used to effectively filter X. Finally, the TF mask G is multiplied element-wise with the original noisy input speech signal X to form the final target spectrum. This process then restores a clean speech signal s. Essentially, this step involves selectively enhancing or suppressing the input signal in the time-frequency domain to remove noise and echoes while preserving or enhancing the useful sound signal. Mathematically, this process can be described by equations (21) to (24).

[0121] {U1, ..., U Q} = Sub-bandSplit(U) (21)

[0122] G q =MLP g (Norm(U q )) (twenty two)

[0123] G = Merge(G1, ..., G) Q ) (twenty three)

[0124]

[0125] The improved neural network model according to embodiments of this disclosure combines an encoder-decoder module with a nested information enhancement module, utilizing decoding auxiliary vectors to decode the encoded vectors. These decoding auxiliary vectors integrate task prompts indicating the type of speech enhancement task with semantic enhancement prompts that enable speech enhancement at the signal level. Due to the flexibility of the decoding auxiliary vectors, embodiments of this disclosure can adapt to more types of speech enhancement tasks, significantly improving the flexibility of speech enhancement technology. For example, in some embodiments of this disclosure, decoding auxiliary vectors can be flexibly applied to the connections of the encoder-decoder network to obtain the ability to handle various speech enhancement tasks during encoding-decoding, making the speech enhancement system more efficient and flexible when dealing with complex speech data.

[0126] Figure 8 A flowchart of a method 80 for speech enhancement according to an embodiment of the present disclosure is shown.

[0127] like Figure 8 As shown, method 80 can be used on a terminal device or server (such as...) Figure 1 The method is executed at the terminal 120 or server 110. Method 80 includes the following operations S801 to S806. Of course, method 80 may also include more or fewer operations, and this disclosure is not limited thereto.

[0128] In operation S801, multi-frame feature data is determined based on the time-frequency data corresponding to the voice data.

[0129] In operation S802, multiple encoding vectors are generated based on the feature data of at least one frame of the multi-frame feature data.

[0130] Optionally, generating multiple encoded vectors based on the feature data of at least one frame in the multi-frame feature data includes: using multiple cascaded encoder layers to determine the encoded vector output by each encoder layer, wherein, except for the first encoder layer, the input vector of each encoder layer is the encoded vector output by the previous encoder layer, the input vector of the first encoder layer is the feature data of the at least one frame, and the encoded vector output by the last encoder layer is input to the first decoder layer in the decoder.

[0131] Optionally, determining the encoded vector output by each encoder layer includes: downsampling the input vector to the encoder layer based on the downsampling rate corresponding to the encoder layer; generating a hidden vector that fuses time-domain information based on the downsampled input vector; generating a hidden vector that fuses both time-domain and frequency-domain information based on the hidden vector that fuses the time-domain information; and upsampling the hidden vector that fuses both time-domain and frequency-domain information based on the upsampling rate corresponding to the encoder layer; wherein the downsampling rate and upsampling rate are different for different encoder layers.

[0132] In operation S803, nested prompt information is generated based on task prompt information and enhanced prompt information.

[0133] Optionally, the task cue information includes information for characterizing a specific task type, and the enhancement cue information includes information for enhancing the signal at the signal level. The task type includes any one of the following: denoising task, dereverberation task, bandwidth expansion task, and proximity effect removal task.

[0134] Optionally, generating nested prompt information based on task prompt information and enhanced prompt information includes: generating modulation features for scaling and modulation features for translation based on the task prompt information; modulating the enhanced prompt information using the modulation features for scaling to generate modulated enhanced prompt information; and generating the nested prompt information based on the modulated enhanced prompt information and the modulation features for translation.

[0135] In operation S804, at least one of the multiple encoding vectors is adjusted using the nested prompt information to generate at least one decoding auxiliary vector.

[0136] Optionally, adjusting at least one of the plurality of encoding vectors using the nested cue information to generate at least one decoding auxiliary vector includes: adjusting the encoding vector output by each encoder layer using the nested cue information to generate a decoding auxiliary vector corresponding to the encoder layer, wherein the decoding auxiliary vector corresponding to the encoder layer is used for decoding by a decoder layer at the same level as the encoder layer.

[0137] Optionally, adjusting the encoding vector output of each encoder layer using the nested prompt information to generate the decoding auxiliary vector corresponding to the encoder layer includes: projecting the encoding vector output by the encoder layer onto a query matrix; projecting the nested prompt information onto a key matrix and a value matrix; and determining the decoding auxiliary vector corresponding to the encoder layer based on a nonlinear transformation of the query matrix, the key matrix, and the value matrix.

[0138] In operation S805, a decoding vector is generated based on at least one of the plurality of encoding vectors and the at least one decoding auxiliary vector.

[0139] Optionally, generating a decoding vector based on at least one of the plurality of encoding vectors and the at least one decoding auxiliary vector includes: using a plurality of cascaded decoder layers to determine the decoding vector output by each decoder layer, wherein, except for the first decoder layer, the input vector of each decoder layer is the decoding vector output by the previous decoder layer and the decoding auxiliary vector corresponding to the encoder at the same level as the decoder layer.

[0140] In operation S806, speech enhancement data is generated based on the decoding vector.

[0141] In operation S806, speech enhancement data can also be generated based on both the time-frequency data corresponding to the speech data and the decoding vector. This disclosure is not limited thereto.

[0142] The above has been referenced. Figures 3 to 7The details of the neural network have been described in detail, and will not be repeated here. Therefore, the speech enhancement method 80 according to embodiments of this disclosure decodes the encoded vector based on an encoded vector and a decoding auxiliary vector, wherein the decoding auxiliary vector integrates task prompt information indicating the type of speech enhancement task and enhancement prompt information capable of achieving speech enhancement at the signal level. Due to the flexibility of the decoding auxiliary vector, embodiments of this disclosure can adapt to more types of speech enhancement tasks, significantly improving the flexibility of speech enhancement technology. In some embodiments of this disclosure, the decoding auxiliary vector can be flexibly applied to the connection of the encoder-decoder network to obtain the ability to handle various speech enhancement tasks during encoding-decoding, making the speech enhancement system more efficient and flexible when dealing with complex speech data.

[0143] Figure 9 A flowchart of a method 90 for training a neural network model according to an embodiment of the present disclosure is shown, which can train a neural network model. Figures 3 to 7 The improved neural network model shown is trained. The neural network model includes: a frequency band segmentation module, an encoder-decoder module, a nested information enhancement module, and a frequency band merging module.

[0144] Method 90 can be used on terminal devices or servers (such as...) Figure 1 The method is executed at the terminal 120 or server 110. Method 90 includes the following operations S901 to S907. Of course, method 90 may also include more or fewer operations, and this disclosure is not limited thereto.

[0145] In operation S901, based on the time-frequency data corresponding to the sample speech data, the frequency band segmentation module is used to determine multi-frame feature data.

[0146] In operation S902, based on the feature data of at least one frame of the multi-frame feature data, multiple encoding vectors are generated using the encoder in the encoder-decoder module.

[0147] In operation S903, nested prompt information is generated using the nested information enhancement module based on task prompt information and enhanced prompt information.

[0148] In operation S904, at least one of the multiple encoding vectors is adjusted using the nested prompt information to generate at least one decoding auxiliary vector.

[0149] In operation S905, a decoding vector is generated using the decoder in the encoder-decoder module based on at least one of the plurality of encoding vectors and the at least one decoding auxiliary vector.

[0150] In operation S906, speech enhancement data is generated using the frequency band combining module based on the decoding vector.

[0151] In operation S907, one or more modules in the neural network model are trained using the enhanced data corresponding to the enhanced data of the speech enhancement data and the sample speech data.

[0152] Optionally, training one or more modules in the neural network model includes: calculating the value of one or more of the mean absolute error loss function, signal-to-noise ratio loss function, and multi-resolution short-time Fourier transform loss function based on the enhanced data corresponding to the enhanced data and the sample speech data; and adjusting the parameters in one or more modules of the neural network model based on the value of one or more of the mean absolute error loss function, signal-to-noise ratio loss function, and multi-resolution short-time Fourier transform loss function.

[0153] The training of the neural network model can use the mean absolute error (MAE) loss function, which can be expressed as formula (25).

[0154]

[0155] in, represent the real part, imaginary part, and amplitude part of the estimated speech enhancement data spectrum, respectively, while {S} R S I , where |S|} represents the corresponding part of the augmented data corresponding to the sample speech data.

[0156] Optionally, a multi-resolution short-time Fourier transform (MR-STFT) loss can be further used to train the neural network model to improve speech quality. Specifically, assuming the estimated STFT spectrum of the speech enhancement data is represented as... The STFT spectrum of the augmented data corresponding to the sample speech data is represented as S k , where k∈K represents the size of the Fast Fourier Transform (FFT). The STFT loss is defined as the sum of the normalized amplitude spectral distance and the logarithmic amplitude spectral distance under different FFT sizes, as shown in Equations (26) and (27).

[0157]

[0158]

[0159] Among them, ||·||1 and ||·|| FLet L1 and Frobenius norms represent the L1 norm and Frobenius norm, respectively. K is the set of sizes for the FFT.

[0160] Optionally, to further perceive the state of noise suppression, the signal-to-noise ratio (SNR) loss can also be used. SNR The loss function is used to measure the difference between the estimated speech enhancement data and the enhancement data corresponding to the sample speech data.

[0161] Alternatively, the weighted sum of the above three loss functions can be used as the final loss function, as shown in formula (28).

[0162] L=αL MAE +vL MR-STFT +kL SNR (28)

[0163] Where {α, v, k} refers to hyperparameters.

[0164] The training process described above, combining mean absolute error (MAE), multi-resolution short-time Fourier transform (MR-STFT) loss, and signal-to-noise ratio (SNR) loss, offers significant advantages. The embodiments disclosed herein can evaluate model performance from multiple perspectives, ensuring that the model achieves high standards in several key aspects such as prediction accuracy, spectral quality, and noise suppression. MAE loss helps reduce prediction errors, MR-STFT loss improves spectral detail, and SNR loss focuses on improving speech clarity. Furthermore, by adjusting the hyperparameters α, v, and k, the impact of these loss functions on the training process can be flexibly balanced, making the improved neural network model more adaptable to different application requirements.

[0165] The training method according to embodiments of this disclosure also enhances the robustness and generalization ability of the model. It enables the neural network model to better adapt to various speech characteristics and noisy environments, thereby exhibiting better stability and reliability in practical applications. Furthermore, this training method using a comprehensive loss function also improves the interpretability of the model's performance.

[0166] like Figure 10 As shown, the improved neural network model delivers significantly better speech enhancement results compared to traditional methods.

[0167] According to embodiments of this disclosure, an apparatus for speech enhancement is also provided, the apparatus comprising: a frequency band segmentation module configured to: determine the representation of text data in a latent space based on text data; an encoder configured to: generate a plurality of encoding vectors based on feature data of at least one frame of the multi-frame feature data; a nested information enhancement module configured to: generate nested prompt information based on task prompt information and enhancement prompt information, and adjust at least one of the plurality of encoding vectors using the nested prompt information to generate at least one decoding auxiliary vector; a decoder configured to: generate a decoding vector based on at least one of the plurality of encoding vectors and the at least one decoding auxiliary vector; and a frequency band merging module configured to: generate speech enhancement data based on the decoding vector.

[0168] According to another aspect of this disclosure, an electronic device is also provided for implementing the methods according to embodiments of this disclosure. Figure 11 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0169] like Figure 11 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code that, when executed by the one or more processors 2010, can perform the methods described above.

[0170] The processor in this disclosure embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, operations, and logic block diagrams disclosed in this disclosure embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0171] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0172] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 12 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 12 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 12 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 12 One or more components in the computing device shown.

[0173] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 13 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0174] like Figure 13 As shown, the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the methods according to embodiments of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0175] This disclosure also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to an embodiment of this disclosure.

[0176] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0177] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0178] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A method for speech enhancement, comprising: Based on the time-frequency data corresponding to the speech data, determine multi-frame feature data; Based on the feature data of at least one frame of the multi-frame feature data, generate multiple encoding vectors; Generate nested prompts based on task prompts and enhanced prompts; The nested prompt information is used to adjust at least one of the multiple encoded vectors to generate at least one decoding auxiliary vector; A decoding vector is generated based on at least one of the plurality of encoding vectors and at least one decoding auxiliary vector; as well as Based on the decoding vector, speech enhancement data is generated.

2. The method as described in claim 1, wherein, The task prompt information includes information for characterizing a specific task type, and the enhanced prompt information includes information for enhancing the signal at the signal level.

3. The method as described in claim 1, wherein, The generation of nested prompt information based on task prompt information and enhanced prompt information includes: Based on the task prompt information, modulation features for scaling and modulation features for translation are generated; The enhanced prompt information is modulated using modulation features for scaling to generate modulated enhanced prompt information; and The nested prompt information is generated based on the modulated enhanced prompt information and the modulation features used for translation.

4. The method of claim 2, wherein, The step of generating multiple encoded vectors based on feature data from at least one frame of the multi-frame feature data includes: By utilizing multiple cascaded encoder layers, the encoded vector output by each encoder layer is determined. In this process, except for the first encoder layer, the input vector of each encoder layer is the encoded vector output by the previous encoder layer, the input vector of the first encoder layer is the feature data of at least one frame, and the encoded vector output by the last encoder layer is input to the first decoder layer in the decoder.

5. The method of claim 4, wherein, The process of determining the encoded vector output by each encoder layer includes: Based on the downsampling rate corresponding to the encoder layer, the input vector input to the encoder layer is downsampled; Based on the downsampled input vector, a hidden vector that incorporates time-domain information is generated. Based on the hidden vector fused with both time-domain and frequency-domain information, a hidden vector is generated; and Based on the upsampling rate corresponding to the encoder layer, the hidden vector of the fused time-domain information and frequency-domain information is upsampled; The downsampling rate and upsampling rate are different for different encoder layers.

6. The method of claim 4, wherein, The step of adjusting at least one of the plurality of encoded vectors using the nested prompt information to generate at least one decoding auxiliary vector includes: Using the nested prompt information, the output encoding vector of each encoder layer is adjusted to generate the decoding auxiliary vector corresponding to the encoder layer. The decoding auxiliary vector corresponding to the encoder layer is used for decoding of the decoder layer at the same level as the encoder layer.

7. The method of claim 6, wherein, The step of adjusting the output encoding vector of each encoder layer using the nested cue information to generate the decoding auxiliary vector corresponding to the encoder layer includes: Project the encoded vector output by the encoder layer onto the query matrix; Project the nested prompt information onto the key matrix and value matrix; and Based on the nonlinear transformation of the query matrix, the key matrix, and the value matrix, the decoding auxiliary vector corresponding to the encoder layer is determined.

8. The method of claim 6, wherein, The step of generating a decoding vector based on at least one of the plurality of encoding vectors and at least one decoding auxiliary vector includes: By utilizing multiple cascaded decoder layers, the decoded vector output by each decoder layer is determined. Except for the first decoder layer, the input vector of each decoder layer is the decoded vector output by the previous decoder layer and the decoder auxiliary vector corresponding to the encoder at the same level as the decoder layer.

9. The method of claim 2, wherein, The task types include any one of the following: denoising task, de-reverberation task, bandwidth expansion task, and proximity effect removal task.

10. A method for training a neural network model, the neural network model comprising: The neural network model includes: a frequency band segmentation module, an encoder-decoder module, a nested information enhancement module, and a frequency band merging module; the method includes: Based on the time-frequency data corresponding to the sample speech data, the frequency band segmentation module is used to determine multi-frame feature data. Based on the feature data of at least one frame in the multi-frame feature data, multiple encoding vectors are generated using the encoder in the encoder-decoder module. Based on task prompts and enhanced prompts, nested prompts are generated using the nested information enhancement module. The nested prompt information is used to adjust at least one of the multiple encoded vectors to generate at least one decoding auxiliary vector; Based on at least one of the plurality of encoded vectors and at least one decoding auxiliary vector, a decoding vector is generated using the decoder in the encoder-decoder module; Based on the decoding vector, speech enhancement data is generated using a frequency band combining module; and One or more modules in the neural network model are trained using the speech enhancement data and the enhancement data corresponding to the sample speech data.

11. The method of claim 10, wherein, The training of one or more modules in the neural network model includes: Based on the enhanced data corresponding to the speech enhancement data and the sample speech data, calculate the value of one or more of the mean absolute error loss function, the signal-to-noise ratio loss function, and the multi-resolution short-time Fourier transform loss function; and Based on the value of one or more of the mean absolute error loss function, signal-to-noise ratio loss function, and multi-resolution short-time Fourier transform loss function, the parameters in one or more modules of the neural network model are adjusted.

12. An apparatus for speech enhancement, the apparatus comprising: The frequency band segmentation module is configured to: determine the representation of the text data in the latent space based on the text data; The encoder is configured to generate multiple encoding vectors based on feature data from at least one frame of the multi-frame feature data. The nested information enhancement module is configured to: generate nested prompt information based on task prompt information and enhanced prompt information, and adjust at least one of the multiple encoding vectors using the nested prompt information to generate at least one decoding auxiliary vector; The decoder is configured to generate a decoding vector based on at least one of the plurality of encoded vectors and the at least one decoding auxiliary vector; as well as The frequency band combining module is configured to generate speech enhancement data based on the decoding vector.

13. An electronic device, comprising: One or more processors; and One or more memories, wherein the memories store a computer-executable program that, when executed by the processor, performs the method of any one of claims 1-11.

14. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as claimed in any one of claims 1-11.

15. A computer program product comprising computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads from the computer-readable medium and executes the computer instructions, causing the computer device to perform the method as described in any one of claims 1-11.

Images