Audio processing method, apparatus, device, storage medium and program product

Abstract

The present disclosure provides an audio processing method, device, equipment, storage medium and program product, relating to the technical field of data processing. The method comprises: obtaining audio to be processed; and using an encoder to encode the audio to be processed to obtain an encoded feature map, the encoded feature map comprising first audio features and second audio features; using an attention mechanism to generate attention weights based on the encoded feature map, and performing weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map, the weighted feature map comprising enhanced first audio features and suppressed second audio features; and using a decoder to perform decoding processing on the weighted feature map to obtain separated first audio and second audio. In this scheme, the first audio is enhanced and the second signal is suppressed through the attention mechanism, more accurate audio separation is achieved, and the audio quality of the first audio is improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to an audio processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] During voice recording, recording equipment typically captures various environmental noises, such as vehicle noise, horn sounds, and wind noise. Noise interference weakens the clarity of the valid audio signal, leading to a decrease in the quality of the captured audio.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This disclosure provides an audio processing method, apparatus, device, storage medium, and program product, which at least to some extent overcomes the technical problem in the related art where interfering sounds weaken the clarity of the effective audio signal, leading to a decrease in the quality of the acquired audio.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of this disclosure, an audio processing method is provided, comprising: acquiring an audio to be processed; encoding the audio to be processed using an encoder to obtain an encoded feature map, the encoded feature map including a first audio feature and a second audio feature; generating attention weights based on the encoded feature map using an attention mechanism; performing weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map, the weighted feature map including an enhanced first audio feature and a suppressed second audio feature; and decoding the weighted feature map using a decoder to obtain separated first and second audio.

[0007] In some embodiments, the encoder includes at least one cascaded downsampling block, the attention mechanism includes at least one attention module, and the decoder includes at least one cascaded upsampling block, with each downsampling block corresponding to an upsampling block; the downsampling block is connected to its corresponding upsampling block via an attention module.

[0008] In some embodiments, the number of attention modules is less than or equal to the amount of data in the downsampling block, and each attention module includes a channel attention module and / or a spatial attention module.

[0009] In some embodiments, attention weights include channel attention weights, and the weighted feature map includes a first weighted feature map; generating attention weights based on the encoded feature map using an attention mechanism includes: generating channel attention weights based on the encoded feature map through global pooling and a multilayer perceptron; the channel attention weights include the weights corresponding to each channel; performing weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map includes: calculating based on the feature responses corresponding to each channel and the weights corresponding to each channel in the encoded feature map to obtain a first weighted feature map, the first weighted feature map including a first audio feature for channel enhancement and a second audio feature for channel suppression.

[0010] In some embodiments, channel attention weights are generated based on the encoded feature map through global pooling and a multilayer perceptron, including: performing global average pooling and global max pooling operations on the encoded feature map to obtain two channel-level description vectors; processing the two channel-level description vectors using a multilayer perceptron to obtain two high-level semantic vectors; adding the elements at corresponding positions in the two high-level semantic vectors and generating channel attention weights through an activation function.

[0011] In some embodiments, the attention mechanism includes a spatial attention mechanism, the attention weights include spatial attention weights, and the weighted feature map includes a second weighted feature map; generating attention weights based on the encoded feature map using the attention mechanism includes: recognizing the input feature map using the spatial attention mechanism to generate a spatial attention map, wherein the weights in the spatial attention map are used to characterize the correlation between the corresponding time-frequency point and the input feature map, and the input feature map includes an encoded feature map or a first weighted feature map; performing weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map includes: performing weighted fusion of the spatial attention map and the input feature map to obtain a second weighted feature map, wherein the second weighted feature map includes a spatially enhanced first audio feature and a spatially suppressed second audio feature.

[0012] In some embodiments, using a spatial attention mechanism to identify the input feature map and generate a spatial attention map includes: using a spatial attention mechanism to identify the response intensity of each time-frequency point in the input feature map and the spatial context relationship between each time-frequency point; and generating a spatial attention map corresponding to the input feature map based on the response intensity of each time-frequency point and the spatial context relationship between each time-frequency point.

[0013] In some embodiments, decoding the weighted feature map using a decoder to obtain the separated first and second audio includes: decoding the weighted feature map using a decoder to obtain a first amplitude spectrum and a second amplitude spectrum; calculating the first audio energy and the second audio energy based on the first and second amplitude spectra; calculating a first time-frequency mask and a second time-frequency mask based on the first and second audio energy; and multiplying the audio to be processed by the first time-frequency mask and the second time-frequency mask respectively to obtain the separated first and second audio.

[0014] According to another aspect of this disclosure, an audio processing apparatus is also provided, comprising: an acquisition module for acquiring audio to be processed; an encoding module for encoding the audio to be processed using an encoder to obtain an encoded feature map, the encoded feature map including a first audio feature and a second audio feature; a weight generation module for generating attention weights based on the encoded feature map using an attention mechanism; a weighted processing module for performing weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map, the weighted feature map including an enhanced first audio feature and a suppressed second audio feature; and a decoding module for decoding the weighted feature map using a decoder to obtain separated first and second audio.

[0015] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the audio processing method of any of the above via executing the executable instructions.

[0016] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the audio processing method of any of the above.

[0017] According to another aspect of this disclosure, a computer program product is also provided, comprising: a computer program or instructions that, when executed by a processor, implement the audio processing method of any one of the above.

[0018] The audio processing method provided in the embodiments of this disclosure first acquires the audio to be processed; then, it encodes the audio using an encoder to obtain an encoded feature map, which includes a first audio feature and a second audio feature; next, it generates attention weights based on the encoded feature map using an attention mechanism, and performs weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map, which includes enhanced first audio features and suppressed second audio features; finally, it decodes the weighted feature map using a decoder to obtain the separated first and second audio signals. In this scheme, the attention mechanism enhances the first audio signal and suppresses the second signal, achieving more accurate audio separation and thus improving the audio quality of the first audio signal.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 This diagram illustrates a system architecture diagram of an audio processing method according to an embodiment of the present disclosure.

[0022] Figure 2 This diagram illustrates a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0023] Figure 3 This diagram illustrates an implementation of an audio preprocessing method according to an embodiment of the present disclosure.

[0024] Figure 4 This diagram illustrates a flowchart of yet another audio processing method according to an embodiment of the present disclosure;

[0025] Figure 5 This diagram illustrates another audio processing method according to an embodiment of the present disclosure;

[0026] Figure 6 A flowchart illustrating the implementation of an audio separation model prediction method in an embodiment of this disclosure is shown.

[0027] Figure 7 A flowchart illustrating the implementation of an audio noise reduction method according to an embodiment of this disclosure is shown.

[0028] Figure 8 This diagram illustrates an audio processing apparatus according to an embodiment of the present disclosure;

[0029] Figure 9 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0030] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0031] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] To facilitate understanding, before introducing the embodiments of this disclosure, the following explanations are provided for several terms involved in the embodiments of this disclosure:

[0033] Speech noise reduction: When a speech signal is interfered with or even submerged by various background noises, the goal is to extract useful or clean speech signals from the noisy speech signal as much as possible, and to suppress or reduce noise interference.

[0034] Spleeter is a deep learning-based voice separation technology that uses a U-Net (Convolutional Networks for Biomedical Image Segmentation) network structure to separate mixed audio signals into two parts: voice signals and background signals. It utilizes feature learning in the time-frequency domain to model and separate different sound sources in the audio signal in an end-to-end manner.

[0035] U-Net is a typical convolutional neural network structure, named for its symmetrical "U" shape. U-Net consists of an encoder and a decoder, combining the high-resolution features extracted by the encoder with the reconstruction process of the decoder through skip connections, which helps preserve key information.

[0036] Attention mechanism: It is a neural network mechanism that mimics the human visual focusing ability. It can assign different weights according to the importance of input features, thereby highlighting key features and suppressing redundant information.

[0037] Channel Attention Module (CAM) is a method for dynamically adjusting the feature weights of different channels in a neural network. Its core idea is to model the importance of each channel to the final output, improve the feature channel response related to the target signal, and suppress the channel response related to noise.

[0038] Spatial Attention Module (SAM): It focuses on the contribution of each spatial location in the feature map (such as the pixel position in an image or the time-frequency point in a speech spectrogram) to the task objective. By generating a spatial attention map, it highlights important locations and weakens irrelevant backgrounds, thereby improving the model's ability to extract target signals in complex backgrounds.

[0039] In some applications, it is often necessary to use recording equipment to capture speech, and then use speech recognition technology to convert the captured speech signal into text. However, during the speech capture process, recording equipment usually picks up various environmental noises, such as car noises, horns, engine noises, and wind noises. This noise interference weakens the clarity of the effective speech signal, reduces the intelligibility and perceptual quality of the speech signal, and consequently affects the accuracy of speech recognition, reducing the accuracy of the converted text. Therefore, it is necessary to perform noise reduction on the captured audio signal to improve the accuracy of subsequent speech recognition.

[0040] To solve the above technical problems, there are two main technical solutions: one is based on data signal processing, and the other is based on machine learning.

[0041] Data signal processing-based methods mainly include spectral subtraction, Wiener filtering, etc. These methods achieve noise reduction by estimating the acoustic features of speech and noise. They have good effects on stationary noise signals, but poor noise reduction effects on highly non-stationary signals in real-life environments.

[0042] Machine learning-based methods require extensive training on large amounts of data to establish a non-linear mapping from noisy speech to clean speech. However, the diversity and complexity of pure noise data in real-world applications make its acquisition nearly impossible. Consequently, the noise reduction networks trained in this way often exhibit poor noise reduction performance, resulting in either incomplete or excessive noise suppression, leading to damage to the human voice. For these reasons, the human voice in the audio after noise reduction algorithms either still contains a significant amount of noise or is damaged, resulting in voice distortion and failing to improve the accuracy of subsequent speech recognition.

[0043] Therefore, the audio processing method provided in the embodiments of this disclosure firstly acquires the audio to be processed; then, it encodes the audio using an encoder to obtain an encoded feature map, which includes a first audio feature and a second audio feature; next, it uses an attention mechanism to generate attention weights based on the encoded feature map, and performs weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map, which includes enhanced first audio features and suppressed second audio features; finally, it uses a decoder to decode the weighted feature map to obtain the separated first and second audio. In this scheme, by introducing an attention mechanism on the basis of the human voice separation model, it strengthens the attention to key time-frequency positions and dynamically adjusts the features of different channels, thereby better separating human voice from background sound, extracting clean human voice, and thus improving the audio quality of speech, thereby achieving the effect of indirect noise reduction.

[0044] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0045] Figure 1 A schematic diagram of an exemplary application system architecture to which the audio processing methods of the embodiments of this disclosure can be applied is shown. For example... Figure 1 As shown, the system architecture may include terminal device 101, network 102 and server 103.

[0046] It should be noted that the terminal device 101 in the embodiments of this disclosure refers to an entity on the user side used to receive or transmit signals, and may also be referred to as a terminal device, mobile station (MS), mobile terminal (MT), etc. The embodiments of this disclosure do not limit the specific technology or specific device form used by the terminal.

[0047] In some embodiments of this disclosure, the terminal device 101 may be a mobile phone, tablet computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile internet device (MID), augmented reality (AR), virtual reality (VR) device, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipboard equipment, pedestrian user equipment (PUE), smart home (home devices with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), game console, personal computer (PC), ATM, or self-service machine, etc. Wearable devices include: smartwatches, smart bracelets, smart earphones, smart glasses, smart jewelry (smart bracelets, smart necklaces, smart anklets, smart ankle chains, etc.), smart wristbands, smart clothing, etc. Among these, in-vehicle devices can also be referred to as in-vehicle terminals, in-vehicle controllers, in-vehicle modules, in-vehicle components, in-vehicle chips, or in-vehicle units, etc. It should be noted that the specific type of terminal device 101 is not limited in the embodiments disclosed herein.

[0048] Optionally, the client of the application installed on different terminal devices 101 may be the same, or the client of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also be different; for example, the application client may be a mobile client, a PC client, etc.

[0049] Network 102 is a medium used to provide a communication link between terminal device 101 and server 103, and can be a wired network or a wireless network.

[0050] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats, including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0051] Server 103 can be a server that provides various services, such as a backend management server that supports the device operated by the user using terminal device 101. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal device.

[0052] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0053] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.

[0054] Under the above system architecture, this disclosure provides an audio processing method that can be executed by any electronic device with computing power.

[0055] In some embodiments, the audio processing method provided in this disclosure can be executed by the terminal of the system architecture described above; in other embodiments, the audio processing method provided in this disclosure can be implemented by the terminal and the server in the system architecture through interaction.

[0056] Figure 2 This diagram illustrates a flowchart of an audio processing method on the terminal side according to an embodiment of the present disclosure, such as... Figure 2 As shown, the audio processing method provided in this embodiment includes the following steps S202-S210.

[0057] In step S202, the audio to be processed is obtained.

[0058] The audio to be processed can be understood as digitized audio information comprising multiple mixed audio signals. In other words, the audio data to be processed includes at least a first audio and a second audio that need to be separated. For example, the first audio could be the target human voice, and the second audio could be background music or environmental noise; or, the first and second audios could be the speech of two different speakers. The forms of the audio to be processed include, but are not limited to: time-domain representation, audio representation, and feature representation.

[0059] In one possible implementation, acquiring the audio to be processed includes reading an audio file from a local storage medium such as a hard drive, solid-state drive, memory, or flash memory. The audio file format may include any one or more of the following: WAV (Waveform Audio File Format), MP3 (Moving Picture Experts Group Audio Layer III), AAC (Advanced Audio Coding), and FLAC (Free Lossless Audio Codec).

[0060] Alternatively, audio received via a wired or wireless network interface can be used as the audio to be processed. Examples include audio downloaded from an internet server, data packets from an ongoing VoIP call, or audio clips uploaded by an IoT device.

[0061] Alternatively, the analog signal directly acquired in real time from audio acquisition devices such as microphones and audio acquisition cards can be converted into digital audio through analog-to-digital converter (ADC) and used as the audio to be processed.

[0062] In real-world applications, raw audio sources are diverse, including but not limited to: different microphone models, mobile phone recordings, network audio streams, and historical audio archives, resulting in a wide variety of audio samples, formats, and channel counts to be processed. To achieve optimal processing performance and simplify model design, subsequent encoders, attention mechanisms, and decoders require input data to have a standardized format.

[0063] Therefore, one or more preprocessing operations can be performed on the acquired audio to be processed, converting it into a preset standard audio format, thereby providing a unified input data for the subsequent encoder and ensuring the stability and reliability of the separation effect.

[0064] like Figure 3 As shown, the preprocessing operation S310 above includes at least one of the following: S311, sampling rate conversion; S312, channel conversion; S313, bit depth conversion; and S314, audio file format conversion.

[0065] Sampling rate conversion can be understood as converting the sampling rate of the audio to be processed to a target sampling rate so that all audio is processed on the same time scale. For example, the target sampling rate could be 16kHz.

[0066] Channel conversion can be understood as converting a stereo or multi-channel audio mix into mono audio. The core of voice separation is the separation of frequency and temporal characteristics, not spatial information. Merging stereo into mono does not lose the crucial information needed to complete the task; on the contrary, it eliminates interference caused by channel differences.

[0067] Bit depth conversion can be understood as converting the bit depth of the audio to be processed into a target bit depth. For example, the target bit depth can be 16 bits.

[0068] Audio file format conversion can be understood as decoding compressed or other uncompressed audio and repackaging it into a target format. For example, the target format can be an uncompressed WAV format.

[0069] Compression formats introduce audio artifacts and information loss, which in turn interfere with feature extraction by the model. Converting to uncompressed WAV format allows the subsequent model to process the purest waveform data.

[0070] In one possible implementation, the audio data to be processed after preprocessing has a sampling rate of 16kHz, a file format of WAV, a bit depth of 16 bits, and a mono channel.

[0071] In one possible implementation, the above preprocessing operation further includes: performing a Fourier transform on the audio to be processed to obtain the original time spectrum of the audio to be processed, the original time spectrum being sorted according to a certain time sequence, and using the original time spectrum as the input of the encoder.

[0072] In step S204, the audio to be processed is encoded using an encoder to obtain an encoded feature map, which includes a first audio feature and a second audio feature.

[0073] This embodiment provides an audio separation model, also known as an audio track separation model. This model is built upon a U-Net architecture and includes an encoder and a decoder. The encoder extracts audio features from the original time-spectrum image, and the decoder reconstructs the audio signal from these features. A skip connection exists between the encoder and decoder to supplement higher-level information and improve separation accuracy.

[0074] The encoder described above includes an input block and at least one consecutive downsampling block cascaded together. The encoder progressively extracts coded features from the time-spectrum of the audio to be processed. Each downsampling block is a stage of its respective encoder, used to reduce the size of the time-spectrum of the audio to be processed and double the channel data, thereby obtaining audio features at different scales. For example, the encoder in the audio separation model described above includes four downsampling blocks, which can perform four downsampling operations to generate four feature maps of different sizes.

[0075] As discussed above, the encoded feature map can include multiple downsampled feature maps of different sizes, and the number of downsampled feature maps it includes is consistent with the number of downsampled blocks in the encoder. In other words, each downsampled block performs a downsampling operation on the feature map transmitted to it, obtaining the downsampled feature map corresponding to that downsampled block. The downsampled feature maps output by multiple downsampled blocks constitute the encoded feature map.

[0076] In one possible implementation, taking an encoder comprising four downsampling blocks as an example, the encoder's input block receives the time-spectrum of the audio to be processed, transmits this time-spectrum to the first downsampling block, performs the first downsampling operation to obtain the first downsampling feature map, transmits the first downsampling feature map to the second downsampling block, performs the second downsampling operation to obtain the second downsampling feature map, transmits the second downsampling feature map to the third downsampling block, performs the third downsampling operation to obtain the third downsampling feature map, and transmits the third downsampling feature map to the fourth downsampling block, performs the fourth downsampling operation to obtain the fourth downsampling feature map. Therefore, the encoded feature maps include: the first downsampling feature map, the second downsampling feature map, the third downsampling feature map, and the fourth downsampling feature map.

[0077] The first audio feature can be understood as the audio feature obtained after feature extraction from the first audio audio, and the second audio feature can be understood as the audio feature obtained after feature extraction from the second audio audio. The encoded feature map includes multiple first audio features and multiple second audio features. That is, each downsampled feature map includes both first and second audio features.

[0078] It should be noted that the first audio feature and the second audio feature are not physically isolated or mutually exclusive subsets of the downsampled feature map. The first audio feature can be understood as a feature in the downsampled feature map that has a strong statistical correlation with the acoustic properties of the first audio source (e.g., human voice). The second audio feature can be understood as a feature in the downsampled feature map that has a strong statistical correlation with the acoustic properties of the second audio source (e.g., background noise).

[0079] Within the same downsampled feature map, the first and second audio features coexist linearly or nonlinearly superimposed and intertwined. Specifically, the same feature channel may simultaneously respond to specific patterns from different sound sources, but the intensity of its response varies at different times and frequencies. The encoder, through its distributed representation capabilities, simultaneously and completely encodes the information from the two sound sources in the mixed signal within the same feature tensor. The information from the first and second audio sources shares the same tensor basis in the feature space, but its information content is distributed across different combinations of feature dimensions.

[0080] In step S206, attention weights are generated based on the encoded feature map using an attention mechanism.

[0081] In some embodiments, the encoder includes at least one cascaded downsampling block, the attention mechanism includes at least one attention module, and the decoder includes at least one cascaded upsampling block, with each downsampling block corresponding to an upsampling block; the downsampling block is connected to its corresponding upsampling block via an attention module.

[0082] In the audio separation model, the decoder and encoder exist symmetrically, and the decoder includes at least one cascaded upsampling block. The number of upsampling blocks in the decoder is the same as the number of downsampling blocks in the encoder, and there is a one-to-one correspondence between the upsampling blocks in the decoder and the downsampling blocks in the encoder. That is, for each downsampling block in the encoder, there is a corresponding upsampling block in the decoder, and the corresponding downsampling blocks and upsampling blocks are connected in a skip-cascade manner.

[0083] The decoder uses transposed convolution or interpolation operations to progressively restore the low-resolution, high-dimensional abstract features output by the encoder to the original input size. After upsampling at each level, the decoder fuses feature maps from the corresponding level of the encoder through skip connections to reconstruct a high-resolution output mask.

[0084] The attention mechanism provided in this embodiment includes at least one attention module. The number of attention modules is less than or equal to the number of downsampling blocks. For example, the number of attention modules is the same as the number of downsampling blocks. An attention module is added in each skip cascade of downsampling blocks and their corresponding upsampling blocks. In other words, a downsampling block is skip cascaded with its corresponding upsampling block through an attention module. The attention module is used to generate the attention weights corresponding to the downsampling feature map based on the downsampling feature map output by its corresponding downsampling block.

[0085] In one possible implementation, taking an attention mechanism comprising four attention modules and a decoder comprising four upsampling blocks as an example, the first downsampling block is cascaded with the first upsampling block via the first attention module; the second downsampling block is cascaded with the second upsampling block via the second attention module; the third downsampling block is cascaded with the third upsampling block via the third attention module; and the fourth downsampling block is cascaded with the fourth upsampling block via the fourth attention module.

[0086] In one possible implementation, the attention module can be understood as a parameterized discriminative network that uses an attention mechanism to generate attention weights based on the encoded feature map. This includes: performing a global analysis of the input downsampled feature map, quantifying the correlation between each element in the downsampled feature map and the first audio, and outputting an attention weight tensor that matches the dimension of the downsampled feature map. Each value in the attention weight tensor represents the relative importance of the corresponding audio feature element in the downsampled feature map in the subsequent audio data reconstruction process.

[0087] The values ​​in the attention weight tensor range from 0 to 1. The closer the weight value is to 1, the stronger its association with the first audio element; the closer the weight value is to 0, the weaker its association with the first audio element. Each element in the downsampled feature map includes the spatial location and / or channel dimension of that audio feature element.

[0088] In step S208, a weighted feature map is obtained by weighting the attention weights and the encoded feature map. The weighted feature map includes enhanced first audio features and suppressed second audio features.

[0089] In this embodiment, the attention weights and the encoded feature map are multiplied by a scalar multiplication operation on each corresponding audio feature element to obtain a weighted feature map.

[0090] In the encoded feature map, audio feature elements with large attention weights maintain or enhance their numerical strength in the weighted feature map. Conversely, audio feature elements with small attention weights have their numerical strength significantly reduced in the weighted feature map.

[0091] The weighted feature map is a feature representation that has undergone non-linear selection and is highly biased towards the first audio frequency. In the weighted feature map, features used to reconstruct the first audio frequency (target sound source) are enhanced and highlighted, while features used to reconstruct the second audio frequency (interference sound source) are suppressed.

[0092] In this embodiment, for the downsampled feature map output by each downsampled block, the attention mechanism processes the downsampled feature map to obtain the attention weight corresponding to the downsampled map, and then weights the attention weight with the downsampled feature map to obtain a weighted feature map.

[0093] In step S210, the weighted feature map is decoded using a decoder to obtain the separated first and second audio.

[0094] In the audio separation model, the decoder and encoder have a symmetrical structure, consisting of at least one cascaded upsampling block. Each upsampling block recovers the spatial resolution of the feature map through an upsampling operation and is fused with weighted feature maps from the corresponding level through skip connections, ultimately reconstructing the predicted temporal spectrogram of the audio to be processed. The value at each time-frequency point in the predicted temporal spectrogram represents the energy intensity of the first audio signal estimated by the decoder at that location.

[0095] The predicted time-spectrum graph is used as the numerator, and the original time-spectrum graph of the audio to be processed is used as the denominator. Element-by-element division is performed. To ensure the stability of the numerical calculation, a very small positive constant is added to the denominator to prevent division by zero.

[0096] The output of the above division operation is the final time-frequency mask. This time-frequency mask is a two-dimensional matrix with the same spectral dimension as the original time-frequency spectrum of the audio to be processed, where each element has a value between 0 and 1. The value of each element represents the estimated proportion of the first audio energy to the total energy of the mixed signal at the corresponding specific time and frequency point. When the value of an element approaches 1, it indicates that the time-frequency point contains almost entirely the energy of the first audio. When the value of an element approaches 0, it indicates that the time-frequency point contains almost no energy of the first audio.

[0097] The generated time-frequency mask is used as a separation function and applied to the original time-spectrum of the audio to be processed. Through element-wise multiplication, the energy components identified as the first audio signal in the original time-spectrum are preserved, while the energy components that are not the first audio signal are suppressed, thus obtaining the spectrum of the separated first audio signal. The spectrum of the first audio signal is then subjected to an inverse transform to reconstruct the separated first audio signal.

[0098] In one possible embodiment, the encoder includes an input block and at least one cascaded downsampling block, the attention mechanism includes multiple attention modules, and the decoder includes an output block and at least one cascaded upsampling block. The input block receives the raw time-spectrum of the audio to be processed. The input block transmits the raw time-spectrum to the first downsampling block. The first downsampling block performs a downsampling operation on the received raw time-spectrum, outputting a first downsampling feature map, which is transmitted in parallel in two directions: to the second downsampling block as its input for further downsampling and feature extraction; and to the first attention module, which is connected between the first downsampling block and the first upsampling block. The second downsampling block downsamples the first downsampling feature map, outputting a second downsampling feature map, and similarly transmits it to subsequent downsampling blocks and corresponding attention modules. This process is repeated until encoding is complete.

[0099] For each attention module, it receives a downsampled feature map from its corresponding downsampled block, processes the received downsampled feature map to generate attention weights, uses the attention weights to weight the downsampled feature map to obtain a weighted feature map corresponding to the downsampled feature map, and transmits the weighted feature map to the corresponding upsampled block.

[0100] The Nth (last) upsampling block receives the Nth weighted feature map transmitted from its corresponding Nth attention module, and the Nth downsampled feature map output from the Nth (last) downsampling block. The Nth upsampling block fuses the Nth downsampled feature map with the Nth weighted feature map, and then performs an upsampling operation to restore the spatial resolution of the feature maps, obtaining the Nth upsampled feature map. The Nth upsampling block outputs the Nth upsampled feature map and transmits it to the next upsampling block (i.e., the (N-1)th upsampling block). Each upsampling block then receives the weighted feature map transmitted from its corresponding attention module, and the upsampled feature map output from the previous upsampling block. The upsampling block fuses the upsampled feature map with the weighted feature map, and then performs an upsampling operation to obtain an upsampled feature map. The upsampling block outputs an upsampled feature map, which is then transmitted to the next upsampling block. This process is repeated until the output block performs final processing on the upsampled feature map output by the first upsampled block to generate the prediction-time spectrogram. Here, N is the number of downsampled blocks in the encoder.

[0101] In this scheme, the first audio signal is enhanced and the second signal is suppressed through an attention mechanism, thereby achieving more accurate audio separation and improving the audio quality of the first audio signal.

[0102] In one possible implementation, an attention mechanism is inserted between each downsampling block in the encoder and each upsampling block in the decoder. The encoder performs feature extraction to obtain an encoded feature map, which is then adjusted via the attention mechanism to enhance its focus in both the time-frequency domain and the feature channels. Finally, a skip connection is made to the corresponding decoder. This effectively highlights the first audio feature and suppresses the second audio feature during feature extraction and reconstruction, thus emphasizing the human voice signal and suppressing various complex background noises, achieving noise reduction of the original audio.

[0103] Based on the above embodiments, the audio processing method disclosed herein has been optimized, such as... Figure 4 As shown, the optimized audio processing method provided in this embodiment includes steps S402-S414.

[0104] In step S402, the audio to be processed is acquired.

[0105] In step S404, the audio to be processed is encoded using an encoder to obtain an encoded feature map, which includes a first audio feature and a second audio feature.

[0106] Steps S402-S404 in this embodiment are implemented in the same way as steps S202-S204 in the above example. For details, please refer to the description in the above embodiment. They will not be repeated in this embodiment.

[0107] In step S406, channel attention weights are generated based on the encoded feature map through global pooling and a multilayer perceptron; the channel attention weights include the weights corresponding to each channel.

[0108] As discussed in the above embodiments, an attention module is added in the skip cascading of the encoder's downsampling block and its corresponding decoder's upsampling block. This attention module includes a channel attention module and / or a spatial attention module. The channel attention weights are represented in the form of a channel attention weight vector.

[0109] In this embodiment, the channel attention module performs pooling operations on the received downsampled feature map to obtain a global representation of each channel in the downsampled feature map, which is an initial channel-level descriptor. The pooling operations include global average pooling and / or global max pooling.

[0110] Channel-level descriptors are input into a multilayer perceptron for nonlinear transformation. This multilayer perceptron contains at least one bottleneck layer structure, and nonlinear dependencies between channels are captured through dimensionality reduction and dimensionality expansion operations. The final output layer of the multilayer perceptron generates a normalized channel attention weight vector through a sigmoid activation function, where each element in the channel attention weight vector takes a value between 0 and 1.

[0111] The channel attention weight vector corresponds one-to-one with the channel dimension of the corresponding downsampled feature map. Each element in the channel attention weight vector represents the degree of contribution of the corresponding channel to the first audio. The closer an element in the channel attention weight vector is to 1, the more highly correlated the channel feature is with the first audio, and it should be enhanced; the closer an element in the attention weight vector is to 0, the more highly correlated the channel feature is with the second audio, and it should be suppressed.

[0112] In some possible embodiments, channel attention weights are generated based on the encoded feature map through global pooling and a multilayer perceptron, including: performing global average pooling and global max pooling operations on the encoded feature map respectively to obtain two channel-level description vectors; processing the two channel-level description vectors respectively using a multilayer perceptron to obtain two high-level semantic vectors; adding the elements at corresponding positions in the two high-level semantic vectors and generating channel attention weights through an activation function.

[0113] In this embodiment, global statistical information of the corresponding downsampled feature map is extracted through parallel dual-channel pooling operations. Specifically, global average pooling is performed on each channel of the corresponding downsampled feature map to calculate the arithmetic mean of all spatial locations of that channel, resulting in a first channel-level description vector, which represents the overall average activation level of each channel. Global max pooling is then performed on each channel of the corresponding downsampled feature map to extract the maximum activation value of all spatial locations of that channel, resulting in a second channel-level description vector, which represents the most salient feature response of each channel.

[0114] Two channel-level description vectors (the first channel-level description vector and the second channel-level description vector) are input into a multilayer perceptron for nonlinear transformation. The multilayer perceptron includes at least two fully connected layers. The first fully connected layer reduces the dimensionality of the channel-level description vectors to 1 / r of the original number of channels (r is the compression ratio, usually 16), and the second fully connected layer restores the original number of channels, which is the high-level semantic vector.

[0115] In one possible implementation, a multilayer perceptron can be replaced by a single-layer perceptron to balance performance and efficiency. Alternatively, a multilayer perceptron can also employ a three- or more-layer MLP structure to enhance nonlinear transformation capabilities.

[0116] The elements at the corresponding channel positions of the two high-level semantic vectors are added together, and the dual information of average statistics and maximum statistics is fused. The Sigmoid activation function is applied to the fused vector to normalize the channel weight values ​​to the (0,1) interval, generating the final channel attention weight vector.

[0117] In this embodiment, channel attention weights are generated through a dual-channel pooling and fusion mechanism to improve the robustness of the feature channels. In subsequent weighted processing, the first audio feature is enhanced more accurately while the second audio feature is suppressed, ultimately achieving the noise reduction effect.

[0118] In step S408, a first weighted feature map is obtained by calculating based on the feature responses and weights of each channel in the encoded feature map; the first weighted feature map includes a first audio feature for channel enhancement and a second audio feature for channel suppression.

[0119] The attention weight value of each channel is multiplied by the feature map of the corresponding channel by a scalar multiplication. While keeping the spatial structure and time series of the feature map unchanged, the feature response of each channel is scaled as a whole.

[0120] The closer the attention weight value of the channel highly correlated with the first audio source is to 1, the stronger the feature response of these channels is after weighting. The closer the attention weight value of the channel highly correlated with the second audio source is to 0, the weaker the feature response of these channels is after weighting.

[0121] In deep neural networks, each channel corresponds to a feature response in the original temporal spectrogram of the audio to be processed, including energy distribution, spectral texture, formant patterns, and speech start boundaries. After convolution, the expressive power of the features in each channel is not balanced; some channels may be closer to human voice features, while others are mainly background noise or invalid information.

[0122] In this embodiment, a channel attention mechanism is incorporated, which allows the network to adaptively model the response intensity of each channel. By setting different weights for different channels, key features are highlighted and redundant features are suppressed, thereby improving the modeling ability of human voice.

[0123] The addition of channel attention mechanism has the following beneficial effects:

[0124] 1) Adaptive enhancement of human voice feature channels: CAM extracts global statistical information of each channel in the overall speech frame through global average pooling and global max pooling, and then generates channel weights through a fully connected network to dynamically enhance feature channels that are strongly correlated with human voice.

[0125] 2) Suppressing noise-related invalid channels: For some feature channels dominated by background noise, CAM will automatically assign lower weights to suppress their influence, which helps the network concentrate its resources on learning feature dimensions related to human voice.

[0126] 3) Enhance deep semantic modeling capabilities: Through the channel attention mechanism, the network forms more semantic and stable human voice feature vectors at the deep stage of feature expression, which helps the subsequent separation network to accurately locate and recover human voice signals.

[0127] In step S410, the spatial attention mechanism is used to identify the input feature map and generate a spatial attention map. The weights in the spatial attention map are used to characterize the correlation between the corresponding time-frequency point and the input feature map. The input feature map includes an encoded feature map or a first weighted feature map.

[0128] In one possible implementation, the input feature map can be an encoded feature map. In other words, the input feature map can be a downsampled feature map output by the downsampled block corresponding to the attention module. That is, the attention mechanism does not include a channel attention module, but only a spatial attention module.

[0129] In one possible implementation, the input feature map can be a first weighted feature map, and the attention mechanism includes a channel attention module and a spatial attention module. The channel attention module processes the downsampled feature map output by the downsampled block to obtain the first weighted feature map, and then transmits the first weighted feature map to the spatial attention module.

[0130] In this embodiment, the spatial attention module evaluates the contribution of each time-frequency point to the first audio source based on the feature response patterns at various spatial locations in the input feature map, generating a spatial attention map. The spatial attention map is a two-dimensional weight matrix, where each element represents the spatial correlation strength between the corresponding time-frequency point and the first audio source. Larger values ​​in the spatial attention map indicate a stronger correlation between the corresponding time-frequency point and the first audio source; these time-frequency regions typically contain key acoustic features of the first audio source. Smaller values ​​in the spatial attention map indicate that the corresponding time-frequency point is related to interfering sound sources or contains irrelevant acoustic information.

[0131] Spatial attention mechanisms can adaptively adjust the region of interest based on the time-frequency characteristics of the audio being processed. In the time domain, they focus on the period during which the first audio source is active.

[0132] In one possible implementation, the spatial attention mechanism is used to identify the input feature map and generate the spatial attention map by: using the spatial attention mechanism to identify the response intensity of each time frequency point in the input feature map and the spatial context relationship between each time frequency point; and generating a spatial attention map corresponding to the input feature map based on the response intensity of each time frequency point and the spatial context relationship between each time frequency point.

[0133] The spatial attention module detects the feature activation levels at various locations in the input feature map, identifying time-frequency regions with significant responses. Time-frequency points with high response intensity typically contain important acoustic information, such as the harmonic structure or salient features of the first audio source. The spatial context relationships between time-frequency points are analyzed, i.e., the correlations between adjacent time-frequency points are analyzed, identifying the distribution patterns and structural features of features in the time-frequency domain.

[0134] Based on a comprehensive analysis of the response intensity at time and frequency points and the spatial context, a spatial attention map corresponding to the size of the input feature map is generated. Each weight value in this attention map reflects the relative importance of the corresponding time and frequency point in the current audio separation task. The weight value is determined by the strength of the feature of the time and frequency point itself and its contextual position in the surrounding time and frequency environment.

[0135] The generated spatial attention map has clear spatiotemporal characteristics. In the time dimension, the attention weights can track the temporal changes of sound source activity; in the frequency dimension, they can highlight the characteristic frequency bands related to the target sound source.

[0136] In the process of generating spatial attention maps, spatial context information at different scales is integrated. Local context ensures accurate capture of detailed features, global context maintains the rationality of the overall structure, and the fusion of multi-scale information makes attention allocation more accurate and robust.

[0137] In step S412, the spatial attention map and the input feature map are weighted and fused to obtain a second weighted feature map, which includes a first audio feature with spatial enhancement and a second audio feature with spatial suppression.

[0138] In this embodiment, when fusing the spatial attention map with the input feature map, a corresponding weighting coefficient is applied to each time frequency point to obtain the second weighted feature map.

[0139] Spatial attention weighting enhances the time-frequency region associated with the first audio source, while effectively suppressing the time-frequency region associated with the second audio source. The time-frequency points occupied by interfering sound sources are assigned lower weights, weakening the intensity of their acoustic features in the output feature map and reducing the negative impact of interfering sound sources on the target sound source.

[0140] In this embodiment, different time and frequency positions in the original time-frequency spectrum of the audio to be processed carry different acoustic information. Human voices are usually concentrated in a specific frequency band (e.g., 300 Hz to 3400 Hz) and have obvious periodicity and formant structure; while environmental noise often has wider bandwidth, irregularity, and instantaneous changes. Therefore, the "critical region" in the time-frequency domain is the core of achieving the separation of human voices and noise.

[0141] The Spatial Attention Mechanism (SAM) automatically generates a spatial attention map by learning the response intensity of the input feature map at various time-frequency points. This map is used to highlight key regions related to human voices while suppressing irrelevant or noise-dominated regions.

[0142] The advantages of incorporating spatial attention mechanisms mainly include:

[0143] 1) It can automatically locate regions where human voice features are concentrated. SAM will give higher attention weight to positions in the speech spectrum that are strongly correlated with human voice, such as the fundamental frequency region and formant region.

[0144] 2) Reduce the impact of background noise interference. For frequency bands with no speech activity or areas dominated by background noise, SAM automatically assigns a lower response to avoid noise interference being mistakenly amplified.

[0145] 3) Enhanced contextual understanding: SAM's dynamic modeling of spatial dimensions helps to understand the patterns of speech such as pronunciation coherence and speech rate changes, thereby improving the network's recognition and separation capabilities.

[0146] Therefore, the spatial attention mechanism enables the model to no longer process the entire time-frequency image equally, but to focus on the valuable parts of the speech, thereby effectively improving the model's accuracy and robustness in separating human voice from complex background noise.

[0147] In step S414, the weighted feature map is decoded using a decoder to obtain the separated first and second audio.

[0148] Steps S402-S404 in this embodiment are implemented in the same way as steps S202-S204 in the above example. For details, please refer to the description in the above embodiment. They will not be repeated in this embodiment.

[0149] Based on the above embodiments, this disclosure optimizes the processing flow of "using a decoder to decode the weighted feature map to obtain the separated first and second audio" in the audio processing method, such as... Figure 5 As shown, the optimized audio processing method includes steps S502-S516.

[0150] In step S502, the audio to be processed is obtained.

[0151] In step S504, the audio to be processed is encoded using an encoder to obtain an encoded feature map, which includes a first audio feature and a second audio feature.

[0152] In step S506, attention weights are generated based on the encoded feature map using an attention mechanism.

[0153] In step S508, a weighted feature map is obtained by weighting the attention weights and the encoded feature map. The weighted feature map includes enhanced first audio features and suppressed second audio features.

[0154] Steps S502-S504 in this embodiment are implemented in the same way as steps S202-S208 in the above example. For details, please refer to the description in the above embodiment. They will not be repeated in this embodiment.

[0155] In step S510, the weighted feature map is decoded using a decoder to obtain the first amplitude spectrum and the second amplitude spectrum.

[0156] In this embodiment, the decoder is used to upsample and fuse the weighted feature map and the encoded feature map to gradually restore the time-frequency resolution. Finally, two independent output channels are used to generate the first amplitude spectrum and the second amplitude spectrum, which represent the audio separation model's estimate of the clean spectrum of the two sound sources and have the same time-frequency dimension as the original input.

[0157] In step S512, the first audio energy and the second audio energy are calculated based on the first amplitude spectrum and the second amplitude spectrum.

[0158] The two generated amplitude spectra are squared to convert them into corresponding audio energy spectra. The first amplitude spectrum is squared to obtain the first audio energy spectrum, and the second amplitude spectrum is squared to obtain the second audio energy spectrum. The energy spectrum reflects the energy distribution of each sound source at different time and frequency points.

[0159] In step S514, a first time-frequency mask and a second time-frequency mask are calculated based on the first audio energy and the second audio energy.

[0160] Based on the two audio energy spectra, the energy proportion at each time frequency point is calculated.

[0161] Dividing the first audio energy by the sum of the first and second audio energies yields the normalized first time-frequency mask. Dividing the second audio energy by the sum of the first and second audio energies yields the normalized second time-frequency mask.

[0162] In step S516, the audio to be processed is multiplied by the first time-frequency mask and the second time-frequency mask respectively to obtain the separated first audio and second audio.

[0163] The generated time-frequency mask is applied to the original time-frequency spectrum of the audio to be processed. Specifically, the first time-frequency mask is multiplied by the original time-frequency spectrum of the audio to be processed to extract the time-frequency spectrum of the first audio; the second time-frequency mask is multiplied by the original time-frequency spectrum of the audio to be processed to extract the time-frequency spectrum of the second audio. For example, the audio processed by the audio separation model is divided into human voice audio and noise audio. The human voice audio is the noise-reduced speech signal, which is used for subsequent speech recognition.

[0164] The first audio signal after separation is reconstructed by performing an inverse transform on its time spectrum and combining it with the original phase information.

[0165] In one possible implementation, such as Figure 6 As shown, the audio separation model includes a human voice model 610 and a noise model 620. Both the human voice model 610 and the noise model 620 adopt a structure that adds spatial attention mechanism and channel attention mechanism to the Spleeter human voice separation technology based on U-Net network. The specific architecture can be referred to the description in the above example.

[0166] like Figure 6As shown, the input is the audio to be processed 630, which is a fusion of human voice and background noise. After the audio to be processed is processed by calculating the spectrum in step S602 and the amplitude spectrum in step S604, it is input into the human voice model 610 and the noise model 620 respectively. The noise amplitude spectrum and the human voice amplitude spectrum 640 are predicted by the human voice model 610 and the noise model 620 respectively. The noise amplitude spectrum and the human voice amplitude spectrum are squared respectively to obtain the human voice energy. and noise energy Then, the proportion of human voice in each frequency band of the original audio at each moment is calculated using the following formula (1). That is, the human voice mask is 650. Then, the proportion of noise in each frequency band of the original audio at each moment is calculated using the following formula (2). That is, noise mask660.

[0167] (1);

[0168] (2).

[0169] Finally, the spectrum of the input audio to be processed is multiplied by... and The human voice spectrum 670 and noise spectrum 680 are obtained, and the human voice audio 690 and noise audio 6100 are obtained by using inverse STFT.

[0170] In a specific instance, such as Figure 7 As shown, the actual audio of the scene was recorded using a recording device. The audio file is a human voice recording file containing background noise. The file format is usually .mp3, the sampling rate is 44.1kHz, and the audio channel is stereo.

[0171] The original audio is processed by the preprocessing module 710 as follows: the sampling rate of the original audio is converted to 16kHz; the number of channels is extracted and converted to mono; the audio format is saved as .wav lossless format; and the bit depth is unified to 16bit.

[0172] The preprocessed audio is input into the audio separation model 720 provided in this example. After processing by the model, two spectrograms are output: the human voice spectrum and the noise spectrum. The inverse Fourier transform is used to convert the human voice spectrum into a time-domain waveform, generating a clean human voice audio file.

[0173] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution comply with the relevant provisions of laws and regulations. All types of data, such as personal identity data, operational data, and behavioral data related to individuals, customers, and groups, obtained in this disclosed embodiment have been agreed upon by the users.

[0174] Based on the same inventive concept, this disclosure also provides an audio processing apparatus, as shown in the following embodiments. Since the principle by which this audio processing apparatus solves the problem is similar to that of the above-described method embodiments, the implementation of this audio processing apparatus embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0175] Figure 8 This illustration shows a schematic diagram of a network-side device according to an embodiment of the present disclosure, such as... Figure 8 As shown, the audio processing device includes: an acquisition module 810, an encoding module 820, a weight generation module 830, a weighted processing module 840, and a decoding module 850.

[0176] The system comprises the following modules: an acquisition module for acquiring the audio to be processed; an encoding module for encoding the audio using an encoder to obtain an encoded feature map, which includes a first audio feature and a second audio feature; a weight generation module for generating attention weights based on the encoded feature map using an attention mechanism; a weighted processing module for performing weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map, which includes an enhanced first audio feature and a suppressed second audio feature; and a decoding module for decoding the weighted feature map using a decoder to obtain the separated first and second audio.

[0177] In some embodiments, the encoder includes at least one cascaded downsampling block, the attention mechanism includes at least one attention module, and the decoder includes at least one cascaded upsampling block, with each downsampling block corresponding to an upsampling block; the downsampling block is connected to its corresponding upsampling block via an attention module.

[0178] In some embodiments, the number of attention modules is less than or equal to the amount of data in the downsampling block, and each attention module includes a channel attention module and / or a spatial attention module.

[0179] In some embodiments, the attention weights include channel attention weights, and the weighted feature map includes a first weighted feature map; the weight generation module 830 is specifically used to generate channel attention weights based on the encoded feature map through global pooling and a multilayer perceptron; the channel attention weights include the weights corresponding to each channel; the weighting processing module 840 is specifically used to calculate based on the feature responses corresponding to each channel and the weights corresponding to each channel in the encoded feature map to obtain the first weighted feature map, the first weighted feature map including a first audio feature for channel enhancement and a second audio feature for channel suppression.

[0180] In some embodiments, the weight generation module 830 is specifically used to perform global average pooling and global max pooling operations on the encoded feature map respectively to obtain two channel-level description vectors; process the two channel-level description vectors respectively using a multilayer perceptron to obtain two high-level semantic vectors; add the elements at corresponding positions in the two high-level semantic vectors, and generate channel attention weights through an activation function.

[0181] In some embodiments, the attention mechanism includes a spatial attention mechanism, the attention weights include spatial attention weights, and the weighted feature map includes a second weighted feature map; the weight generation module 830 is specifically used to identify the input feature map using the spatial attention mechanism to generate a spatial attention map, wherein the weights in the spatial attention map are used to characterize the correlation between the corresponding time-frequency point and the input feature map, and the input feature map includes an encoded feature map or a first weighted feature map; the weighting processing module 840 is specifically used to perform weighted fusion of the spatial attention map and the input feature map to obtain a second weighted feature map, wherein the second weighted feature map includes a spatially enhanced first audio feature and a spatially suppressed second audio feature.

[0182] In some embodiments, the weight generation module 830 is specifically used to identify the response intensity of each time-frequency point in the input feature map and the spatial context relationship between each time-frequency point using a spatial attention mechanism; and to generate a spatial attention map corresponding to the input feature map based on the response intensity of each time-frequency point and the spatial context relationship between each time-frequency point.

[0183] In some embodiments, the decoding module is specifically used to decode the weighted feature map using a decoder to obtain a first amplitude spectrum and a second amplitude spectrum; calculate a first audio energy and a second audio energy based on the first amplitude spectrum and the second amplitude spectrum; calculate a first time-frequency mask and a second time-frequency mask based on the first audio energy and the second audio energy; and multiply the audio to be processed by the first time-frequency mask and the second time-frequency mask respectively to obtain the separated first audio and second audio.

[0184] It should be noted that the examples and application scenarios implemented by the modules in the above device embodiments and the corresponding steps in the method embodiments are the same, but are not limited to the content disclosed in the above method embodiments. It should also be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0185] Those skilled in the art will understand that various aspects of this disclosure can be implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which can be collectively referred to herein as a "circuit", "module" or "system".

[0186] Based on the same inventive concept, this disclosure also provides an electronic device, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the audio processing method described above by executing the executable instructions. Since the principle by which this electronic device solves the problem is similar to that of the above method embodiments, the implementation of this electronic device embodiment can refer to the implementation of the above method embodiments, and repeated details will not be described again.

[0187] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0188] like Figure 9 As shown, the electronic device 900 is manifested in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including storage unit 920 and processing unit 910).

[0189] The storage unit stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0190] In some embodiments, the processing unit 910 may perform the following steps of the above method embodiments: acquiring the audio to be processed; encoding the audio to be processed using an encoder to obtain an encoded feature map, the encoded feature map including a first audio feature and a second audio feature; generating attention weights based on the encoded feature map using an attention mechanism; performing weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map, the weighted feature map including an enhanced first audio feature and a suppressed second audio feature; and performing decoding processing on the weighted feature map using a decoder to obtain the separated first audio and second audio.

[0191] Storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 9201 and / or a cache 9202, and may further include a read-only memory unit (ROM) 9203.

[0192] The storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0193] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0194] Electronic device 900 can also communicate with one or more external devices 940 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0195] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0196] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the audio processing method described above. Since the principle by which this computer-readable storage medium solves the problem is similar to that of the above method embodiments, the implementation of this computer-readable storage medium embodiment can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0197] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0198] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0199] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0200] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0201] Based on the same inventive concept, this disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the audio processing method of any one of the above method embodiments. Since the principle by which this computer program product embodiment solves the problem is similar to that of the above method embodiments, the implementation of this computer program product embodiment can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0202] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0203] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0204] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0205] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. An audio processing method, characterized in that, include: Obtain the audio to be processed; The audio to be processed is encoded using an encoder to obtain an encoded feature map. The encoded feature map includes multiple downsampled feature maps of different sizes, and each downsampled feature map includes a first audio feature and a second audio feature. Attention weights are generated based on the encoded feature map using an attention mechanism; The attention weights and the encoded feature map are weighted to obtain a weighted feature map, which includes an enhanced first audio feature and a suppressed second audio feature. The weighted feature map described by the decoder is used for decoding to obtain the separated first and second audio; The encoder includes at least one cascaded downsampling block, the attention mechanism includes at least one attention module, and the decoder includes at least one cascaded upsampling block. The downsampling block corresponds one-to-one with the upsampling block. The downsampling block is connected to its corresponding upsampling block via the attention module. The attention module is used to generate attention weights corresponding to the downsampled feature map based on the downsampled feature map output by the corresponding downsampled block. The attention weights characterize the degree of correlation between each element in the downsampled feature map and the first audio.

2. The audio processing method according to claim 1, characterized in that, The number of attention modules is less than or equal to the data volume of the downsampling block, and each attention module includes a channel attention module and / or a spatial attention module.

3. The audio processing method according to claim 2, characterized in that, The attention weights include channel attention weights, and the weighted feature map includes a first weighted feature map; The generation of attention weights based on the encoded feature map using an attention mechanism includes: Based on the encoded feature map, channel attention weights are generated through global pooling and a multilayer perceptron; the channel attention weights include the weights corresponding to each channel. The weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map includes: Based on the feature responses corresponding to each channel in the encoded feature map and the weights corresponding to each channel, a first weighted feature map is obtained. The first weighted feature map includes a first audio feature for channel enhancement and a second audio feature for channel suppression.

4. The audio processing method according to claim 3, characterized in that, The process of generating channel attention weights based on the encoded feature map through global pooling and a multilayer perceptron includes: Global average pooling and global max pooling operations are performed on the encoded feature map respectively to obtain two channel-level description vectors; The two channel-level description vectors are processed using a multilayer perceptron to obtain two high-level semantic vectors. The elements at corresponding positions in the two high-level semantic vectors are added together, and channel attention weights are generated by an activation function.

5. The audio processing method according to claim 2 or 3, characterized in that, The attention weights include spatial attention weights, and the weighted feature map includes a second weighted feature map; The generation of attention weights based on the encoded feature map using an attention mechanism includes: The spatial attention mechanism is used to identify the input feature map and generate a spatial attention map. The weights in the spatial attention map are used to characterize the correlation between the corresponding time frequency point and the input feature map. The input feature map includes the encoded feature map or the first weighted feature map. The weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map includes: The spatial attention map and the input feature map are weighted and fused to obtain a second weighted feature map, which includes a first audio feature with spatial enhancement and a second audio feature with spatial suppression.

6. The audio processing method according to claim 5, characterized in that, The step of using spatial attention mechanism to identify the input feature map and generate a spatial attention map includes: The spatial attention mechanism is used to identify the response intensity of each time-frequency point in the input feature map, as well as the spatial context relationship between each time-frequency point; Based on the response intensity of each time-frequency point and the spatial context relationship between each time-frequency point, a spatial attention map corresponding to the input feature map is generated.

7. The audio processing method according to claim 1, characterized in that, The weighted feature map is decoded using the decoder to obtain the separated first and second audio, including: The weighted feature map is decoded using the decoder to obtain a first amplitude spectrum and a second amplitude spectrum; Calculate the first audio energy and the second audio energy based on the first amplitude spectrum and the second amplitude spectrum; Calculate the first time-frequency mask and the second time-frequency mask based on the first audio energy and the second audio energy; The audio to be processed is multiplied by the first time-frequency mask and the second time-frequency mask respectively to obtain the separated first audio and second audio.

8. An audio processing apparatus, characterized in that, include: The acquisition module is used to acquire the audio to be processed; The encoding module is used to encode the audio to be processed using an encoder to obtain an encoded feature map. The encoded feature map includes multiple downsampled feature maps of different sizes, and each downsampled feature map includes a first audio feature and a second audio feature. The weight generation module is used to generate attention weights based on the encoded feature map using an attention mechanism; The weighted processing module is used to perform weighted processing based on the attention weights and the encoded feature map to obtain a weighted feature map, wherein the weighted feature map includes an enhanced first audio feature and a suppressed second audio feature; The decoding module is used to perform decoding processing using the weighted feature map of the decoder to obtain the separated first audio and second audio. The encoder includes at least one cascaded downsampling block, the attention mechanism includes at least one attention module, and the decoder includes at least one cascaded upsampling block. The downsampling block corresponds one-to-one with the upsampling block. The downsampling block is connected to its corresponding upsampling block via the attention module. The attention module is used to generate attention weights corresponding to the downsampled feature map based on the downsampled feature map output by the corresponding downsampled block. The attention weights characterize the degree of correlation between each element in the downsampled feature map and the first audio.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the audio processing method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the audio processing method according to any one of claims 1 to 7.

11. A computer program product, comprising: A computer program or instruction, characterized in that, when executed by a processor, the computer program or instruction implements the audio processing method according to any one of claims 1 to 7.

Images