An audio processing method, apparatus, storage medium, and electronic device

Abstract

The present disclosure relates to the technical field of audio processing, and provides an audio processing method and device, a storage medium and an electronic device. The method comprises: obtaining an input audio signal; processing the audio signal based on a short-time Fourier transform to obtain at least one complex spectrum feature; in the case that N complex spectrum features have been buffered, generating a complex spectrum feature block based on the N complex spectrum features; encoding the complex spectrum feature block based on a first multi-scale convolution layer to obtain an encoded feature corresponding to the complex spectrum feature block; decoding the encoded feature based on a second multi-scale convolution layer to obtain a feature mask value; performing masking calculation on the N complex spectrum features using the feature mask value to obtain separated complex spectrum features corresponding to each sound source component respectively; and processing the separated complex spectrum features based on an inverse short-time Fourier transform to obtain a multi-track separated audio signal. In this way, the audio track separation effect of the electronic device is improved, and high-fidelity audio track separation is achieved.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, and in particular to an audio processing method, apparatus, storage medium and electronic device. Background Technology

[0002] In automotive settings, real-time audio processing algorithms used to tune audio systems typically operate on already mixed audio. If the audio contains multiple sound source components (such as vocals, drums, bass, etc.), current real-time audio processing algorithms have poor track separation capabilities and cannot perform audio tuning and 3D spatial rendering at the track level, leading to issues such as distorted timbre and abnormal spatial perception. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides an audio processing method, apparatus, storage medium, and electronic device, which improves the audio track separation effect of the electronic device and achieves high-fidelity audio track separation.

[0004] The first aspect of this disclosure provides an audio processing method, comprising: The process involves: acquiring the input audio signal; processing the audio signal using short-time Fourier transform to obtain at least one complex spectral feature; generating a complex spectral feature block based on N cached complex spectral features; encoding the complex spectral feature block using a first multi-scale convolutional layer to obtain the corresponding encoded features; decoding the encoded features using a second multi-scale convolutional layer to obtain feature mask values; performing masking calculations on the N complex spectral features using the feature mask values ​​to obtain the separated complex spectral features corresponding to each sound source component; and processing the separated complex spectral features using inverse short-time Fourier transform to obtain a multi-track separated audio signal.

[0005] A second aspect of this disclosure provides an audio processing apparatus, comprising: The system comprises the following modules: an acquisition module for acquiring the input audio signal; a first processing module for processing the audio signal based on short-time Fourier transform to obtain at least one complex spectral feature; a generation module for generating a complex spectral feature block based on N cached complex spectral features; an encoding module for encoding the complex spectral feature block based on a first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectral feature block; a decoding module for decoding the encoded features based on a second multi-scale convolutional layer to obtain a feature mask value; a calculation module for performing masking calculations on the N complex spectral features using the feature mask value to obtain the separated complex spectral features corresponding to each sound source component; and a second processing module for processing the separated complex spectral features based on inverse short-time Fourier transform to obtain a multi-track separated audio signal.

[0006] A third aspect of this disclosure provides a computer program product that, when executed by an instruction processor, performs the audio processing method provided in the first aspect of this disclosure.

[0007] A fourth aspect of this disclosure provides an electronic device, the electronic device comprising: Processor; memory for storing processor-executable instructions; processor for reading executable instructions from memory and executing the instructions to implement the audio processing method provided in the first aspect of this disclosure.

[0008] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing the audio processing method provided in the first aspect of this disclosure.

[0009] The audio processing method, apparatus, storage medium, and electronic device provided in the above embodiments of this disclosure acquire an input audio signal; process the audio signal based on short-time Fourier transform to obtain at least one complex spectrum feature; generate a complex spectrum feature block based on N complex spectrum features when N complex spectrum features are already cached; encode the complex spectrum feature block based on a first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectrum feature block; decode the encoded features based on a second multi-scale convolutional layer to obtain a feature mask value; perform masking calculation on the N complex spectrum features using the feature mask value to obtain the separated complex spectrum features corresponding to each sound source component; process the separated complex spectrum features based on inverse short-time Fourier transform to obtain a multi-track separated audio signal. This solves the problem of poor track separation effect of current real-time sound effect processing algorithms, which cannot debug and render audio at the track level, leading to problems such as timbre destruction and abnormal spatial sense. It improves the track separation effect of electronic devices and achieves high-fidelity track separation. Attached Figure Description

[0010] Figure 1 This is a diagram of an in-vehicle system architecture provided in an exemplary embodiment of the present disclosure; Figure 2 This is an exemplary embodiment of the audio processing system architecture provided in this disclosure; Figure 3 This is a schematic flowchart of an audio processing method provided in an exemplary embodiment of the present disclosure; Figure 4 This is a flowchart of step S200 provided in an exemplary embodiment of this disclosure; Figure 5 This is a flowchart of step S400 provided in an exemplary embodiment of this disclosure; Figure 6 This is a flowchart of step S440 provided in an exemplary embodiment of this disclosure; Figure 7 This is another schematic flowchart of an audio processing method provided in an exemplary embodiment of this disclosure; Figure 8 This is a flowchart of step S450 provided in an exemplary embodiment of this disclosure; Figure 9 This is a flowchart of step S500 provided in an exemplary embodiment of this disclosure; Figure 10 This is a flowchart of step S510 provided in an exemplary embodiment of this disclosure; Figure 11 This is another schematic flowchart of an audio processing method provided in an exemplary embodiment of the present disclosure; Figure 12 This is a flowchart of step S513 provided in an exemplary embodiment of this disclosure; Figure 13 This is a schematic diagram of the structure of an audio processing apparatus provided in an exemplary embodiment of the present disclosure; Figure 14 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0011] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0012] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0013] Application Overview With the development of smart cockpit technology, in-vehicle audio systems have become an important component for enhancing the driving experience. Current mainstream in-vehicle audio tuning solutions primarily rely on a series of basic real-time audio signal processing algorithms, including upmixing, parametric equalizers, gain and delay control, dynamic range compression, and reverberation. These algorithms post-process the mixed stereo or multi-channel audio signals to achieve functions such as channel allocation, sound image localization, and in-vehicle sound field correction.

[0014] However, the aforementioned real-time audio signal processing algorithms all operate on the mixed audio signal, lacking the ability to perceive and separate the original multi-track audio sources (such as independent tracks for vocals, drums, bass, strings, synthesizers, etc.). Because various sound sources in the input signal are highly intertwined in the frequency, time, and spatial domains, current real-time audio signal processing algorithms struggle to achieve precise control over specific sound source components. For example, it is difficult to enhance vocal clarity independently without altering the overall timbre of the accompaniment, effectively suppress excessively strong low-frequency drum beats in complex musical passages, or apply differentiated three-dimensional spatial rendering strategies for different instrument types. This can easily lead to problems such as timbre distortion and abnormal spatial perception in the audio.

[0015] Furthermore, due to limitations in the computing power and real-time requirements of in-vehicle platforms, current in-vehicle systems are generally unable to deploy complex source separation models or deep learning-based audio demixing models. This results in audio systems having a single tuning strategy and poor adaptability when faced with diverse music content, making it difficult to meet users' growing demand for immersive and personalized listening experiences.

[0016] In summary, real-time audio processing algorithms in in-vehicle scenarios are unable to adjust audio at the track level and render it in three-dimensional space, which can easily lead to problems such as timbre distortion and abnormal spatial sense.

[0017] To address the aforementioned technical problems, this disclosure provides an audio processing method, apparatus, storage medium, and electronic device. The method acquires an input audio signal, processes the audio signal based on a short-time Fourier transform, and obtains at least one complex spectral feature. Given N cached complex spectral features, a complex spectral feature block is generated based on the N features. The complex spectral feature block is encoded using a first multi-scale convolutional layer to obtain the corresponding encoded features. The encoded features are decoded using a second multi-scale convolutional layer to obtain a feature mask value. The feature mask value is used to perform masking calculations on the N complex spectral features to obtain the separated complex spectral features corresponding to each sound source component. The separated complex spectral features are processed using an inverse short-time Fourier transform to obtain a multi-track separated audio signal. This method ensures the stability of the audio signal's sound quality during track separation while avoiding track leakage, achieving high-fidelity track separation.

[0018] Exemplary System Figure 1 This is a diagram of an in-vehicle system architecture provided in an exemplary embodiment of this disclosure.

[0019] like Figure 1 As shown, the vehicle system 100 in this disclosure may include: a processor 110, a memory 120, and an audio device 130.

[0020] The processor 110 is used to execute program instructions to achieve coordinated control and data processing of various modules (devices) in the vehicle system 100. The processor 110 is electrically connected to the memory 120 and the audio device 130 respectively to perform instruction processing and data processing.

[0021] The memory 120 is used to store program instructions that can be executed by the processor 110. The processor 110 can load and execute the program instructions in the memory 120 to implement the audio processing method.

[0022] The memory 120 is also used to store data, such as data related to audio processing methods, for use by the processor 110. The memory 120 is also used to store the operating system, applications, and various other data resources, providing an operating environment and data support for other functions of the vehicle.

[0023] In some embodiments, the memory 120 may include volatile memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.; it may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, etc.

[0024] The audio device 130 is used to process, transmit, receive or play audio signals. It can independently perform specific audio-related functions and can be electrically connected to the processor 110 or the memory 120 through a standardized interface.

[0025] Processor 110 is configured to execute instructions stored in memory 120 to perform the following operations: The input audio signal is acquired. The audio signal is processed using a short-time Fourier transform to obtain at least one complex spectral feature. Given N cached complex spectral features, a complex spectral feature block is generated based on these N features. The complex spectral feature block is encoded using a first multi-scale convolutional layer to obtain the corresponding encoded features. The encoded features are decoded using a second multi-scale convolutional layer to obtain feature mask values. The feature mask values ​​are used to perform masking calculations on the N complex spectral features to obtain the separated complex spectral features corresponding to each sound source component. The separated complex spectral features are processed using an inverse short-time Fourier transform to obtain a multi-track separated audio signal.

[0026] The vehicle-mounted system disclosed in this embodiment executes executable instructions in the memory to acquire the input audio signal, processes the audio signal based on short-time Fourier transform, and obtains at least one complex spectrum feature. With N complex spectrum features already cached, a complex spectrum feature block is generated based on the N complex spectrum features. The complex spectrum feature block is encoded based on a first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectrum feature block. The encoded features are decoded based on a second multi-scale convolutional layer to obtain a feature mask value. The feature mask value is used to perform masking calculations on the N complex spectrum features to obtain the separated complex spectrum features corresponding to each sound source component. The separated complex spectrum features are processed based on inverse short-time Fourier transform to obtain multi-track separated audio signals. These operations ensure the stability of the audio signal quality during the track separation process, while avoiding the problem of track leakage, achieving high-fidelity track separation.

[0027] Figure 2 This is an exemplary embodiment of the audio processing system architecture diagram provided in this disclosure.

[0028] like Figure 2 As shown, the audio processing system 200 includes an audio transformation module 210, a complex spectrum feature block cache module 220, a multi-scale subband coding module 230, a first convolutional layer temporal feature cache module 240, a multi-scale subband mask estimator module 250, a second convolutional layer temporal feature cache module 260, a signal separation calculation module 270, and an audio inverse transformation module 280.

[0029] The following embodiments of this disclosure will provide a detailed description of the functions of each module in the audio processing system 200.

[0030] The audio processing system 200 disclosed in this embodiment acquires the input audio signal based on the interaction between various modules, processes the audio signal based on short-time Fourier transform, and obtains at least one complex spectrum feature. With N complex spectrum features already cached, a complex spectrum feature block is generated based on the N complex spectrum features. The complex spectrum feature block is encoded using a first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectrum feature block. The encoded features are decoded using a second multi-scale convolutional layer to obtain feature mask values. The feature mask values ​​are used to perform masking calculations on the N complex spectrum features to obtain the separated complex spectrum features corresponding to each sound source component. The separated complex spectrum features are processed using inverse short-time Fourier transform to obtain multi-track separated audio signals, etc. This ensures the stability of the audio signal quality during the track separation process, while avoiding the problem of track leakage, achieving high-fidelity track separation.

[0031] Exemplary methods Figure 3 This is a schematic flowchart of an audio processing method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 3 As shown, it includes the following steps: Step S100: Acquire the input audio signal.

[0032] In some embodiments, the electronic device acquires the input audio signal based on the audio conversion module 210.

[0033] By employing this approach, the electronic device provides the necessary input for implementing the audio processing method.

[0034] Step S200: Process the audio signal based on short-time Fourier transform to obtain at least one complex spectral feature.

[0035] In some embodiments, the electronic device performs a short-time Fourier transform on the audio signal based on the audio transformation module to obtain at least one complex spectral feature.

[0036] The short-time fourier transform (STFT) is a time-frequency analysis tool used to analyze non-stationary signals (i.e., signals whose frequency changes over time). It can segment the signal (also known as windowing), perform a Fourier transform on each segment, and thus obtain the time and frequency information corresponding to that segment of the signal simultaneously.

[0037] Complex spectral features are features extracted from the complex spectrum (such as the output of a short-time Fourier transform) that contains amplitude and phase information during signal processing. Complex spectral features preserve complete frequency domain information and can more accurately characterize the time-frequency structure of a signal.

[0038] By using this approach, electronic devices can explicitly retain the phase information in the complex spectrum based on the complex spectrum characteristics, providing a complete time-frequency structure for subsequent track separation models.

[0039] Step S300: Given that N complex spectral features have been cached, generate a complex spectral feature block based on the N complex spectral features.

[0040] In some embodiments, after the electronic device generates complex spectral features, it caches the complex spectral features in a pre-created first memory space corresponding to the complex spectral feature caching module.

[0041] The complex spectral feature cache module can be used to provide the address of the first memory space.

[0042] When N consecutive complex spectral features are cached in the memory space, the electronic device packages the N complex spectral features into a complex spectral feature block based on the complex spectral feature caching module.

[0043] Wherein, N is greater than or equal to 1. The specific value of N is not limited in the embodiments disclosed herein.

[0044] It's important to note that a larger value for N results in a greater delay in the audio processing method, but also better performance of the audio signal separated by the electronic device. Specifically, the value of N affects the delay of the audio processing method. For example, if the frame shift for the audio is 10 milliseconds, then the delay of the audio processing method is N × 10 milliseconds. Thus, a larger value for N allows the electronic device to acquire more information about future frames (such as similar harmonic structures in consecutive frames) within a delay of N × 10 milliseconds, resulting in higher separation quality of the audio signal. The electronic device can achieve a balance between separation quality and algorithm delay by controlling the value of N.

[0045] By adopting this approach, the electronic device can perform subsequent calculations based on the complex spectrum feature blocks. In this way, the electronic device can not only process complex spectrum features in batches, improving hardware utilization, but also estimate the time-frequency mask more accurately based on the context information between adjacent complex spectrum features in the complex spectrum feature blocks, significantly improving the separation quality.

[0046] Furthermore, this implementation breaks through the limitations of the frame-level separation model. It uses complex spectrum feature blocks composed of complex spectrum features from multiple frames as processing units, eliminating audio dropouts or artifacts caused by inter-frame discontinuity, and ensuring that the output audio has a more natural and smooth sound quality.

[0047] Furthermore, this implementation uses complex spectrum feature blocks composed of multiple frames of complex spectrum features as processing units. In the subsequent audio processing using feature dimensionality reduction and an efficient fully convolutional model inference architecture, it significantly reduces the multiplication and addition operation requirements per unit time while maintaining the same separation quality. This enables the audio processing algorithm to run efficiently on the automotive chip platform, greatly expanding the application scenarios of the technology.

[0048] Step S400: Based on the first multi-scale convolutional layer, the complex spectral feature block is encoded to obtain the encoded features corresponding to the complex spectral feature block.

[0049] In some embodiments, the electronic device inputs cached complex spectral feature blocks into a fully convolutional neural network model based on a complex spectral feature caching module.

[0050] The fully convolutional neural network model includes a multi-scale subband encoding module and a multi-scale subband mask estimator module.

[0051] The electronic device first inputs the complex spectral feature block into the multi-scale subband coding module in the fully convolutional neural network model, so as to encode the complex spectral features within the multi-scale subband coding module and obtain the encoded features corresponding to the complex spectral feature block.

[0052] Among them, the multi-scale subband coding module is used to extract and encode the input complex spectral features in parallel across multiple frequency subbands and multiple time / frequency scales to obtain more discriminative and robust hierarchical coding features.

[0053] Using this approach, the electronic device encodes complex spectral feature blocks based on a multi-scale subband coding module, combining features from both the time and frequency domains. This provides more refined time-frequency domain features, effectively enhancing the ability of the fully convolutional neural network model to resolve complex acoustic scenarios (such as the mixing of human voices and accompaniment, and non-stationary noise interference), and providing a high-quality initial representation for subsequent mask estimation and track separation tasks.

[0054] Step S500: Based on the decoding and encoding features of the second multi-scale convolutional layer, the feature mask value is obtained.

[0055] In some embodiments, the electronic device outputs the encoded features from the multi-scale subband coding module to the multi-scale subband mask estimator module, and decodes the encoded features based on the multi-scale subband mask estimator module to obtain the feature mask value.

[0056] The multi-scale sub-band mask estimator module is used to estimate the spectral mask values ​​for separating the target sound source in parallel across multiple frequency sub-bands and multiple time-frequency receptive field scales, i.e., the feature mask values ​​in this embodiment of the present disclosure.

[0057] Feature mask values ​​refer to one or more complex or real values ​​generated by a fully convolutional neural network model for each time-frequency bin in the input complex spectrum feature block during audio track separation. Feature mask values ​​characterize the presence probability, relative energy ratio, or phase information of a target sound source (e.g., human voice) in the corresponding time-frequency bin. They are used to weighted modulate the original complex spectrum of the mixed audio to suppress non-target sound source components and preserve or enhance target sound source components. Feature mask values ​​enable the separation of audio signals from different audio tracks.

[0058] By employing this approach, electronic devices utilize multi-scale processing mechanisms (such as using convolutional or attention units with different receptive fields in parallel) to enable neural network models to simultaneously capture short-term transient events (such as the start of a drumbeat) and long-term harmonic continuity (such as the fundamental frequency trajectory of a human voice), thereby generating more accurate and coherent feature mask values, resulting in better naturalness and clarity in the subsequently separated audio signals.

[0059] Step S600: Use the feature mask value to perform masking calculation on N complex spectral features to obtain the separated complex spectral features corresponding to each sound source component.

[0060] In some embodiments, the electronic device obtains the feature mask value output by the multi-scale sub-band mask estimator module based on the signal separation calculation module, and obtains N complex spectrum features from the complex spectrum feature block cache module based on the signal separation calculation module. Further, the electronic device performs masking calculations on the N complex spectrum features using the feature mask value based on the signal separation calculation module to obtain the separated complex spectrum features corresponding to each sound source component.

[0061] Masking computation is the process of performing element-wise mathematical operations on feature mask values ​​and complex spectral feature blocks to generate an estimated complex spectrum corresponding to the target sound source. This process typically involves multiplying, adding, or performing more complex nonlinear combinations of the real and imaginary parts of the feature mask values ​​with the real and imaginary parts of the complex spectral feature blocks, respectively, to obtain the filtered spectral representation of the target sound source, i.e., the separated complex spectral features.

[0062] By using this method, the electronic device applies the feature mask value to multiple complex spectral features (such as spectral representations under different channels, different time frames, or different sound source assumptions), which can achieve parallel output of the separation results of multiple target sound sources (such as vocals, drums, bass, piano, etc.), significantly improving the efficiency and separation consistency of multi-track audio processing.

[0063] Step S700: Based on the inverse short-time Fourier transform, the complex spectrum features after separation are processed to obtain the multi-track separated audio signal.

[0064] In some embodiments, the electronic device uses an inverse short-time fourier transform (ISTFT) to process the complex spectral features after separation based on the signal separation calculation module to obtain a multi-track separated audio signal.

[0065] The inverse short-time Fourier transform is the inverse process of the short-time Fourier transform, and it is used to convert processed frequency domain features into time-domain audio signals.

[0066] Among them, the complex spectral features after separation are essentially frequency domain features, which electronic devices need to convert into audible time-domain audio signals, i.e., multi-track separated audio signals.

[0067] Using this method, the electronic device converts the complex spectrum characteristics after separation based on the inverse short-time Fourier transform, ensuring the audibility, phase consistency, and real-time performance of the output multi-track separated audio signal, thus achieving high-fidelity audio track separation.

[0068] The technical solution of this disclosure involves an electronic device acquiring an input audio signal, processing the audio signal based on a short-time Fourier transform, and obtaining at least one complex spectrum feature. With N complex spectrum features already cached, a complex spectrum feature block is generated based on these N features. The complex spectrum feature block is encoded using a first multi-scale convolutional layer to obtain the corresponding encoded features. The encoded features are decoded using a second multi-scale convolutional layer to obtain a feature mask value. The feature mask value is used to perform masking calculations on the N complex spectrum features to obtain the separated complex spectrum features corresponding to each sound source component. The separated complex spectrum features are processed using an inverse short-time Fourier transform to obtain a multi-track separated audio signal. These operations ensure the stability of the audio signal quality during the track separation process, while avoiding track leakage and achieving high-fidelity track separation.

[0069] Figure 4 This is a flowchart of step S200 provided in an exemplary embodiment of the present disclosure.

[0070] like Figure 4 As shown, step S200, which processes the audio signal based on short-time Fourier transform to obtain at least one complex spectral feature, may include the following steps: Step S210: Process the audio signal in frames to obtain a frame sequence, which includes at least one audio frame.

[0071] In some embodiments, the electronic device uses an audio conversion module to perform time-domain framing processing on continuously input audio signals according to a fixed duration (e.g., 20 milliseconds) to obtain a frame sequence. The frame sequence includes at least one audio frame.

[0072] By employing this approach, the electronic device transforms an infinitely long audio stream into finite-length processing units (i.e., audio frames). This allows for the computation of subsequent complex spectral features within a fixed duration, reducing the complexity of a single computation and meeting the stringent end-to-end latency requirements of automotive scenarios. Furthermore, subsequent calculations are performed frame-by-frame, with the electronic device's current frame-segmentation providing the input foundation for subsequent calculations.

[0073] Step S220: Window each audio frame to obtain at least one windowed audio frame, wherein each audio frame corresponds to one windowed audio frame.

[0074] In some embodiments, the electronic device uses an audio transformation module to multiply each audio frame by a window function to achieve windowing processing and obtain windowed audio frames.

[0075] By employing this method, the electronic device, through a windowing process, smooths the frame boundaries of audio frames, significantly reduces spectral sidelobes, and improves frequency resolution accuracy, thereby effectively suppressing spectral leakage. Furthermore, windowed audio frames also improve sound quality stability and reduce dropped audio.

[0076] Step S230: Process each windowed audio frame based on discrete Fourier transform to obtain at least one complex spectral feature, wherein each windowed audio frame corresponds to a complex spectral feature.

[0077] In some embodiments, the electronic device performs a discrete Fourier transform on each windowed audio frame based on an audio transformation module to convert a finite-length time-domain discrete signal to a frequency-domain representation, thereby obtaining at least one complex spectral feature from the complex spectrum.

[0078] Using this method, the complex spectrum features obtained by the electronic device simultaneously contain amplitude and phase information. Thus, during subsequent calculations of the complex spectrum features, the electronic device can learn the phase pattern differences between different sound sources (e.g., continuous phase for vocals, abrupt phase changes for percussion), thereby effectively suppressing track leakage (e.g., vocals remaining in the accompaniment).

[0079] The technical solution of this disclosure embodiment is that the electronic device processes the audio signal based on the short-time Fourier transform to obtain at least one complex spectral feature, which can provide a complete time-frequency characterization for the subsequent calculation process, thereby achieving high-precision audio track separation and determining the input basis for audio track-level fine rendering.

[0080] Figure 5 This is a flowchart of step S400 provided in an exemplary embodiment of the present disclosure.

[0081] like Figure 5 As shown, step S400, which encodes complex spectral feature blocks based on the first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectral feature blocks, may include the following steps: Step S410: Based on the time indices corresponding to the N complex spectral features, perform multi-scale segmentation of the complex spectral feature block in the time dimension to obtain at least one time index group.

[0082] In some embodiments, the dimension of the complex spectral feature block is [N1, F1] as an example.

[0083] For example, N1=10, F1=1024.

[0084] Where N1 represents the number of time frame indices in the time dimension, and F1 represents the number of frequency indices in the frequency dimension.

[0085] Based on N1=10, if the electronic device performs multi-scale segmentation based on the number of time frame indexes, and needs to ensure that the number of frames M in the segmented group can be divided by N1, then the value of M includes 1, 2, 5, and 10.

[0086] In this way, for each value of M, the electronic device evenly divides the N1 time frames into N1 / M consecutive subsequences, so that each subsequence contains M adjacent time frames, thus forming N1 / M time index groups. The feature sub-block dimension corresponding to each time index group is [M, F1].

[0087] In other words, when M=1, the electronic device can obtain 10 time index groups, each containing 1 time frame, corresponding to 10 feature sub-blocks with dimensions [1, 1024].

[0088] Correspondingly, when M=2, the electronic device can obtain 5 time index groups, each containing 2 time frames, corresponding to 5 feature sub-blocks with dimensions [2, 1024].

[0089] When M=5, the electronic device can obtain 2 time index groups, each containing 5 time frames, corresponding to 2 feature sub-blocks with dimensions [5, 1024].

[0090] When M=10, the electronic device can obtain one time index group, each group containing 10 time frames, corresponding to one feature sub-block with dimensions [10, 1024].

[0091] Thus, the electronic device generates a total of 18 (i.e., 10+5+2+1) time index groups. Each group represents a multi-scale temporal structure ranging from fine-grained transient response (i.e., M=1) to coarse-grained long-term context dependence (i.e., M=10). Subsequently, the electronic device can uniformly flatten or pool the feature sub-blocks of each group to maintain consistency in the frequency dimension and stack them in the group dimension, ultimately forming a multi-scale time-grouped feature representation with dimensions [18, 1024].

[0092] By adopting this approach, electronic devices can use a multi-scale time segmentation mechanism to enable subsequent algorithms to perceive the spectral dynamics under different time receptive fields in parallel. This effectively balances local detail modeling with global temporal consistency, significantly improving the resolution and robustness of non-stationary acoustic events in tasks such as audio separation, speech enhancement, or sound source localization.

[0093] Step S420: Based on the frequency indices corresponding to the N complex spectral features, the complex spectral feature block is divided into multiple scales according to the frequency dimension to obtain at least one frequency index group.

[0094] In some embodiments, the dimension of the complex spectral feature block is [N1, F1] as an example.

[0095] For example, N1=10, F1=1024.

[0096] Where N1 represents the number of time frame indices in the time dimension, and F1 represents the number of frequency indices in the frequency dimension.

[0097] Based on F1=1024, if the electronic device performs multi-scale segmentation based on the number of frequency indices, and needs to ensure that 2 X When dividing the group unit, the value of X ranges from 1 to 9, and the sub-band sizes corresponding to X are 2, 4, 8, 16, 32, 64, 128, 256, and 512, respectively.

[0098] In this way, the electronic device can be divided based on different sub-band sizes to obtain 9 sets of frequency domain features. The feature sub-block dimensions corresponding to the frequency index group include [2, 512], [4, 256], [8, 128], [16, 64], [32, 32], [64, 16], [128, 8], [256, 4], and [512, 2].

[0099] Thus, the electronic device generates a total of nine time index groups. Each group represents a multi-scale temporal structure ranging from fine-grained transient response to coarse-grained long-term context-dependent structure.

[0100] By adopting this approach, electronic devices can use a multi-scale frequency segmentation mechanism to enable subsequent algorithms to perceive the spectral dynamics of different frequency receptive fields in parallel. This effectively balances local detail modeling with global temporal consistency, significantly improving the resolution and robustness of non-stationary acoustic events in tasks such as audio separation, speech enhancement, or sound source localization.

[0101] Step S430: Construct a multi-scale time-frequency feature map based on all time index groups and all frequency index groups.

[0102] In some embodiments, the electronic device zero-padding is performed on the complex spectral features corresponding to all time index groups and all frequency index groups to construct a multi-scale time-frequency feature map.

[0103] Zero-padding alignment is an operation that uses zero-value padding to make feature blocks of different scales and sizes uniform in shape or fixed in length in a specific dimension.

[0104] Further based on the aforementioned example, if the dimension of the complex spectrum feature block is [10, 1024], then the dimension of the corresponding constructed time-frequency feature map is [18, 512, 512].

[0105] In the time-frequency feature map, the first dimension represents the number of groups corresponding to a time frame. As shown in the previous example, if the number of groups corresponding to a time frame is 18, then the first dimension corresponds to 18. The second dimension represents the number of frequency points in the sub-band. The third dimension represents the number of groups in the sub-band. As shown in the previous example, the number of frequency points in a sub-band can be 512, and the maximum number of groups in a sub-band is 512. Therefore, both the second and third dimensions can be uniformly zero-padded to align to 512.

[0106] By employing this approach, the electronic device zero-pasting and aligning the multi-scale time-frequency feature maps ensures that subsequent calculations can be performed based on a fixed dimension, thus meeting batch processing requirements. Furthermore, since sub-bands of different scales (such as narrowband or broadband) originally have different shapes and cannot be directly spliced ​​or compared, the electronic device can process all scales in parallel within the same tensor space by zero-padding the sub-bands to a uniform length, facilitating the design of cross-scale interaction mechanisms.

[0107] Step S440: Encode the multi-scale time-frequency feature map based on the first multi-scale convolutional layer to obtain the encoded features.

[0108] In some embodiments, the electronic device performs convolution calculations on the multi-scale time-frequency feature map based on a first multi-scale convolutional layer to encode the multi-scale time-frequency feature map and obtain encoded features.

[0109] Using this approach, the electronic device encodes the multi-scale time-frequency feature map based on the first multi-scale convolutional layer, providing a high-quality initial representation for subsequent mask estimation and audio track separation tasks.

[0110] The technical solution of this disclosure embodiment enables the electronic device to encode complex spectral feature blocks based on a multi-scale subband coding module, which effectively enhances the ability of the fully convolutional neural network model to resolve complex acoustic scenes (such as the mixing of human voice and accompaniment, and non-stationary noise interference), and provides a high-quality initial representation for subsequent mask estimation and track separation tasks.

[0111] Figure 6 This is a flowchart of step S400 provided in an exemplary embodiment of the present disclosure.

[0112] like Figure 6 As shown, step S440, which encodes the multi-scale time-frequency feature map based on the first multi-scale convolutional layer to obtain the encoded features, may include the following steps: Step S441: The multi-scale time-frequency features in the multi-scale time-frequency feature map are concatenated with the pre-stored convolutional layer time-series features to obtain the concatenated feature map.

[0113] In some embodiments, the electronic device uses a multi-scale subband coding module to concatenate the multi-scale time-frequency features in the multi-scale time-frequency feature map with the pre-stored convolutional layer temporal features obtained from the first convolutional layer temporal feature caching module to obtain a concatenated feature map.

[0114] Further based on the aforementioned example, the multi-scale time-frequency features are [18, 512, 512].

[0115] For example, the temporal features of the convolutional layer are [6, 128, 128].

[0116] Electronic devices can form spliced ​​feature maps based on [18, 512, 512] and [6, 128, 128].

[0117] By employing this approach, the pre-stored temporal features of the convolutional layer originate from the encoding results of the previous few frames, carrying the dynamic evolution trajectory of the historical audio. Therefore, by fusing the pre-stored temporal features of the convolutional layer with the current multi-scale time-frequency features, the electronic device can effectively avoid temporal discontinuity issues and improve the auditory coherence and naturalness of the output audio.

[0118] Step S442: Parallel processing and splicing of feature maps based on convolutional branches with different numbers of channels in the first multi-scale convolutional layer.

[0119] For example, the electronic device uses 1×1, 3×3, and 5×5 convolutional branches in parallel to process the stitched feature maps. The number of convolutional layers in each of the three branches decreases exponentially; for instance, the 1×1 branch has 16 convolutional layers, the 3×3 branch has 8, and the 5×5 branch has 4. These convolutional branches form a residual convolutional structure.

[0120] By employing this approach, electronic devices provide different time-frequency receptive fields based on different convolution kernel sizes, significantly improving their ability to resolve complex mixed audio.

[0121] Step S443: Based on the processed spliced ​​feature map, feature fusion and feature dimension compression are performed to obtain the encoded features.

[0122] In some embodiments, the electronic device performs feature fusion and feature dimension compression on the processed spliced ​​feature map based on a 1×1 convolutional layer to obtain coded features.

[0123] Further based on the foregoing example, the compressed feature dimension obtained by the electronic device based on the dimension in the foregoing example is, for example, [1, 512, 512].

[0124] By adopting the implementation method of this step, the coded features obtained by the electronic device after feature dimension compression retain the core separation clues after multi-scale and multi-context fusion, which can be directly used in the subsequent mask estimation process, greatly reducing the burden on the decoder.

[0125] The technical solution of this disclosure embodiment is that the electronic device encodes the multi-scale time-frequency feature map based on the first multi-scale convolutional layer, providing a high-quality initial representation for subsequent mask estimation and audio track separation tasks.

[0126] Figure 7 This is another schematic flowchart of an audio processing method provided in an exemplary embodiment of this disclosure.

[0127] like Figure 7 As shown, after obtaining the encoded features by encoding the multi-scale time-frequency feature map based on the first multi-scale convolutional layer in step S440, the following steps may also be included: Step S450: Perform feature compression, similarity calculation, and temporal rearrangement on the encoded features to obtain updated convolutional layer temporal features.

[0128] In some embodiments, the electronic device performs feature caching temporal processing (e.g., feature compression, similarity calculation, and temporal rearrangement) within the multi-scale subband coding module based on the interaction between the multi-scale subband coding module and the first convolutional layer temporal feature caching module, in order to obtain updated convolutional layer temporal features.

[0129] The temporal feature caching module of the first convolutional layer corresponds to a pre-created second memory space. This module can be used to provide the address of the second memory space.

[0130] Electronic devices can cache updated convolutional layer temporal features to the second memory space corresponding to the first convolutional layer temporal feature caching module.

[0131] By employing this approach, electronic devices enhance the contextual information of the fully convolutional neural network model based on updated temporal features of the convolutional layers, thereby improving audio separation quality.

[0132] Figure 8 This is a flowchart of step S450 provided in an exemplary embodiment of this disclosure.

[0133] like Figure 8 As shown, step S450, which involves performing feature compression, similarity calculation, and temporal rearrangement on the encoded features to obtain updated convolutional layer temporal features, may include the following steps: Step S451: Perform feature compression processing on the encoded features based on max pooling and convolution operations to obtain the first compressed feature.

[0134] In some embodiments, the electronic device uses a multi-scale subband coding module to perform feature compression processing on the coded features using max pooling and 2×2 convolution operations to obtain the first compressed feature.

[0135] Further based on the aforementioned example, the electronic device can perform feature compression processing on the encoded features with dimensions [1, 512, 512] to obtain a first compressed feature with dimensions [1, 128, 128].

[0136] By adopting this approach, the electronic device reduces the feature dimension of the encoded features, thereby improving the efficiency of subsequent calculations.

[0137] Step S452: Load the convolutional layer temporal features. The convolutional layer temporal features include a first-dimensional feature, a second-dimensional feature, and a third-dimensional feature. The first-dimensional feature is the time dimension feature of the historical cache, and the second-dimensional feature and the third-dimensional feature both represent the frequency dimension feature corresponding to each inference process.

[0138] In some embodiments, the electronic device loads convolutional layer temporal features from the second memory space corresponding to the first convolutional layer temporal feature cache module based on the multi-scale subband coding module.

[0139] For example, the temporal features of the convolutional layer are [6, 128, 128]. That is, the first dimension feature is 6, representing the time dimension feature cached in 6 historical inference processes. The second and third dimensions are both 128, representing the frequency dimension feature corresponding to each inference process, which is 128.

[0140] Using this approach, the electronic device loads the temporal features of the convolutional layer to obtain the dynamic evolution trajectory of historical audio, thereby effectively avoiding temporal breakage issues in subsequent calculations and improving the coherence and naturalness of the output audio.

[0141] Step S453: Calculate the convolutional layer feature similarity based on the first compressed feature and the convolutional layer temporal feature to obtain the first absolute value corresponding to the feature similarity value. The first absolute value is less than or equal to 1.

[0142] For example, the electronic device calculates the cosine similarity of the first compressed feature with feature dimensions [1, 128, 128] and the temporal features of the convolutional layer with feature dimensions [6, 128, 128] based on the formula cos(A, B) = (A × B) / (|A| × |B|).

[0143] In this way, the electronic device can obtain the absolute values ​​corresponding to the 6 sets of feature similarity values.

[0144] Furthermore, the electronic device can normalize the absolute values ​​corresponding to the 6 sets of feature similarity values ​​to obtain the first absolute value.

[0145] For example, the first absolute values ​​include [0.83, 0.01, 0.65, 0.99, 0.002, 0.36].

[0146] Using this method, electronic devices can quantify the acoustic correlation between the current audio frame and historical audio frames. The closer the first absolute value is to 1, the more likely the current audio frame and historical audio frames belong to the same speaker, the same instrument, or the same semantic unit, and their context should be taken into account first.

[0147] Step S454: Filter the first absolute value based on a preset similarity threshold, and arrange the filtered first absolute values. The filtered first absolute values ​​do not include the minimum value among the first absolute values, nor do they include at least one first target absolute value that is less than the preset similarity threshold.

[0148] In some embodiments, the electronic device may filter out first absolute values ​​that are less than a preset similarity threshold, and filter out the smallest first absolute value to obtain the filtered first absolute value.

[0149] For example, the preset similarity threshold is 0.5.

[0150] It should be noted that the specific value of the preset similarity threshold is not limited in the embodiments disclosed herein.

[0151] Further based on the aforementioned example, if the first absolute value includes [0.83, 0.01, 0.65, 0.99, 0.002, 0.36], then based on the above filtering method, the electronic device can filter out the first absolute values ​​less than 0.5, namely 0.01 and 0.002, and filter out the smallest first absolute value, namely 0.002. Thus, the filtered first absolute values ​​include [0.83, 0.65, 0.99, 0.36].

[0152] By using this method, electronic devices can filter out irrelevant historical audio frames, avoid introducing incorrect prior information, and improve the anti-interference capability of fully convolutional neural network models.

[0153] Step S455: Based on the temporal rearrangement features corresponding to the first sort, determine the updated convolutional layer temporal features, wherein each value in the first sort corresponds to a temporal rearrangement feature, the first value in the first sort is 1, the values ​​after the first value are the first absolute values ​​after filtering, the values ​​after the first absolute values ​​after filtering are the first target absolute values ​​arranged from largest to smallest, and the first value in the first sort corresponds to the first compressed feature.

[0154] Further based on the aforementioned example, the electronic device obtains a first ranking of [1.0, 0.83, 0.65, 0.99, 0.36, 0.01]. Based on the temporal rearrangement features corresponding to the first ranking, the electronic device determines the updated temporal features of the convolutional layer.

[0155] By employing this approach, the electronic device ensures that the most relevant historical context information is cached in the second memory space corresponding to the temporal feature caching module of the first convolutional layer.

[0156] The technical solution of this disclosure embodiment enhances the contextual information of the fully convolutional neural network model based on updated convolutional layer temporal features, thereby improving audio separation quality.

[0157] Figure 9 This is a flowchart of step S500 provided in an exemplary embodiment of this disclosure.

[0158] like Figure 9 As shown, step S500, based on the decoding and encoding features of the second multi-scale convolutional layer, obtains the feature mask value, including: Step S510: Based on the decoding and processing of the coding features by the second multi-scale convolutional layer, a multi-scale time-frequency feature mask and a complex spectrum feature mask are obtained.

[0159] In some embodiments, the electronic device uses a multi-scale subband mask estimator module to decode and process the encoded features using a second multi-scale convolutional layer to obtain a multi-scale time-frequency feature mask and a complex spectrum feature mask.

[0160] Using this method, the electronic device simultaneously generates two types of masks, ensuring the accuracy of the audio separation process in subsequent calculations.

[0161] Step S520: Process the multi-scale time-frequency feature map based on the multi-scale time-frequency feature map estimation method to obtain the first feature mask value corresponding to the multi-scale time-frequency feature map.

[0162] In some embodiments, the electronic device calculates branches based on a 1×1 convolutional layer and a hyperbolic tangent (tanh) function, and processes the multi-scale time-frequency feature mask using a multi-scale time-frequency feature map estimation method to obtain the first feature mask value corresponding to the multi-scale time-frequency feature map.

[0163] In this implementation method, the electronic device is used to implement one step of the two-step mask estimation method, so that the neural network model can learn mutually constrainedly based on two dimensions, namely, more granular multi-scale time-frequency features and complex spectrum features, during the training phase, thereby solving the leakage problem between different audio tracks.

[0164] Step S530: Process the complex spectrum feature mask based on the complex spectrum feature mask estimation method to obtain the second feature mask value corresponding to the complex spectrum feature.

[0165] In some embodiments, the electronic device calculates branches based on a 1×1 convolutional layer and a hyperbolic tangent function, processes the complex spectrum feature mask using a complex spectrum feature mask estimation method, and obtains the second feature mask value corresponding to the complex spectrum feature.

[0166] In this implementation method, the electronic device is used to implement one step of the two-step mask estimation method, so that the neural network model can learn mutually constrainedly based on two dimensions, namely, more granular multi-scale time-frequency features and complex spectrum features, during the training phase, thereby solving the leakage problem between different audio tracks.

[0167] The technical solution disclosed herein introduces a more refined separation mechanism in electronic devices while maintaining low processing latency. By optimizing the signal modeling scale and mask reconstruction logic, it significantly reduces signal leakage between audio tracks, enabling the separated audio to maintain extremely high purity in real-time rendering scenarios.

[0168] Figure 10 This is a flowchart of step S510 provided in an exemplary embodiment of this disclosure.

[0169] like Figure 10 As shown, step S510 involves decoding and processing the encoded features based on the second multi-scale convolutional layer to obtain a multi-scale time-frequency feature mask and a complex spectrum feature mask, including: Step S511: The encoded features and the pre-stored temporal features of the decoded convolutional layer are concatenated to obtain the first concatenated features.

[0170] In some embodiments, the electronic device, based on the multi-size subband mask estimator module, concatenates the encoded features output by the multi-scale subband coding module with the pre-stored decoded convolutional layer temporal features obtained from the second convolutional layer temporal feature cache module to obtain the first concatenated feature.

[0171] Further based on the aforementioned example, the encoding feature is [1, 512, 512].

[0172] For example, the temporal features of the decoded convolutional layer are [6, 128, 128].

[0173] Electronic devices can form a first splicing feature based on [1, 512, 512] and [6, 128, 128].

[0174] By adopting this approach, electronic devices can improve the timing consistency of the mask based on the first splicing feature, thus facilitating subsequent calculations.

[0175] Step S512: Based on the convolution branches with different numbers of channels in the second multi-scale convolutional layer, the first spliced ​​feature is processed in parallel to obtain the multi-scale time-frequency feature mask and complex spectrum feature mask corresponding to the first spliced ​​feature.

[0176] For example, the electronic device processes the first stitched feature in parallel using 1×1, 3×3, and 5×5 convolutional branches, respectively. The number of convolutional layers in each of the three branches decreases exponentially; for instance, the 1×1 branch has 16 convolutional layers, the 3×3 branch has 8, and the 5×5 branch has 4. These convolutional branches form a residual convolutional structure.

[0177] In particular, the second multi-scale convolutional layer consists entirely of deconvolution operators.

[0178] By employing this approach, electronic devices provide different time-frequency receptive fields based on different convolution kernel sizes, significantly improving their ability to resolve complex mixed audio.

[0179] The technical solution of this disclosure embodiment enables electronic devices to perform audio processing based on masks, which can significantly improve the timing stability, spectral precision, and waveform reconstruction quality of audio separation / speech.

[0180] Figure 11 This is another schematic flowchart of an audio processing method provided in an exemplary embodiment of the present disclosure.

[0181] like Figure 11 As shown, in step S510, after obtaining the multi-scale time-frequency feature mask and the complex spectrum feature mask based on the decoding and processing of the encoded features by the second multi-scale convolutional layer, the following steps are also included: Step S513 involves performing feature compression, similarity calculation, and temporal rearrangement on the multi-scale time-frequency feature mask and complex spectrum feature mask to obtain updated temporal features of the decoded convolutional layer.

[0182] In some embodiments, the electronic device performs feature caching temporal processing (e.g., feature compression, similarity calculation, and temporal rearrangement) within the multi-scale subband mask estimator module based on the interaction between the multi-scale subband mask estimator module and the second convolutional layer temporal feature caching module, in order to obtain updated decoded convolutional layer temporal features.

[0183] The temporal feature caching module of the second convolutional layer corresponds to a pre-created third memory space. This module can be used to provide the address of the third memory space.

[0184] Electronic devices can cache updated temporal features of the decoded convolutional layer to the third memory space corresponding to the temporal feature caching module of the second convolutional layer.

[0185] By employing this approach, the electronic device enhances the contextual information of the fully convolutional neural network model based on updated temporal features of the decoding convolutional layers, thereby improving audio separation quality.

[0186] Figure 12 This is a flowchart of step S513 provided in an exemplary embodiment of this disclosure.

[0187] like Figure 12 As shown, step S513 involves performing feature compression, similarity calculation, and temporal rearrangement on the multi-scale time-frequency feature mask and the complex spectrum feature mask to obtain updated temporal features of the decoding convolutional layer, including: Step S5131: Based on max pooling and convolution operations, feature compression processing is performed on the multi-scale time-frequency feature mask and complex spectrum feature mask to obtain the second compressed feature.

[0188] In some embodiments, the electronic device uses a multi-scale subband mask estimator module to perform feature compression processing on the multi-scale time-frequency feature mask and complex spectrum feature mask using max pooling and 2×2 convolution operations to obtain a second compressed feature.

[0189] This process can be derived based on the way the first compression feature is obtained from the aforementioned electronic device, and will not be described in detail in this embodiment.

[0190] By adopting this approach, the electronic device reduces the feature dimension of the multi-scale time-frequency feature mask and the complex spectrum feature mask, thereby improving the efficiency of subsequent calculations.

[0191] Step S5132: Load the temporal features of the decoded convolutional layer. The temporal features of the decoded convolutional layer include a first-dimensional decoded feature, a second-dimensional decoded feature, and a third-dimensional decoded feature. The first-dimensional decoded feature is the temporal dimension decoded feature of the historical cache, and the second-dimensional decoded feature and the third-dimensional decoded feature both represent the frequency dimension decoded feature corresponding to each inference process.

[0192] In some embodiments, the electronic device loads the decoded convolutional layer temporal features from the third memory space corresponding to the second convolutional layer temporal feature cache module based on the multi-scale subband mask estimator module.

[0193] For example, the temporal features of the decoded convolutional layer are [6, 128, 128]. That is, the first dimension of the decoded feature is 6, representing the time dimension features cached during the 6 historical inference processes. The second and third dimensions of the decoded feature are both 128, representing the frequency dimension features corresponding to each inference process, which are 128.

[0194] Using this approach, the electronic device loads the temporal features of the decoding convolutional layer to obtain the dynamic evolution trajectory of historical audio, thereby effectively avoiding temporal breakage issues in subsequent calculations and improving the coherence and naturalness of the output audio.

[0195] Step S5133: Calculate the convolutional layer feature similarity based on the second compressed feature and the temporal features of the decoded convolutional layer, and obtain the second absolute value corresponding to the feature similarity value. The second absolute value is less than or equal to 1.

[0196] Step S5134: Filter the second absolute value based on a preset similarity threshold, and arrange the filtered second absolute values. The filtered second absolute values ​​do not include the minimum value among the second absolute values, nor do they include at least one second target absolute value that is less than the preset similarity threshold.

[0197] Step S5135: Based on the temporal rearrangement features corresponding to the second sorting, determine the updated temporal features of the decoding convolutional layer. Each value in the second sorting corresponds to a temporal rearrangement feature. The first value in the second sorting is 1, and the values ​​after the first value are the filtered second absolute values. The values ​​after the filtered second absolute values ​​are the second target absolute values ​​arranged from largest to smallest. The first value in the second sorting corresponds to the second compression feature.

[0198] The specific implementation of steps S5133-S5135 can be derived based on the aforementioned steps S453-S455, and will not be elaborated here.

[0199] The technical solution of this disclosure embodiment enhances the contextual information of the fully convolutional neural network model based on updated temporal features of the decoding convolutional layer, thereby improving the audio separation quality.

[0200] The solutions disclosed herein are not limited to the embodiments mentioned above.

[0201] Exemplary device The audio processing method provided by the embodiments of this disclosure has been described above. It is understood that, in order to implement the various functions of the audio processing method, the audio processing apparatus may include corresponding hardware and software for implementing the hardware functions.

[0202] Those skilled in the art will readily recognize that the steps of the audio processing method described in conjunction with the embodiments of this disclosure can be implemented in hardware or in a combination of software-driven hardware. Whether a function is implemented in hardware or software-driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0203] Figure 13 This is a schematic diagram of the structure of an audio processing apparatus provided in an exemplary embodiment of the present disclosure.

[0204] like Figure 13 As shown, in one embodiment, the audio processing device 1300 includes: The acquisition module 1310 is used to acquire the input audio signal.

[0205] The first processing module 1320 is used to process the audio signal based on short-time Fourier transform to obtain at least one complex spectral feature.

[0206] The generation module 1330 is used to generate a complex spectrum feature block based on the N complex spectrum features, given that N complex spectrum features have been cached.

[0207] The encoding module 1340 is used to encode the complex spectral feature block based on the first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectral feature block.

[0208] The decoding module 1350 is used to decode the encoded features based on the second multi-scale convolutional layer to obtain the feature mask value.

[0209] The calculation module 1360 is used to perform masking calculations on the N complex spectrum features using the feature mask value to obtain the separated complex spectrum features corresponding to each sound source component.

[0210] The second processing module 1370 is used to process the separated complex spectrum features based on the inverse short-time Fourier transform to obtain a multi-track separated audio signal.

[0211] The audio processing apparatus provided in the above embodiments of this disclosure acquires an input audio signal; processes the audio signal based on short-time Fourier transform to obtain at least one complex spectrum feature; generates a complex spectrum feature block based on the N complex spectrum features, given that N complex spectrum features have been cached; encodes the complex spectrum feature block based on a first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectrum feature block; decodes the encoded features based on a second multi-scale convolutional layer to obtain a feature mask value; performs masking calculations on the N complex spectrum features using the feature mask value to obtain the separated complex spectrum features corresponding to each sound source component; processes the separated complex spectrum features based on inverse short-time Fourier transform to obtain a multi-track separated audio signal. This solves the problem that current real-time sound effect processing algorithms cannot debug and render audio at the track level, leading to problems such as timbre destruction and abnormal spatial sense, thus improving sound quality stability and achieving high-fidelity track separation.

[0212] In one possible implementation, the first processing module 1320 is further configured to process the audio signal in frames to obtain a frame sequence, the frame sequence including at least one audio frame; to window each audio frame to obtain at least one windowed audio frame, wherein each audio frame corresponds to a windowed audio frame; and to process each windowed audio frame based on discrete Fourier transform to obtain at least one complex spectral feature, wherein each windowed audio frame corresponds to a complex spectral feature.

[0213] In one possible implementation, the encoding module 1340 is further configured to: perform multi-scale segmentation of the complex spectrum feature block in the time dimension based on the time indices corresponding to the N complex spectrum features, to obtain at least one time index group; perform multi-scale segmentation of the complex spectrum feature block in the frequency dimension based on the frequency indices corresponding to the N complex spectrum features, to obtain at least one frequency index group; construct a multi-scale time-frequency feature map based on all the time index groups and all the frequency index groups; and encode the multi-scale time-frequency feature map based on the first multi-scale convolutional layer to obtain the encoded features.

[0214] In one possible implementation, the encoding module 1340 is further configured to: concatenate the multi-scale time-frequency features in the multi-scale time-frequency feature map with the pre-stored convolutional layer temporal features to obtain a concatenated feature map; process the concatenated feature map in parallel based on the convolutional branches with different channel numbers in the first multi-scale convolutional layer; and perform feature fusion and feature dimension compression based on the processed concatenated feature map to obtain encoded features.

[0215] In one possible implementation, the apparatus further includes a first update module. After the encoding module 1340 encodes a multi-scale time-frequency feature map based on a first multi-scale convolutional layer to obtain encoded features, the first update module is used to perform feature compression processing, similarity calculation processing, and temporal rearrangement processing on the encoded features to obtain updated convolutional layer temporal features.

[0216] In one possible implementation, the first update module is further configured to perform feature compression processing on the encoded features based on max pooling and convolution operations to obtain a first compressed feature; load convolutional layer temporal features, which include a first-dimensional feature, a second-dimensional feature, and a third-dimensional feature, wherein the first-dimensional feature is the time-dimensional feature of the historical cache, and the second-dimensional feature and the third-dimensional feature both represent the frequency-dimensional feature corresponding to each inference process; calculate the convolutional layer feature similarity based on the first compressed feature and the convolutional layer temporal features to obtain a first absolute value corresponding to the feature similarity value, wherein the first absolute value is less than or equal to 1; filter the first absolute value based on a preset similarity threshold and arrange the filtered first absolute values; determine the updated convolutional layer temporal features based on the temporal reordering features corresponding to the first sort, wherein each value in the first sort corresponds to a temporal reordering feature, the first value in the first sort is 1, and the values ​​after the first value are the first absolute values ​​arranged in descending order, and the first value in the first sort corresponds to the first compressed feature.

[0217] In one possible implementation, the decoding module 1350 is further configured to decode and process the encoded features based on the second multi-scale convolutional layer to obtain a multi-scale time-frequency feature mask and a complex spectrum feature mask; process the multi-scale time-frequency feature mask based on a multi-scale time-frequency feature mask estimation method to obtain a first feature mask value corresponding to the multi-scale time-frequency feature map; and process the complex spectrum feature mask based on a complex spectrum feature mask estimation method to obtain a second feature mask value corresponding to the complex spectrum feature.

[0218] In one possible implementation, the decoding module 1350 is further configured to concatenate the encoded features and the pre-stored temporal features of the decoded convolutional layer to obtain a first concatenated feature; and to process the first concatenated feature in parallel based on the convolutional branches with different numbers of channels in the second multi-scale convolutional layer to obtain the multi-scale time-frequency feature mask and the complex spectrum feature mask corresponding to the first concatenated feature.

[0219] In one possible implementation, the apparatus further includes a second update module. After the decoding module 1350 performs decoding processing of the encoded features based on the second multi-scale convolutional layer to obtain a multi-scale time-frequency feature mask and a complex spectrum feature mask, the second update module is used to perform feature compression processing, similarity calculation processing, and temporal rearrangement processing on the multi-scale time-frequency feature mask and the complex spectrum feature mask to obtain updated decoded convolutional layer temporal features.

[0220] In one possible implementation, the second update module is further configured to: perform feature compression processing on the multi-scale time-frequency feature mask and complex spectrum feature mask based on max pooling and convolution operations to obtain a second compressed feature; load the temporal features of the decoded convolutional layer, which include a first-dimensional decoded feature, a second-dimensional decoded feature, and a third-dimensional decoded feature, wherein the first-dimensional decoded feature is the time-dimensional decoded feature of the historical cache, and the second-dimensional decoded feature and the third-dimensional decoded feature both represent the frequency-dimensional decoded feature corresponding to each inference process; calculate the convolutional layer feature similarity based on the second compressed feature and the temporal features of the decoded convolutional layer to obtain a second absolute value corresponding to the feature similarity value, wherein the second absolute value is less than or equal to 1; filter the second absolute value based on a preset similarity threshold and arrange the filtered second absolute values; and determine the updated temporal features of the decoded convolutional layer based on the temporal reordering feature corresponding to the second sort, wherein each value in the second sort corresponds to a temporal reordering feature, the first value in the second sort is 1, and the values ​​after the first value are the second absolute values ​​arranged in descending order, and the first value in the second sort corresponds to the second compressed feature.

[0221] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects in the exemplary method section above, and will not be repeated here.

[0222] Exemplary electronic devices Figure 14 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure.

[0223] like Figure 14 As shown, the electronic device 1400 includes at least one processor 1410 and a memory 1420.

[0224] The processor 1410 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1400 to perform desired functions.

[0225] The memory 1420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1410 may execute one or more computer program instructions to implement the audio processing methods and / or other desired functions of the various embodiments of this disclosure described above.

[0226] In one example, the electronic device 1400 may also include an input device 1430 and an output device 1440, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0227] The input device 1430 may also include, for example, a keyboard, a mouse, etc.

[0228] The output device 1440 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0229] Of course, for the sake of simplicity, Figure 14 Only some of the components of the electronic device 1400 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1400 may include any other suitable components depending on the specific application.

[0230] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the audio processing methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0231] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0232] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the audio processing methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0233] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0234] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0235] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An audio processing method, comprising: Acquire the input audio signal; The audio signal is processed based on the short-time Fourier transform to obtain at least one complex spectral feature; Given that N complex spectral features have been cached, a complex spectral feature block is generated based on the N complex spectral features; The complex spectral feature block is encoded based on the first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectral feature block; The encoded features are decoded based on the second multi-scale convolutional layer to obtain the feature mask value; The feature mask value is used to perform masking calculations on the N complex spectral features to obtain the separated complex spectral features corresponding to each sound source component; The separated complex spectrum features are processed by inverse short-time Fourier transform to obtain multi-track separated audio signals.

2. The method according to claim 1, wherein, The audio signal is processed based on short-time Fourier transform to obtain at least one complex spectral feature, including: The audio signal is processed in frames to obtain a frame sequence, the frame sequence including at least one audio frame; Each audio frame is windowed to obtain at least one windowed audio frame, wherein each audio frame corresponds to one windowed audio frame; Each windowed audio frame is processed by discrete Fourier transform to obtain at least one complex spectral feature, wherein each windowed audio frame corresponds to one complex spectral feature.

3. The method according to claim 1, wherein, The process of encoding the complex spectral feature block based on the first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectral feature block includes: Based on the time indices corresponding to the N complex spectral features, the complex spectral feature block is divided into multiple scales in the time dimension to obtain at least one time index group; Based on the frequency indices corresponding to the N complex spectral features, the complex spectral feature block is divided into multiple scales according to the frequency dimension to obtain at least one frequency index group; A multi-scale time-frequency feature map is constructed based on all of the time index groups and all of the frequency index groups; The encoded features are obtained by encoding the multi-scale time-frequency feature map based on the first multi-scale convolutional layer.

4. The method according to claim 3, wherein, The process of encoding the multi-scale time-frequency feature map based on the first multi-scale convolutional layer to obtain the encoded features includes: The multi-scale time-frequency features in the multi-scale time-frequency feature map are concatenated with the pre-stored convolutional layer temporal features to obtain a concatenated feature map. The stitched feature map is processed in parallel based on the convolutional branches with different numbers of channels in the first multi-scale convolutional layer; Based on the processed spliced ​​feature map, feature fusion and feature dimension compression are performed to obtain the encoded features.

5. The method according to claim 4, wherein, After obtaining the encoded features by encoding the multi-scale time-frequency feature map based on the first multi-scale convolutional layer, the method further includes: The encoded features are subjected to feature compression, similarity calculation, and temporal rearrangement to obtain the updated temporal features of the convolutional layer.

6. The method according to claim 5, wherein, The process of performing feature compression, similarity calculation, and temporal rearrangement on the encoded features to obtain updated temporal features of the convolutional layer includes: The encoded features are compressed using max pooling and convolution operations to obtain the first compressed feature. Load the temporal features of the convolutional layer, which include a first-dimensional feature, a second-dimensional feature, and a third-dimensional feature. The first-dimensional feature is the temporal dimension feature of the historical cache, and the second-dimensional feature and the third-dimensional feature both represent the frequency dimension feature corresponding to each inference process. Based on the first compressed feature and the convolutional layer temporal feature, the convolutional layer feature similarity is calculated to obtain the first absolute value corresponding to the feature similarity value, wherein the first absolute value is less than or equal to 1; The first absolute value is filtered based on a preset similarity threshold, and the filtered first absolute values ​​are arranged. The filtered first absolute values ​​do not include the minimum value among the first absolute values, nor do they include at least one first target absolute value that is less than the preset similarity threshold. Based on the temporal rearrangement features corresponding to the first sorting, the updated temporal features of the convolutional layer are determined, wherein each value in the first sorting corresponds to a temporal rearrangement feature, the first value in the first sorting is 1, the values ​​after the first value are the filtered first absolute values, the values ​​after the filtered first absolute values ​​are the first target absolute values ​​arranged from largest to smallest, and the first value in the first sorting corresponds to the first compression feature.

7. The method according to claim 1, wherein, The step of decoding the encoded features based on the second multi-scale convolutional layer to obtain the feature mask value includes: Based on the decoding processing of the encoded features by the second multi-scale convolutional layer, a multi-scale time-frequency feature mask and a complex spectrum feature mask are obtained; The multi-scale time-frequency feature map is processed based on the multi-scale time-frequency feature map estimation method to obtain the first feature mask value corresponding to the multi-scale time-frequency feature map; The complex spectrum feature mask is processed based on the complex spectrum feature mask estimation method to obtain the second feature mask value corresponding to the complex spectrum feature.

8. The method according to claim 7, wherein, The decoding process based on the second multi-scale convolutional layer to obtain the encoded features, resulting in a multi-scale time-frequency feature mask and a complex spectral feature mask, includes: The encoded features are concatenated with the pre-stored temporal features of the decoded convolutional layer to obtain the first concatenated feature; The first spliced ​​feature is processed in parallel by convolutional branches with different numbers of channels in the second multi-scale convolutional layer to obtain the multi-scale time-frequency feature mask and the complex spectrum feature mask corresponding to the first spliced ​​feature.

9. The method according to claim 8, wherein, After decoding the encoded features based on the second multi-scale convolutional layer to obtain the multi-scale time-frequency feature mask and the complex spectral feature mask, the method further includes: The multi-scale time-frequency feature mask and the complex spectrum feature mask are subjected to feature compression, similarity calculation, and temporal rearrangement to obtain the updated temporal features of the decoded convolutional layer.

10. The method according to claim 9, wherein, The process of performing feature compression, similarity calculation, and temporal rearrangement on the multi-scale time-frequency feature mask and the complex spectrum feature mask to obtain the updated temporal features of the decoded convolutional layer includes: Based on max pooling and convolution operations, feature compression processing is performed on the multi-scale time-frequency feature mask and the complex spectrum feature mask to obtain the second compressed feature; Load the temporal features of the decoding convolutional layer, which include a first-dimensional decoding feature, a second-dimensional decoding feature, and a third-dimensional decoding feature. The first-dimensional decoding feature is a temporal-dimensional decoding feature of the historical cache, and the second-dimensional decoding feature and the third-dimensional decoding feature both represent the frequency-dimensional decoding feature corresponding to each inference process. Based on the second compression feature and the temporal feature of the decoded convolutional layer, the convolutional layer feature similarity is calculated to obtain the second absolute value corresponding to the feature similarity value, and the second absolute value is less than or equal to 1; The second absolute value is filtered based on a preset similarity threshold, and the filtered second absolute values ​​are arranged. The filtered second absolute values ​​do not include the minimum value among the second absolute values, nor do they include at least one second target absolute value that is less than the preset similarity threshold. Based on the temporal rearrangement features corresponding to the second sorting, the updated temporal features of the decoding convolutional layer are determined, wherein each value in the second sorting corresponds to a temporal rearrangement feature, the first value in the second sorting is 1, the values ​​after the first value are the filtered second absolute values, the values ​​after the filtered second absolute values ​​are the second target absolute values ​​arranged from largest to smallest, and the first value in the second sorting corresponds to the second compression feature.

11. An audio processing apparatus, comprising: The acquisition module is used to acquire the input audio signal; The first processing module is used to process the audio signal based on short-time Fourier transform to obtain at least one complex spectral feature; A generation module is used to generate a complex spectrum feature block based on N complex spectrum features, given that N complex spectrum features have been cached. The encoding module is used to encode the complex spectral feature block based on the first multi-scale convolutional layer to obtain the encoded features corresponding to the complex spectral feature block; The decoding module is used to decode the encoded features based on the second multi-scale convolutional layer to obtain the feature mask value; The calculation module is used to perform masking calculations on the N complex spectral features using the feature mask values ​​to obtain the separated complex spectral features corresponding to each sound source component. The second processing module is used to process the separated complex spectrum features based on the inverse short-time Fourier transform to obtain a multi-track separated audio signal.

12. A computer-readable storage medium storing a computer program for performing the audio processing method according to any one of claims 1-10.

13. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the audio processing method according to any one of claims 1-10.

Images