Audio processing method and audio processing apparatus and non-transitory computer readable medium

Abstract

The present disclosure relates to audio processing methods and apparatuses. The audio processing method can include performing convolution processing on an audio mixture signal using a first convolution layer circuit and a second convolution layer circuit to generate an input feature map; performing separation processing on the input feature map corresponding to a predetermined neural network model to obtain first and second output feature maps corresponding to first and second types, respectively; and performing deconvolution processing on different portions of the first output feature map using a first deconvolution layer circuit and a second deconvolution layer circuit, and performing deconvolution processing on different portions of the second output feature map using the first and second deconvolution layer circuits, respectively, to generate corresponding processing results for generating a plurality of audio output signals.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to signal processing, and in particular, to an audio processing method and an audio processing device, such as a multi-length-convolution / deconvolution-layer-based audio processing method and device, and a non-transitory computer-readable medium. BACKGROUND

[0002] Audio separation aims to separate an audio mixture signal and extract the part we are interested in. It can be applied in many different fields, such as the earphone, speaker, and television industries, to improve user experience. For example, singing voice separation can separate an audio mixture signal into a singing voice and background music. The clean singing voice can be used for singer identification, automatic lyrics transcriptions, pitch tracking, vocal enhancement effect, etc. The separated background music can also be an asset for entertainment. Therefore, a novel method and related architecture are needed to achieve reliable audio separation processing without or less likely to bring side effects. SUMMARY

[0003] It is an object of the present disclosure to provide an audio processing method and an audio processing device, such as a multi-length-convolution / deconvolution-layer-based audio processing method and device, and a non-transitory computer-readable medium, to solve the above problems.

[0004] It is another object of the present disclosure to provide an audio processing method and an audio processing device, such as a multi-length-convolution / deconvolution-layer-based audio processing method and device, and a non-transitory computer-readable medium, to improve the overall performance of a neural network architecture for audio processing.

[0005] At least one embodiment of the present disclosure provides an audio processing method, wherein the method is applicable to an electronic device (e.g., a television, an audio-video system, etc.) having an audio processing function. The method can include: performing convolution processing on an audio mixed signal using a first convolution layer circuit and a second convolution layer circuit to generate an input feature map, wherein each convolution layer circuit in the first convolution layer circuit and the second convolution layer circuit includes multiple layers of one-dimensional (1D) convolution kernels, and the convolution kernel layer length of the first convolution layer circuit and the second convolution layer circuit is equal to a first length and a second length, respectively; performing separation processing on the input feature map corresponding to a predetermined neural network model to obtain a first output feature map and a second output feature map corresponding to a first type and a second type, respectively; and performing deconvolution processing on different parts of the first output feature map using a first deconvolution layer circuit and a second deconvolution layer circuit to generate corresponding processing results to generate a first audio output signal, and performing deconvolution processing on different parts of the second output feature map using the first deconvolution layer circuit and the second deconvolution layer circuit to generate corresponding processing results to generate a second audio output signal, wherein each deconvolution layer circuit in the first deconvolution layer circuit and the second deconvolution layer circuit includes multiple layers of one-dimensional convolution kernels, and the convolution kernel layer length of the first deconvolution layer circuit and the second deconvolution layer circuit is equal to the first length and the second length, respectively.

[0006] At least one embodiment of the present disclosure provides an audio processing device. The audio processing device can include an audio signal analyzer, wherein the audio signal analyzer includes a first convolution layer circuit and a second convolution layer circuit; a separator coupled to the audio signal analyzer; and an audio signal synthesizer coupled to the separator, wherein the audio signal synthesizer includes a first deconvolution layer circuit and a second deconvolution layer circuit. For example, the first convolution layer circuit and the second convolution layer circuit can be used to respectively perform convolution processing on an audio mixing signal to generate an input feature map, wherein each of the first convolution layer circuit and the second convolution layer circuit includes multiple layers of one-dimensional convolution kernels, and the convolution kernel layer length of the first convolution layer circuit and the second convolution layer circuit is equal to a first length and a second length, respectively; the separator can be used to perform separation processing on the input feature map corresponding to a predetermined neural network model to obtain a first output feature map and a second output feature map corresponding to a first type and a second type, respectively; and the first deconvolution layer circuit and the second deconvolution layer circuit can be used to respectively perform deconvolution processing on different parts of the first output feature map to generate corresponding processing results to generate a first audio output signal, and the first deconvolution layer circuit and the second deconvolution layer circuit are respectively used to perform deconvolution processing on different parts of the second output feature map to generate corresponding processing results to generate a second audio output signal, wherein each of the first deconvolution layer circuit and the second deconvolution layer circuit includes multiple layers of one-dimensional convolution kernels, and the convolution kernel layer length of the first deconvolution layer circuit and the second deconvolution layer circuit is equal to the first length and the second length, respectively.

[0007] At least one embodiment of the present disclosure provides a non-transitory computer readable medium storing computer readable instruction codes for causing an audio processing device to perform an audio processing procedure when the computer readable instruction codes are executed, the audio processing procedure comprising: performing convolution processing on an audio mixed signal using a first convolution layer circuit and a second convolution layer circuit to generate an input feature map, wherein each of the first convolution layer circuit and the second convolution layer circuit comprises multiple layers of one-dimensional convolution kernels, and the convolution kernel layer length of the first convolution layer circuit and the second convolution layer circuit is equal to a first length and a second length, respectively; performing separation processing on the input feature map corresponding to a predetermined neural network model to obtain a first output feature map and a second output feature map corresponding to a first type and a second type, respectively; and performing deconvolution processing on different parts of the first output feature map using a first deconvolution layer circuit and a second deconvolution layer circuit to generate corresponding processing results to generate a first audio output signal, and performing deconvolution processing on different parts of the second output feature map using the first deconvolution layer circuit and the second deconvolution layer circuit to generate corresponding processing results to generate a second audio output signal, wherein each of the first deconvolution layer circuit and the second deconvolution layer circuit comprises multiple layers of one-dimensional convolution kernels, and the convolution kernel layer length of the first deconvolution layer circuit and the second deconvolution layer circuit is equal to the first length and the second length, respectively.

[0008] The audio processing method and the audio processing device of the present disclosure can perform audio processing by means of convolution layers of different lengths and deconvolution layers of different lengths, so as to improve the overall performance of the neural network architecture for audio processing. Compared with the related art, the audio processing method and the audio processing device of the present disclosure can greatly improve the quality of the sound output of the audio separation processing. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 A schematic diagram of an audio processing device according to an embodiment of the present disclosure.

[0010] Figure 2 A schematic diagram of a control scheme based on multi-length convolution / deconvolution layers of an audio processing method according to an embodiment of the present disclosure.

[0011] Figure 3 An example of a 1D convolution / deconvolution layer involved in the audio processing method is shown.

[0012] Figure 4 An example of a feature map involved in the audio processing method is shown.

[0013] Figure 5An example of a mask involved in the audio processing method is illustrated.

[0014] Figure 6 A flowchart of the audio processing method according to an embodiment of the present disclosure is illustrated.

[0015] Figure 7 A non-transitory computer-readable medium involved in the audio processing method according to an embodiment of the present disclosure is illustrated. DETAILED DESCRIPTION

[0016] One or more embodiments of the present disclosure relate to an audio processing method and an audio processing device to improve the overall performance of a neural network architecture for audio processing. Compared to a traditional audio separation method using a short time Fourier transform (STFT), a time-domain neural network such as a fully-convolutional time-domain audio separation network (Conv-TasNet) can be regarded as an end-to-end audio separation model and can achieve better performance. The lengths of the one-dimensional (1D) convolution kernels in the encoder and the decoder of the Conv-TasNet are relatively short and the same, which means that the content output by the encoder has a high temporal resolution but a low frequency resolution. Due to the low frequency resolution, the time-frequency regions between the harmonics of a sound cannot be properly resolved, so the audio mixed signal cannot be properly separated. Based on the basic architecture of the Conv-TasNet, the audio processing method and the audio processing device of the present disclosure can perform audio processing by means of convolution layers with different lengths and deconvolution layers with different lengths to improve the overall performance of the neural network architecture for audio processing.

[0017] For the above-mentioned Conv-TasNet, please refer to the following document: Yi Luo, and Nima Mesgarani, “Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation”, Transactions on Audio, Speech, and Language Processing (TASLP) 2019, IEEE; by the above reference, the whole of this document is incorporated into the description of the present application to provide support for certain related implementation details.

[0018] Figure 1 FIG. 1 shows a schematic diagram of an audio processing device 100 according to an embodiment of the present application. The audio processing device 100 can be an example of the audio processing device described above. For ease of understanding, the audio processing device 100 can represent an electronic device (e.g., a television, an audio / video system, etc.) having an audio processing function, but the present application is not limited thereto. In some embodiments, the audio processing device 100 can include at least a portion (e.g., a part or all) of the electronic device. For example, the audio processing device 100 can include a portion of the electronic device, such as a control circuit of the electronic device, which can be implemented by an integrated circuit (IC) or the like. For another example, the audio processing device 100 can include specific computer-readable instruction codes (e.g., software scripts or firmware scripts) stored in a non-transitory computer-readable medium (e.g., a memory) of the electronic device, and the control circuit or other special function circuit of the electronic device can implement the operations described in the following embodiments by executing the computer-readable instruction codes. For yet another example, the audio processing device 100 can include all of the electronic device, such as the entirety of the electronic device (e.g., including software, firmware, and hardware). As shown in FIG. 1, the audio processing device 100 can include a processor 110, a memory 120, and a communication interface 130. The processor 110 can be configured to perform the operations described in the following embodiments. The memory 120 can be configured to store data and / or computer-readable instruction codes (e.g., software scripts or firmware scripts) for implementing the operations described in the following embodiments. The communication interface 130 can be configured to perform communication with other devices (e.g., a server, a client, etc.). Figure 1As shown, the audio processing device 100 can include an audio signal analyzer 110 and an audio signal synthesizer 130, and include a separator 120, wherein the separator 120 is coupled to the audio signal analyzer 110, and the audio signal synthesizer 130 is coupled to the separator 120. The audio signal analyzer 110 can include a convolution layer module 110C and a storage circuit 110S, the separator 120 can include a temporal convolution network (TCN) separation module 120T, a mask module 120K, and a plurality of multipliers 120MA and 120MB, and the audio signal synthesizer 130 can include a deconvolution layer module 130D, a storage circuit 130S, and a plurality of adders 130AA and 130AB. In addition, the convolution layer module 110C can include convolution layer circuits 111 and 112, and each of the convolution layer circuits 111 and 112 can include a plurality of layers of 1D convolution kernels. The convolution kernel layer lengths of the convolution layer circuits 111 and 112 are different from each other, and in particular, the convolution kernel layer lengths of the convolution layer circuits 111 and 112 can be equal to a first length such as length M and a second length such as length N, respectively. Furthermore, the deconvolution layer module 130D can include deconvolution layer circuits 131 and 132, and each of the deconvolution layer circuits 131 and 132 can include a plurality of layers of 1D convolution kernels. The convolution kernel layer lengths of the deconvolution layer circuits 131 and 132 are different from each other, and in particular, the convolution kernel layer lengths of the deconvolution layer circuits 131 and 132 can be equal to the first length such as length M and the second length such as length N, respectively. In some embodiments, the audio signal analyzer 110 and the audio signal synthesizer 130 can be an audio signal encoder and an audio signal decoder, respectively, but the present disclosure is not limited thereto.

[0019] For the above-mentioned Conv-TasNet, since the convolution / deconvolution calculation, TCN separation, masking processing, and the like, as well as the implementation of the corresponding convolution / deconvolution layer calculation units, TCN separation modules, masks, and the like, are well known to those skilled in the art of Conv-TasNet, the relevant details are omitted here. The architecture proposed by the present disclosure, such as the architecture shown Figure 1 The architecture shown can use convolution layers of different lengths for audio signal analysis / encoding and deconvolution layers of different lengths for audio signal synthesis / decoding to improve overall performance.

[0020] Based on Figure 1As shown in the architecture, the convolution layer circuits 111 and 112 can respectively perform convolution processing on an audio mixture signal MIXTURE(t) to generate an input feature map FM0. For example, the convolution layer circuit 111 can perform convolution processing on a plurality of samples of the audio mixture signal MIXTURE(t) corresponding to the first length, such as length M, to generate a partial feature map FM01 of the input feature map FM0, and the convolution layer circuit 112 can perform convolution processing on the plurality of samples of the audio mixture signal MIXTURE(t) corresponding to the second length, such as length N, to generate a partial feature map FM02 of the input feature map FM0. A plurality of storage areas (e.g., storage areas #1 and #2) of the storage circuit 110S can temporarily store these partial feature maps FM01 and FM02, and the audio signal analyzer 110 can combine these partial feature maps FM01 and FM02 into the input feature map FM0 for input into the separator 120.

[0021] In addition, the separator 120 can perform separation processing on the input feature map FM0 corresponding to a predetermined neural network model (e.g., the aforementioned end-to-end audio separation model such as the Conv-TasNet model), particularly a separation module therein, to obtain output feature maps FM1 and FM2 respectively corresponding to a first type, such as type A, and a second type, such as type B. For example, the separator 120 can perform the separation processing (e.g., the TCN separation) on the input feature map FM0 using the TCN module 120T corresponding to the predetermined neural network model to generate masks #1 and #2 respectively corresponding to the first type (e.g., type A) and the second type (e.g., type B), and perform multiplication calculation on the input feature map FM0 and the mask #1 corresponding to the first type using the multiplier 120MA to generate the output feature map FM1, and perform multiplication calculation on the input feature map FM0 and the mask #2 corresponding to the second type using the multiplier 120MB to generate the output feature map FM2.

[0022] Further, the deconvolution layer circuits 131 and 132 can respectively perform deconvolution processing on different portions of the output feature map FM1 to generate corresponding processing results to generate a first audio output signal corresponding to the first type, such as type A, such as the audio output signal VOCAL(t). For example, the audio signal synthesizer 130 can receive the output feature map FM1 from the separator 120, and temporarily store the aforementioned different portions of the output feature map FM1 using a set of storage areas of the storage circuit 130S, particularly, perform the following operations:

[0023] (1) using the deconvolution layer circuit 131 to perform deconvolution processing corresponding to the first length (e.g., length M) on a local feature map FM11 of the output feature map FM1 to generate a set of first intermediate samples corresponding to the local feature map FM11;

[0024] (2) using the deconvolution layer circuit 132 to perform deconvolution processing corresponding to the second length (e.g., length N) on a local feature map FM12 (shown in the figure as being located below the layer of another local feature map FM2, only labeled) of the output feature map FM1 to generate a set of second intermediate samples corresponding to the local feature map FM12; and

[0025] (3) using the adder 130AA to perform addition calculation on the set of first intermediate samples corresponding to the local feature map FM11 and the set of second intermediate samples corresponding to the local feature map FM12, respectively, to generate a set of addition calculation results as a plurality of samples of the first audio output signal (such as the audio output signal VOCAL(t)), wherein the first audio output signal (such as the audio output signal VOCAL(t)) can carry the plurality of samples.

[0026] Similarly, the deconvolution layer circuits 131 and 132 can perform deconvolution processing on different parts of the output feature map FM2, respectively, to generate corresponding processing results, to generate a second audio output signal corresponding to the second type (such as type B), such as the audio output signal MUSIC(t). For example, the audio signal synthesizer 130 can receive the output feature map FM2 from the separator 120, and temporarily store the above-mentioned different parts of the output feature map FM2 using another set of storage areas of the storage circuit 130S, in particular, perform the following operations:

[0027] (1) using the deconvolution layer circuit 131 to perform deconvolution processing corresponding to the first length (e.g., length M) on a local feature map FM21 of the output feature map FM2 to generate a set of first intermediate samples corresponding to the local feature map FM21;

[0028] (2) using the deconvolution layer circuit 132 to perform deconvolution processing corresponding to the second length (e.g., length N) on a local feature map FM22 of the output feature map FM2 to generate a set of second intermediate samples corresponding to the local feature map FM22; and

[0029] (3) performing addition calculation on the set of first intermediate samples corresponding to the local feature map FM21 and the set of second intermediate samples corresponding to the local feature map FM22 by the adder 130AB, respectively, to generate a set of addition calculation results as a plurality of samples of the second audio output signal (such as the audio output signal MUSIC(t)), wherein the second audio output signal (such as the audio output signal MUSIC(t)) can carry the plurality of samples.

[0030] For the convenience of understanding, a non-separation path, a type A separation path (e.g., corresponding to a type A separation path), and a type B separation path (e.g., corresponding to a type B separation path) are illustrated in Figure 1 to indicate data paths of related data and data flows thereon, respectively, but the present application is not limited thereto. In addition, the "(t)" in the respective symbols of the above-mentioned signals (such as the audio mixture signal MIXTURE(t), the audio output signal VOCAL(t), and the audio output signal MUSIC(t)) can indicate that these signals are functions of time, respectively. In some embodiments, the type A and the type B can be distinguished according to specific frequencies of audio (e.g., the fundamental frequency of a male voice is approximately 85-180 Hz, and the fundamental frequency of a female voice is approximately 165-255 Hz), the type A approximately corresponds to a vocal component (hence denoted as the audio output signal VOCAL(t)) in the audio mixture signal MIXTURE(t), and the type B approximately corresponds to a music (or background) component (hence denoted as the audio output signal MUSIC(t)) in the audio mixture signal MIXTURE(t).

[0031] According to certain embodiments, the convolution layer circuit 111 can include a set of 1D convolution layers each having the first length (e.g., length M) as the multi-layer 1D convolution kernel in the convolution layer circuit 111. The convolution layer circuit 112 can include a set of 1D convolution layers each having the second length (e.g., length N) as the multi-layer 1D convolution kernel in the convolution layer circuit 112. In addition, the deconvolution layer circuit 131 can include a set of 1D deconvolution layers each having the first length (e.g., length M) as the multi-layer 1D convolution kernel in the deconvolution layer circuit 131. The deconvolution layer circuit 132 can include a set of 1D deconvolution layers each having the second length (e.g., length N) as the multi-layer 1D convolution kernel in the deconvolution layer circuit 132.

[0032] Figure 2 is a schematic diagram of a control scheme of a multi-length convolution / deconvolution layer based on an audio processing method according to an embodiment of the present application. The method can be applied to Figure 1The audio processing apparatus 100 shown is, in particular, an electronic device (e.g., a television, an audio-visual system, etc.) equipped with audio processing capabilities. For example, the first length may represent length M, and the second length may represent length N. For ease of understanding, in... Figure 2 The diagram illustrates 1D convolutional / deconvolutional layers of length M (e.g., the group of 1D convolutional layers in convolutional layer circuit 111 and the group of 1D deconvolutional layers in deconvolutional layer circuit 131), 1D convolutional / deconvolutional layers of length N (e.g., the group of 1D convolutional layers in convolutional layer circuit 112 and the group of 1D deconvolutional layers in deconvolutional layer circuit 132), feature maps (e.g., local feature maps FM01, FM11, and FM21) corresponding to 1D convolutional layers of length M (e.g., the group of 1D convolutional layers in convolutional layer circuit 111), and feature maps (e.g., local feature maps FM02, FM12, and FM22) corresponding to 1D convolutional layers of length N (e.g., the group of 1D convolutional layers in convolutional layer circuit 112), but the invention is not limited thereto. For the sake of simplicity, similar descriptions in this embodiment will not be repeated here.

[0033] Figure 3 An example of a 1D convolutional / deconvolutional layer involved in the described audio processing method is illustrated. Figure 4 An example of a feature map involved in the audio processing method is illustrated, and Figure 5 Examples of masks involved in the described audio processing method are illustrated, where parameters such as M, N, H, and K indicate the associated 1D convolutional / deconvolutional layers, feature maps / local feature maps, and the size of the mask. For example, any one of M, N, H, and K can be a positive integer greater than one, and in some embodiments, H > M > N and H can be much larger than M and N, but the invention is not limited thereto. Figure 3 As shown, the length of any convolutional layer in convolutional layer circuit 111 and the length of any deconvolutional layer in deconvolutional layer circuit 131 are both equal to length M, the length of any convolutional layer in convolutional layer circuit 112 and the length of any deconvolutional layer in deconvolutional layer circuit 132 are both equal to length N, and the number of layers in each convolutional layer circuit in convolutional layer circuits 111 and 112 and the number of layers in each deconvolutional layer circuit in deconvolutional layer circuits 131 and 132 are both equal to K. Figure 4 As shown, the width and height of each feature map in feature maps FM0, FM1, and FM2 are equal to H and 2K respectively (for example, (K+K)=2K), and the width and height of each local feature map in the respective local feature maps {FM01,FM02}, {FM11,FM12}, and {FM21,FM22} of feature maps FM0, FM1, and FM2 are equal to H and K respectively. Figure 5As shown, the width and height of each of the masks #1 and #2 are equal to H and 2K, respectively. For the sake of brevity, similar contents in the present embodiments are not repeated here.

[0034] As mentioned above, if the type A corresponds to the vocal component in the audio mixture signal MIXTURE(t) and the type B corresponds to the music component in the audio mixture signal MIXTURE(t), it is shown that the audio processing method of the present application can employ different kernel lengths for different frequency components (vocal component and music component in the foregoing embodiments) in the audio mixture signal MIXTURE(t) for processing, in the hope of obtaining better results than the prior art with the same kernel length for processing.

[0035] According to some embodiments, H = 1001, M = 480 (corresponding to the kernel length of the convolutional / deconvolutional layers in the type A separation path), N = 32 (corresponding to the kernel length of the convolutional / deconvolutional layers in the type B separation path), and K = 256, but the present application is not limited thereto. For the sake of brevity, similar contents in these embodiments are not repeated here.

[0036] Table 1

[0037]

[0038] Table 2

[0039]

[0040] Table 3

[0041]

[0042] Table 1 shows the experimental results obtained by processing the DSD100 audio data set disclosed on the Internet by the audio processing device 100 according to the embodiments of the present application, Table 2 shows the experimental results obtained by processing the DSD100 audio data set by the audio processing device 100 according to the embodiments of the present application, and Table 3 shows the experimental results obtained by processing the DSD100 audio data set according to the prior art, wherein any set of experimental results in each of Tables 1 and 2 corresponds to a plurality of kernel lengths (indicated as "(long, short)" for brevity), and any set of experimental results in Table 3 corresponds to one kernel length.

[0043] Table 4

[0044]

[0045] Table 5

[0046]

[0047] Table 6

[0048]

[0049] Table 4 shows experimental results obtained by processing the MUSDB18 audio dataset disclosed on the Internet by the audio processing device 100 according to some embodiments of the present application, Table 5 shows experimental results obtained by processing the MUSDB18 audio dataset disclosed on the Internet by the audio processing device 100 according to some embodiments of the present application, and Table 6 shows experimental results obtained by processing the MUSDB18 audio dataset disclosed on the Internet according to the prior art, wherein each set of experimental results in Table 4 and Table 5 corresponds to a plurality of kernel lengths (indicated as “(long, short)” for brevity), and each set of experimental results in Table 6 corresponds to one kernel length.

[0050] The field “Stride” in Tables 1-6 refers to the number of steps that the kernel of a convolutional layer moves when performing convolution operation in the training of a neural network model, the meaning of which is well known to those skilled in the art and thus will not be described herein. In addition, the subsequent fields in Tables 1-6 include the Source to Distortion Ratio (SDR) and the Source to Distortion Ratio improvement (SDRi) Mean of vocals and music, respectively, which are hereinafter referred to as SDR and SDRi, respectively.

[0051] Tables 1-6 show the difference between the experimental results obtained according to some embodiments of the present application and the experimental results obtained according to the prior art. Tables 1-2 and Tables 4-5 respectively show experimental results obtained by processing specific audio datasets (e.g., the DSD100 audio dataset disclosed on the Internet is processed in Tables 1-2, and the MUSDB18 audio dataset disclosed on the Internet is processed in Tables 4-5) by the audio processing device 100 according to the present application with different kernel lengths of the convolutional layers / deconvolutional layers in some embodiments. Tables 3 and 6 respectively show experimental results obtained by processing the specific audio datasets (e.g., the DSD100 audio dataset is processed in Table 3, and the MUSDB18 audio dataset is processed in Table 6) according to the prior art (traditional Conv-TasNet) with the same kernel lengths of the convolutional layers / deconvolutional layers.

[0052] Among the various combinations shown in the experimental results, the case of M = 480 and N = 32 described above is preferred. In this case, the largest SDR value and SDRi value can be obtained.

[0053] As shown in the experimental results, the greater the SDR value, the closer the separated signal is to the target signal and the less the separated signal is interfered by other sound source signals. SDRi is an enhanced SDR, and its calculation method can include:

[0054] (1) performing SDR calculation with the unseparated signal (e.g., the aforementioned audio mixture signal MIXTURE(t)) and the target signal to obtain a first SDR;

[0055] (2) performing SDR calculation with the separated signal and the target signal to obtain a second SDR; and

[0056] (3) performing subtraction calculation according to the first SDR and the second SDR to generate an SDR difference, such as the value obtained by subtracting the first SDR from the second SDR, as SDRi;

[0057] Wherein SDRi can be used to represent the optimization degree of the separated signal compared with the unseparated signal.

[0058] Regardless of Tables 1-3 or Tables 4-6, the experimental results obtained by the audio processing device 100 according to the present application when performing audio processing with different convolution kernel lengths of the convolution layer / deconvolution layer are better than the experimental results obtained by the prior art when performing audio processing with the same convolution kernel length of the convolution layer / deconvolution layer. However, it should be understood that the characteristics of different data sets may affect the selected convolution kernel length, and the application of the present application is not limited to the data sets and convolution kernel lengths used to obtain the experimental results. In general, when the unseparated signal contains more complex instrument components, the device / method of the present application is used to process to achieve better results.

[0059] Figure 6 According to an embodiment of the present application, a flowchart of the audio processing method is shown.

[0060] In step S10, the audio processing device 100 can use the convolution layer circuits 111 and 112 to perform convolution processing on the audio mixture signal MIXTURE(t) respectively to generate input feature maps FM0.

[0061] In step S20, the audio processing device 100 can use the separator 120 to perform separation processing on the input feature maps FM0 corresponding to the predetermined neural network model to obtain output feature maps FM1 and FM2 corresponding to the first type such as type A and the second type such as type B, respectively.

[0062] At step S30, the audio processing device 100 can utilize the deconvolution layer circuits 131 and 132 to respectively perform deconvolution processing on different portions of the output feature map FM1 to generate corresponding processing results, to generate the first audio output signal corresponding to the first type such as type A, such as audio output signal VOCAL(t), and utilize the deconvolution layer circuits 131 and 132 to respectively perform deconvolution processing on different portions of the output feature map FM2 to generate corresponding processing results, to generate the second audio output signal corresponding to the second type such as type B, such as audio output signal MUSIC(t).

[0063] For the sake of brevity, similar content in the present embodiment is not repeated here.

[0064] For better understanding, the audio processing method can be illustrated by the workflow shown in Figure 6 but the present application is not limited thereto. According to certain embodiments, one or more steps can be added, deleted or modified in the workflow shown in Figure 6 but the present application is not limited thereto. According to certain embodiments, one or more steps can be added, deleted or modified in the workflow shown in

[0065] According to certain embodiments, the audio processing method can further comprise performing a training procedure, wherein the training procedure comprises: training the model according to predetermined input audio signals, predetermined audio output signals corresponding to type A and predetermined audio output signals corresponding to type B, to train (e.g. adjust) the parameters of the model, to complete the training by performing the training for multiple times to make the parameters of the model reach respective final values of the parameters, for performing the above-mentioned audio signal processing.

[0066] Figure 7 According to an embodiment of the present application, a non-transitory computer-readable medium 100M involved in the audio processing method is shown, wherein the non-transitory computer-readable medium 100M stores computer-readable instruction code 100P so that the audio processing device 100 (e.g. the control circuit of the electronic device, such as the control circuit 100C of the present embodiment, especially the processor 101 therein) performs an audio processing procedure such as the above-mentioned audio processing (e.g. various operations mentioned in the above-mentioned embodiments) when executing the above-mentioned computer-readable instruction code 100P. The processor 101 can represent at least one processor (e.g. one or more processors) in the control circuit 100C. For the sake of understanding, the processor 101 (e.g. multiple local circuits of the processor 101, such as multiple sub-circuits of a single processor, multiple processor cores of a single processor, multiple processors / processor cores, etc.) running the above-mentioned computer-readable instruction code 100P can be configured to Figure 1The illustrated architecture (e.g., audio signal analyzer 110, separator 120, and audio signal synthesizer 130, in Figure 7 The non-transitory computer-readable medium 100M can represent a storage device / component that can be implemented using, for example, a hard disk drive, a solid-state drive, a Universal Flash Storage (UFS) device, a non-volatile memory component (e.g., an Electrically-Erasable Programmable Read-Only Memory (EEPROM), a Flash memory), and the like, but the present disclosure is not limited to this. For the sake of brevity, similar concepts will not be repeated below.

[0067] According to certain embodiments, the aforementioned audio signal analyzer 110, separator 120, and audio signal synthesizer 130 can be implemented by a processing circuit (e.g., processor 101) running software, and intermediate data generated by the processing circuit running the software can be stored / buffered by using a memory of the electronic device. For example, in the case where the control circuit of the electronic device (e.g., a television) is implemented as an IC (e.g., a television IC), this memory can be located on this IC.

[0068] The multi-frequency-resolution architecture (MF architecture) of the present disclosure, such as the architecture of the embodiments illustrated in Figure 1 , Figure 2 The MF architecture can apply two types of 1D convolution kernels having different lengths to a neural network model to simultaneously provide content having different frequency resolutions (as described above, for example, a vocal component and a music component), and in particular, enable convolution kernels in a corresponding MF model (e.g., the neural network model having the MF architecture) to function as band-pass filters having different frequency resolutions and to learn to analyze signals having different frequencies using different resolution kernels, which is consistent with the function of a human cochlea. Accordingly, the audio processing method and the audio processing device of the present disclosure can significantly improve the quality of the sound output of the audio separation process.

[0069] The above-described embodiments are merely possible implementations of the present disclosure, and any equivalent changes and modifications made in accordance with the scope of the claims of the present disclosure should be construed as being within the scope of the present disclosure.

[0070] Legend of Reference Numerals

[0071] 100: audio processing device

[0072] 110: audio signal analyzer

[0073] 110C: convolution layer module

[0074] 110S: storage circuit

[0075] 111, 112: convolution layer circuit

[0076] 120: separator

[0077] 120T: TCN separation module

[0078] 120K: mask module

[0079] 120MA, 120MB: multiplier

[0080] 130: audio signal synthesizer

[0081] 130D: deconvolution layer module

[0082] 130S: storage circuit

[0083] 131, 132: deconvolution layer circuit

[0084] 130AA, 130AB: adder

[0085] FM0, FM1, FM2: feature map

[0086] FM01, FM02, FM11, FM12, FM21, FM22: local feature map

[0087] MIXTURE(t): audio mixture signal

[0088] VOCAL(t), MUSIC(t): audio output signal

[0089] 120M: multiplication circuit

[0090] 130A: addition circuit

[0091] M, N, H, K: parameter

[0092] S10 ~ S30: step

[0093] 100M: non-transitory computer readable medium

[0094] 100P: computer readable instruction code

[0095] 100C: control circuit

[0096] 101: processor

Claims

1. A method of audio processing applicable to an electronic device having an audio processing function, the method comprising: performing convolution processing on an audio mixture signal using a first convolution layer circuit and a second convolution layer circuit to generate an input feature map FM0, wherein each of the first convolution layer circuit and the second convolution layer circuit comprises a plurality of layers of one-dimensional convolution kernels, and a length of the layers of convolution kernels of the first convolution layer circuit and the second convolution layer circuit is equal to a first length and a second length, respectively, and performing convolution processing on the audio mixture signal using the first convolution layer circuit and the second convolution layer circuit to generate the input feature map FM0 further comprises performing convolution processing on a plurality of samples of the audio mixture signal using the first convolution layer circuit corresponding to the first length to generate a local feature map FM01 of the input feature map FM0 for input to a separator for performing separation processing, and performing convolution processing on the plurality of samples of the audio mixture signal using the second convolution layer circuit corresponding to the second length to generate a local feature map FM02 of the input feature map FM0 for input to the separator for performing the separation processing; performing separation processing on the input feature map FM0 corresponding to a predetermined neural network model to obtain a first output feature map FM1 and a second output feature map FM2 corresponding to a first type and a second type, respectively; and performing deconvolution processing on different parts of the first output feature map FM1 using a first deconvolution layer circuit and a second deconvolution layer circuit to generate corresponding processing results to generate a first audio output signal, and performing deconvolution processing on different parts of the second output feature map FM2 using the first deconvolution layer circuit and the second deconvolution layer circuit to generate corresponding processing results to generate a second audio output signal, wherein each of the first deconvolution layer circuit and the second deconvolution layer circuit comprises a plurality of layers of one-dimensional convolution kernels, and a length of the layers of convolution kernels of the first deconvolution layer circuit and the second deconvolution layer circuit is equal to the first length and the second length, respectively, the different parts of the first output feature map FM1 include a local feature map FM11 and a local feature map FM12, and the first deconvolution layer circuit and the second deconvolution layer circuit perform deconvolution processing on the local feature map FM11 and the local feature map FM12, respectively, to generate corresponding processing results to generate the first audio output signal, wherein the local feature map FM11 corresponds to a group of one-dimensional deconvolution layers having the first length, and the local feature map FM12 corresponds to a group of one-dimensional deconvolution layers having the second length; and The different portions of the second output feature map FM2 include a local feature map FM21 and a local feature map FM22. The first deconvolution layer circuit and the second deconvolution layer circuit respectively perform deconvolution processing on the local feature map FM21 and the local feature map FM22 to generate corresponding processing results to generate the second audio output signal. The local feature map FM21 corresponds to a set of one-dimensional deconvolution layers with the first length, and the local feature map FM22 corresponds to a set of one-dimensional deconvolution layers with the second length.

2. The audio processing method of claim 1, wherein, Performing the separation process corresponding to the predetermined neural network model on the input feature map FM0 to obtain the first output feature map FM1 and the second output feature map FM2 corresponding to the first type and the second type respectively further includes: The input feature map FM0 is subjected to the separation process using a temporal convolutional network module corresponding to the predetermined neural network model to generate a first mask and a second mask corresponding to the first type and the second type, respectively; and The input feature map FM0 and the first mask are multiplied to generate the first output feature map FM1, and the input feature map FM0 and the second mask are multiplied to generate the second output feature map FM2.

3. The audio processing method of claim 1, wherein, The process of generating the first audio output signal by deconvolving different parts of the first output feature map FM1 using the first deconvolution layer circuit and the second deconvolution layer circuit to produce corresponding processing results further includes: The first deconvolution layer circuit is used to perform deconvolution processing on the local feature map FM11 of the first output feature map FM1, corresponding to the first length, to generate a set of first intermediate samples corresponding to the local feature map FM11 of the first output feature map FM1. The second deconvolutional layer circuit is used to perform deconvolution processing on a local feature map FM12 of the first output feature map FM1, corresponding to the second length, to generate a set of second intermediate samples corresponding to the local feature map FM12 of the first output feature map FM1; and Addition calculations are performed on the first set of intermediate samples in the local feature map FM11 corresponding to the first output feature map FM1 and the second set of intermediate samples in the local feature map FM12 corresponding to the first output feature map FM1, respectively, to generate a set of addition calculation results as multiple samples of the first audio output signal.

4. The audio processing method of claim 1, wherein, The process of generating the second audio output signal by deconvolving different parts of the second output feature map FM2 using the first deconvolution layer circuit and the second deconvolution layer circuit to produce corresponding processing results further includes: The first deconvolution layer circuit is used to perform deconvolution processing on a local feature map FM21 of the second output feature map FM2, corresponding to the first length, to generate a set of first intermediate samples corresponding to the local feature map FM21 of the second output feature map FM2. performing deconvolution processing on a local feature map FM21 of the first output feature map FM1 corresponding to the first length to generate a set of first intermediate samples corresponding to the local feature map FM21 of the first output feature map FM1; and performing addition calculation on the set of first intermediate samples corresponding to the local feature map FM21 of the first output feature map FM1 and the set of second intermediate samples corresponding to the local feature map FM22 of the second output feature map FM2 respectively to generate a set of addition calculation results as a plurality of samples of the first audio output signal.

5. An audio processing device, comprising: an audio signal analyzer, comprising: a first convolution layer circuit and a second convolution layer circuit for performing convolution processing on an audio mixture signal respectively to generate an input feature map FM0, wherein each of the first convolution layer circuit and the second convolution layer circuit comprises a plurality of layers of one-dimensional convolution kernels, and a convolution kernel layer length of the first convolution layer circuit and the second convolution layer circuit is equal to a first length and a second length respectively, performing convolution processing on a plurality of samples of the audio mixture signal corresponding to the first length by the first convolution layer circuit to generate a local feature map FM01 of the input feature map FM0 for inputting to a separator for performing separation processing, and performing convolution processing on the plurality of samples of the audio mixture signal corresponding to the second length by the second convolution layer circuit to generate a local feature map FM02 of the input feature map FM0 for inputting to the separator for performing the separation processing; a separator coupled to the audio signal analyzer for performing separation processing on the input feature map FM0 corresponding to a predetermined neural network model to obtain a first output feature map FM1 and a second output feature map FM2 corresponding to a first type and a second type respectively; and an audio signal synthesizer coupled to the separator, the audio signal synthesizer comprising: a first deconvolution layer circuit and a second deconvolution layer circuit for performing deconvolution processing on different parts of the first output feature map FM1 respectively to generate corresponding processing results to generate a first audio output signal, and performing deconvolution processing on different parts of the second output feature map FM2 respectively by the first deconvolution layer circuit and the second deconvolution layer circuit to generate corresponding processing results to generate a second audio output signal, wherein each of the first deconvolution layer circuit and the second deconvolution layer circuit comprises a plurality of layers of one-dimensional convolution kernels, and a convolution kernel layer length of the first deconvolution layer circuit and the second deconvolution layer circuit is equal to the first length and the second length respectively, the different portions of the first output feature map FM1 include a local feature map FM11 and a local feature map FM12, the first deconvolutional layer circuit and the second deconvolutional layer circuit respectively perform deconvolutional processing on the local feature map FM11 and the local feature map FM12 to generate corresponding processing results, to generate the first audio output signal, wherein the local feature map FM11 corresponds to a set of one-dimensional deconvolutional layers having the first length, and the local feature map FM12 corresponds to a set of one-dimensional deconvolutional layers having the second length; and the different portions of the second output feature map FM2 include a local feature map FM21 and a local feature map FM22, the first deconvolutional layer circuit and the second deconvolutional layer circuit respectively perform deconvolutional processing on the local feature map FM21 and the local feature map FM22 to generate corresponding processing results, to generate the second audio output signal, wherein the local feature map FM21 corresponds to a set of one-dimensional deconvolutional layers having the first length, and the local feature map FM22 corresponds to a set of one-dimensional deconvolutional layers having the second length.

6. The audio processing apparatus of claim 5, wherein, the first convolutional layer circuit includes a set of one-dimensional convolutional layers having the first length respectively, as the multi-layer one-dimensional convolutional kernels in the first convolutional layer circuit, and the first deconvolutional layer circuit includes a set of one-dimensional deconvolutional layers having the first length respectively, as the multi-layer one-dimensional convolutional kernels in the first deconvolutional layer circuit; and the second convolutional layer circuit includes a set of one-dimensional convolutional layers having the second length respectively, as the multi-layer one-dimensional convolutional kernels in the second convolutional layer circuit, and the second deconvolutional layer circuit includes a set of one-dimensional deconvolutional layers having the second length respectively, as the multi-layer one-dimensional convolutional kernels in the second deconvolutional layer circuit.

7. A non-transitory computer-readable medium storing computer-readable instruction codes causing an audio processing device to perform an audio processing procedure when the computer-readable instruction codes are executed, the audio processing procedure comprising: performing convolution processing on an audio mixed signal by using a first convolution layer circuit and a second convolution layer circuit to generate an input feature map FM0, wherein each of the first convolution layer circuit and the second convolution layer circuit comprises multiple layers of one-dimensional convolution kernels, and the convolution kernel layer length of the first convolution layer circuit and the second convolution layer circuit is equal to a first length and a second length respectively, performing convolution processing on the audio mixed signal by using the first convolution layer circuit and the second convolution layer circuit to generate the input feature map FM0 further comprises: performing convolution processing on multiple samples of the audio mixed signal by using the first convolution layer circuit corresponding to the first length to generate a local feature map FM01 of the input feature map FM0 for inputting to a separator for performing separation processing; and performing convolution processing on the multiple samples of the audio mixed signal by using the second convolution layer circuit corresponding to the second length to generate a local feature map FM02 of the input feature map FM0 for inputting to the separator for performing the separation processing; performing separation processing on the input feature map FM corresponding to a predetermined neural network model to obtain a first output feature map FM1 and a second output feature map FM2 corresponding to a first type and a second type respectively; and performing deconvolution processing on different parts of the first output feature map FM1 by using a first deconvolution layer circuit and a second deconvolution layer circuit respectively to generate corresponding processing results to generate a first audio output signal, and performing deconvolution processing on different parts of the second output feature map FM2 by using the first deconvolution layer circuit and the second deconvolution layer circuit respectively to generate corresponding processing results to generate a second audio output signal, wherein each of the first deconvolution layer circuit and the second deconvolution layer circuit comprises multiple layers of one-dimensional convolution kernels, and the convolution kernel layer length of the first deconvolution layer circuit and the second deconvolution layer circuit is equal to the first length and the second length respectively, the different parts of the first output feature map FM1 include a local feature map FM11 and a local feature map FM12, the first deconvolution layer circuit and the second deconvolution layer circuit perform deconvolution processing on the local feature map FM11 and the local feature map FM12 respectively to generate corresponding processing results to generate the first audio output signal, wherein the local feature map FM11 corresponds to a group of one-dimensional deconvolution layers with the first length, and the local feature map FM12 corresponds to a group of one-dimensional deconvolution layers with the second length; and The different portions of the second output feature map FM2 include a local feature map FM21 and a local feature map FM22, and the first de-convolution layer circuit and the second de-convolution layer circuit respectively perform de-convolution processing on the local feature map FM21 and the local feature map FM22 to generate corresponding processing results, so as to generate the second audio output signal, wherein the local feature map FM21 corresponds to a set of one-dimensional de-convolution layers with the first length, and the local feature map FM22 corresponds to a set of one-dimensional de-convolution layers with the second length.

Images