Devices and methods for audio processing

Abstract

A data processing apparatus (100) for processing audio data, in particular musical audio data is disclosed. The data processing apparatus (100) comprises a processing circuitry (101) configured to generate based on one or more spectra of the audio data an audio data tensor, wherein the audio data tensor comprises a plurality of spectral intensity values for a plurality of frequency ranges defined by a first dimension of the audio data tensor and for a plurality of logarithmic sub-frequencies within the respective frequency range defined by a second dimension of the audio data tensor. The processing circuitry (101) is further configured to implement a neural network with at least one convolutional layer configured to generate one or more pluralities of output values by convolving the plurality of spectral intensity values of the audio data tensor with one or more kernel tensors, wherein the one or more kernel tensors comprise a plurality of kernel weights for at least two of the plurality of frequency ranges defined by the first dimension of the audio data tensor and for at least two of the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor. Moreover, a corresponding audio data processing method is disclosed.

Description

[0001] Devices and methods for audio processing

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to audio processing. More specifically, the disclosure relates to devices and methods for processing audio data, in particular music audio data using neural networks.

[0004] BACKGROUND

[0005] Music audio data can be analyzed in many different ways. Automatic music transcription (AMT) refers to the task of extracting musical notes from a given music audio signal within the precise time intervals in which they occur. Since music has many different aspects, a music audio signal is often transformed into a representation that captures information useful for the task to be accomplished. In the case of AMT, one of the most common representations is the Constant-Q Transform (CQT) resulting in a CQT spectrum (also referred to as spectrogram). A CQT spectrum is a frequency-logarithmic representation of the music audio data, where octaves are separated by a fixed number of frequency bins. Modern AMT approaches employ Neural Networks (NN), in particular Convolutional Neural Networks (CNNs) for retrieving the musical notes from CQT spectra generated based on music audio data.

[0006] SUMMARY

[0007] It is an objective to provide improved devices and methods for processing audio data, in particular music audio data using neural networks.

[0008] The foregoing and other objectives are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

[0009] According to a first aspect a data processing apparatus for processing audio data, in particular music audio data is provided. The data processing apparatus comprises a processing circuitry configured to generate based on one or more spectra of the audio data (also referred to as spectrograms) an audio data tensor, wherein the audio data tensor comprises a plurality of spectral intensity values for a plurality of frequency ranges defined by, i.e. along a first dimension of the audio data tensor and for a plurality of logarithmic sub-frequencies within the respective frequency range defined by, i.e. along a second dimension of the audio data tensor. The processing circuitry of the data processing apparatus is further configured to implement a neural network with at least one convolutional layer configured to generate one or more pluralities of output values (also referred to as feature values or features) by convolving the plurality of spectral intensity values of the audio data tensor with one or more kernel tensors, wherein the one or more kernel tensors comprise a plurality of kernel weights for at least two of the plurality of frequency ranges defined by, i.e. along the first dimension of the audio data tensor and for at least two of the plurality of logarithmic sub-frequencies defined by, i.e. along the second dimension of the audio data tensor. The data processing apparatus according to the first aspect requires less padding compared to some conventional approaches. Moreover, the one or more kernel tensors may process the audio data tensor at the relevant portions to extract valuable harmonic information without having to shift or make redundant copies of portions of the audio data tensor. Furthermore, the data processing apparatus allows each kernel tensor to process multiple relevant portions for harmonic information, where conventional approaches require multiple kernel tensors or shifts to process those same portions.

[0010] In a further possible implementation form, the audio data tensor comprises a further plurality of spectral intensity values for a plurality of time intervals, i.e. time bins defined by, i.e. along a third dimension of the audio data tensor. In an implementation form, the audio data tensor may comprise one or more further dimensions, such as a batch dimension and / or a channel dimension.

[0011] In a further possible implementation form, the one or more kernel tensors comprise a further plurality of kernel weights for at least two of the plurality of time intervals defined by, i.e. along the third dimension of the audio data tensor.

[0012] In a further possible implementation form, the plurality of frequency ranges defined by, i.e. along the first dimension of the audio data tensor comprise a plurality of octaves and the plurality of logarithmic sub-frequencies within the respective frequency range comprise a plurality of pitches within the respective octave.

[0013] In a further possible implementation form, the processing circuitry is configured to generate the audio data tensor based on the one or more spectra of the audio data by padding and / or trimming the one or more spectra of the audio data.

[0014] In a further possible implementation form, the one or more kernel tensors comprise a plurality of kernel tensors and wherein each kernel tensor has a different dilation factor for the plurality of logarithmic sub-frequencies defined by, i.e. along the second dimension of the audio data tensor. Each dilation factor may contribute to the analysis of the audio data from different perspectives.

[0015] In a further possible implementation form, the processing circuitry is further configured to combine the one or more pluralities of output values into a output value tensor, for instance, by summing, averaging, and / or concatenating the one or more pluralities of output values.

[0016] In a further possible implementation form, at least one kernel tensor of the plurality of kernel tensors has a dilation factor for the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor corresponding to a perfect fifth music interval.

[0017] In a further possible implementation form, at least one kernel tensor of the plurality of kernel tensors has a dilation factor for the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor corresponding to a major third music interval.

[0018] In a further possible implementation form, at least one kernel tensor of the plurality of kernel tensors has a dilation factor for the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor corresponding to a minor seventh music interval.

[0019] In a further possible implementation form, the data processing apparatus further comprises a communication interface configured to receive the audio data and / or the one or more spectra of the audio data.

[0020] In a further possible implementation form, the data processing apparatus comprises one or more microphones configured to acquire the audio data.

[0021] In a further possible implementation form, the processing circuitry of the data processing apparatus is further configured to generate the one or more spectra of the audio data based on the audio data.

[0022] According to a second aspect a computer-implemented data processing method for processing audio data is provided. The data processing method comprises the steps of: generating based on one or more spectra of the audio data an audio data tensor, wherein the audio data tensor comprises a plurality of spectral intensity values for a plurality of frequency ranges defined by a first dimension of the audio data tensor and for a plurality of logarithmic sub-frequencies within the respective frequency range defined by a second dimension of the audio data tensor; and implementing a neural network with at least one convolutional layer configured to generate a one or more pluralities of output values by convolving the plurality of spectral intensity values of the audio data tensor with one or more kernel tensors, wherein the one or more kernel tensors comprise a plurality of kernel weights for at least two of the plurality of frequency ranges defined by the first dimension of the audio data tensor and for at least two of the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor.

[0023] The method according to the second aspect can be performed by the audio processing apparatus according to the first aspect. Thus, further features of the method according to the second aspect result directly from the functionality of the audio processing apparatus according to the first aspect as well as its different implementation forms and embodiments described above and below.

[0024] According to a third aspect a computer program product is provided, comprising a computer- readable storage medium for storing program code which causes a computer or a processor to perform the method according to the second aspect, when the program code is executed by the computer or the processor.

[0025] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the following, embodiments of the present disclosure are described in more detail with reference to the attached figures and drawings, in which:

[0028] Fig. 1 is a schematic diagram illustrating a data processing apparatus according to an embodiment for processing music audio data;

[0029] Fig. 2 is a schematic diagram illustrating an audio data tensor generated by a data processing apparatus according to an embodiment;

[0030] Fig. 3 is a diagram illustrating an exemplary CQT spectrum used by a data processing apparatus according to an embodiment for generating an audio data tensor;

[0031] Figs. 4a-c are schematic diagrams illustrating a convolutional operation of kernel tensors with different dilation factors on an audio data tensor employed by a data processing apparatus according to an embodiment;

[0032] Fig. 5 is a schematic diagram illustrating processing stages implemented by a data processing apparatus according to an embodiment for processing music audio data; Fig. 6 is a schematic diagram illustrating a trimming and padding operation implemented by a data processing apparatus according to an embodiment for generating an audio data tensor;

[0033] Fig. 7 is a schematic diagrams illustrating a convolutional operation of kernel tensors with different dilation factors on an audio data tensor employed by a data processing apparatus according to an embodiment;

[0034] Fig. 8 is a schematic diagram illustrating a summing and concatenation operation implemented by a data processing apparatus according to an embodiment for generating an output value tensor;

[0035] Fig. 9 is a table illustrating interval distances for different music intervals; and

[0036] Fig. 10 is a flow diagram illustrating a computer-implemented data processing method according to an embodiment for processing audio data, in particular music audio data.

[0037] In the following, identical reference signs refer to identical or at least functionally equivalent features.

[0038] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0040] For instance, it is to be understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with each other, unless specifically noted otherwise.

[0041] Figure 1 is a schematic diagram illustrating a data processing apparatus 100 according to an embodiment for processing audio data, in particular music audio data. The data processing apparatus 100 may be implemented, for instance, as a headset or headphones, a mobile phone, a smartphone, a tablet computer, a laptop computer, a cloud server or a similar device with data processing capabilities. As illustrated in figure 1 , the data processing apparatus 100 comprises a processing circuitry 101 , e.g. one or more processors 101 configured to process audio data, in particular music audio data in the way described in more detail further below. Generally, the processing circuitry 101 of the data processing apparatus 100 may process the audio data for music transcription related tasks, where capturing the harmonic patterns in the spectrum is of high importance, such as automatic music transcription, melody extraction, or chord recognition.

[0042] The processing circuitry 101 of the data processing apparatus 100 may be implemented in hardware and / or software and may comprise digital circuitry, or both analog and digital circuitry. Digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or general-purpose processors. As illustrated in figure 1 , the data processing apparatus 100 may further comprise a communication interface 103 configured to communicate over a wired and / or wireless communication network, for instance, for receiving audio data, in particular music audio data. The data processing apparatus 100 may further comprise a memory 105 configured to store audio data, in particular music audio data as well as executable program code which, when executed by the processing circuitry 101 , causes the data processing apparatus 100 to perform the functions and methods described herein. As illustrated in figure 1 , in an embodiment, the data processing apparatus 100 may further comprise one or more microphones 106 configured to record audio data, in particular music audio data.

[0043] Under further reference to figure 2, the processing circuitry 101 of the data processing apparatus 100 is configured to generate based on one or more spectra of the audio data, such as the exemplary CQT spectrum 300 illustrated in figure 3, an audio data tensor 210. As illustrated in figure 2, the audio data tensor 210 comprises a plurality of spectral intensity values for a plurality of frequency ranges defined by a first dimension of the audio data tensor 210 (illustrated as the “o” axis in figure 2) and for a plurality of logarithmic sub-frequencies within the respective frequency range defined by a second dimension of the audio data tensor (illustrated as the “p” axis in figure 2). The processing circuitry 101 of the data processing apparatus 100 is further configured to implement a neural network with at least one convolutional layer configured to generate one or more pluralities of output values 220 by convolving the plurality of spectral intensity values of the audio data tensor 210 with one or more kernel tensors 215a-m. In the embodiment of figure 2, the processing circuitry 101 of the data processing apparatus 100 is further configured to generate an output value tensor 230 by summing the plurality of output values 220, for instance, along the first dimension, i.e. the “o” dimension. In further embodiments described below in the context of figure 8, the processing circuitry 101 of the data processing apparatus 100 may be configured to generate the output value tensor 230 by averaging or concatenating, i.e. stacking the plurality of output values 220 along the first dimension, i.e. the “o” dimension.

[0044] The one or more kernel tensors 215a-m comprise a plurality of kernel weights for at least two of the plurality of frequency ranges defined by the first dimension, i.e. the “o” dimension of the audio data tensor 210 and for at least two of the plurality of logarithmic sub-frequencies defined by the second dimension, i.e. the “p” dimension of the audio data tensor 210. In the embodiment illustrated in figure 2, the plurality of frequency ranges defined by the first dimension, i.e. the “o” dimension of the audio data tensor 210 comprise a plurality of octaves and the plurality of logarithmic sub-frequencies within the respective frequency range, i.e. the respective octave comprise a plurality of pitches within the respective octave.

[0045] In the embodiment shown in figure 2, the audio data tensor 210 further comprises a time dimension (denoted by T), i.e. a further plurality of spectral intensity values for a plurality of time intervals or time bins defined by a third dimension of the audio data tensor 210. Although the one or more exemplary kernel tensors 215a-m illustrated in figure 2 comprise a further plurality of kernel weights for at least two of the plurality of time intervals defined by the third dimension of the audio data tensor 210, in further embodiments the one or more kernel tensors 215a-m may comprise kernel weights only for a single respective time frame.

[0046] Before describing further embodiments of the data processing apparatus 100 in more detail some common general knowledge concerning audio processing will be introduced in the following. In this context several terms will be used, which have the following meaning herein:

[0047] A fundamental frequency is a frequency associated with a certain pitch, for instance, the tone A4 with a fundamental frequency of 440 Hz. The harmonics are a series of frequencies that are positive integer multiples of a given fundamental frequency. For instance, given the fundamental frequency f0= 440 Hz, the harmonics are fi = 880 Hz, f2= 1320 Hz and so on. The fundamental frequency f0itself may be regarded as the first harmonic in the series.

[0048] Music intervals are defined by the distance or difference in frequency separating two music pitches.

[0049] An octave is a special music interval between two pitches, when the fundamental frequency of one is twice the frequency of the other, such as the music interval between A4 = 440 Hz and A5= 880HZ.

[0050] A pitch class refers to a set of all pitches that are a whole number of octaves apart, such as Co, Ci, C2, Do, Di, and the like in the case of Western music.

[0051] In the context of neural networks feature maps refer to the inputs and outputs of convolutional operations.

[0052] In the context of neural networks, a channel usually refers to a single feature map in a set of feature maps. For example, in computer vision networks the input image is often represented in three dimensions: height, width, and channels, for instance, a red channel, a green channel and a blue channel. Channels may also refer to the result of applying convolutional operations to an input. For example, if two convolution operations #(•), ■(•)areapplied to an input xmxn, the feature maps Adxb= g(X) and Bdxb= f(X may be obtained as the output. Both outputs A and B may be stacked into ycxdx&, where c is the channel dimension.

[0053] When a music note is being played, for instance, by an instrument, usually not only its fundamental frequency appears in a spectrum, but also the harmonics associated with it. This means that in a Constant-Q transform (CQT) spectrum (also referred to as spectrogram) the frequency bins associated with the harmonics of the fundamental frequencies might reflect high magnitude values, making it difficult to discern which of the frequencies with high values are harmonics and which ones are fundamental frequencies associated with notes to be transcribed. The number of frequency bins separating a fundamental frequency from its nthharmonic in a CQT spectrum can be expressed as: dn= round(p * \og-2n) , (1) where p denotes the number of bins per octave. As will be appreciated from equation 1 , firstly the number of frequency bins separating a fundamental and its nthharmonic is the same for all the fundamental frequencies and secondly, dndepends logarithmically on n, and thus, the distance between the fundamentals and their harmonics is not linear. For example, given p = 12, the first 6 harmonics would be separated from the fundamental by: d1= 0 (the fundamental itself), d2= 12 , d3= 19, d4= 24, d5= 28, d6= 31. Figure 3 shows an example of a CQT spectrum 300, illustrating the harmonic series for a Ci note.

[0054] Music intervals and harmonics are closely related. For instance, as already mentioned above, the octave is exactly the second harmonic from the series. Figure 9 illustrates a table highlighting some of the first harmonics along with their associated music intervals. This table also includes the distance in bins (given a spectrum based on a CQT, a Variable-Q transform or a similar transform is used) associated with each interval. N denotes the number of bins per semitone, which may be easily calculated since there are 12 semitones in an octave as follows:

[0055] In the following the relation between the music intervals of the table of figure 9 and equation 1 are illustrated for the exemplary case p = 12 and, thus, N = 1 (from equation 2). For these parameters, the number of bins separating an octave is 12 and some harmonics are separated from the fundamental by multiple octaves. The pitch classes G and E are respectively the fifth and major third of C. Both the 3rd and 6th harmonics correspond to fifths (interval distance = 7, according to the table of figure 9): the 3rd harmonic is (+1 octave) * (12) + (7) = 19 bins apart from the fundamental; the 6th harmonic is (+2 octaves) * (12) + (7) = 31 bins apart from the fundamental. The 5th harmonic corresponds to a major third (interval distance = 4, according to the table of figure 9), meaning that the 5th harmonic is (+2 octaves) * (12) + (4) = 28 bins apart from the fundamental. These music related interval distances are also illustrated in figure 3, and match the d3, d5, d6values obtained from equation (1).

[0056] As illustrated in figures 4a-c, at least some of the plurality of kernel tensors 215a-m implemented by the processing circuitry 101 of the data processing apparatus 100 for analysing the audio data tensor 210 may have different dilation factors for the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor 210. For instance, the kernel tensor 215a of figure 4a may have a dilation factor of 1 , the kernel tensor 215b of figure 4b may have a dilation factor of 2 and the kernel tensor 215c of figure 4c may have a dilation factor of 3 along the second dimension, i.e. the pitch class dimension of the audio data tensor 210. As will be appreciated, kernel tensors 215a-m with different dilation factors allow to analyse parts of the audio data tensor 210 separated by a respective constant amount. Thereby, the multiple dilated kernel tensors 215a-m may capture, for instance, multiple harmonics of music audio data. In an embodiment, one or more of the plurality of kernel tensors 215a-m may have a dilation factor for the plurality of logarithmic sub-frequencies defined by the second dimension, i.e. the p dimension of the audio data tensor 210 corresponding to a perfect fifth music interval, a major third music interval and / or a minor seventh music interval.

[0057] As already described above, the processing circuitry 101 of the data processing apparatus 100 is configured to generate the audio data tensor 210 with the first octave dimension and the second logarithmic pitch class dimension based on one or more spectra of the audio data, such as the exemplary CQT spectrum 300 illustrated in figure 3. In an embodiment, the processing circuitry 101 of the data processing apparatus 100 may be further configured to generate the one or more spectra, for instance, by applying a CQT to one or more music audio signals in the time domain.

[0058] As already described above, the second dimension of the audio data tensor 210 is a logarithmic representation obtained from the one or more spectra 300, such as CQT, VQT or any log-frequency spectra. In an embodiment, the audio data tensor 210 allows capturing music intervals at multiple octaves by splitting the frequency dimension of the one or more spectra 300 into two independent ones. For instance, as illustrated in figure 5, given a logfrequency spectrogram XCin x Fbins x T300 with Cinchannels, Fbinsfrequency bins, and T time frames, the processing circuitry 101 of the data processing apparatus 100 is configured to split the frequency dimension in the octave and the pitch class dimensions, denoted o and p, respectively. The result of this operation is the audio data tensor YCin x 0 x p x T210, where bins — O X p.

[0059] In an embodiment, the ratio is an integer number of octaves. If this is not the case, according to an embodiment, the processing circuitry 101 of the data processing apparatus 100 is configured to pad or trim the frequency dimension so that the ratio is an integer. The audio data tensor 210 is herein also referred to as “octave / pitch spectrum” of the audio data.

[0060] As already described above, the processing circuitry 101 of the data processing apparatus 100 is further configured to analyze the audio data tensor 210 with kernels represented by the one or more dilated kernel tensors 215a-m capable of capturing information related to music intervals through many octaves. While conventional kernels may have dimensions Cinx cout x H x W, wherein Cinand Coutdenote the number of input and output channels and H and W denote the frequency and time frame dimensions of the input respectively, the one or more kernel tensors 215a-m implemented by the processing circuitry 101 of the data processing apparatus 100 for applying a convolution operation to the audio data tensor 210 may have one additional dimension D so that the one or more kernel tensors 215a-m may have the dimensions Cinx Coutx D x H x W. In the embodiment illustrated in figure 5 the extra dimension D is associated with different octaves, while H may have a fixed size of 2 and is associated with different pitch classes. As already described above, the convolutions implemented by the one or more kernel tensors 215a-m may be dilated along the H dimension in accordance with desired music intervals.

[0061] As will be appreciated, due to the additional dimension D of the one or more kernel tensors 215a-m a single kernel tensor 215a-m may capture multiple harmonics. For instance, for the fundamental frequency Ci, the first 4 harmonics are Ci, C2, G2 and C3. As the one or more kernel tensors 215a-m span in both octave and pitch class dimensions, the one or more kernel tensors 215a-m may compare both pitch classes C and G simultaneously through many octaves. If, by way of example, Ci is played, the one or more kernel tensors 215a-m could observe that Ci , C2, G2 and C3 are active while G1 and G2 are not. Thus, the absence of certain pitches may be useful information for determining which fundamental frequencies are present in the audio data being processed.

[0062] As already described above, in an embodiment, the one or more kernel tensors 215a-m may perform a 3D convolution operation of the audio data tensor 210, namely in the dimensions octave, pitch class, and time frame. Due to this movement through the octave dimension, the same kernel tensor 215a-m may analyse different sets of octaves and, thus, may be considered to operate “octave independently”.

[0063] As already described above, the kernel tensors 215a-m may have different dilation factors associated with selected music intervals. In an embodiment, each kernel tensor 215a-m may be applied in parallel 3D convolutions to the audio tensor 210, i.e. the same pitch / octave spectrum. In an embodiment, the processing circuitry 101 of the data processing apparatus 100 is configured to aggregate the output values 220 of the convolutions of the audio data tensor 210 with the kernel tensors 215a-m, for instance, by means of a concatenation or a summation into an output value tensor 230. This approach according to embodiments disclosed herein may improve the discriminatory analysis. For instance, if three convolutional kernel tensors 215a-m are applied to the audio data tensor 210 and the three kernel tensors 215a-m obtain a high activation for Ci, then it is quite likely that Ci is indeed present in the audio data. If on the other hand, Gi achieves a high activation with respect to only one of three kernel tensors 215a-m, it is less likely that Gi is present in the audio data.

[0064] In the embodiment illustrated in figure 5 the one or more spectra 300 for generating the audio data tensor 210 comprise a batch of B log-frequency spectrograms 300 with C,ninput channels. As already described above, the processing circuitry 101 of the data processing apparatus 100 is configured to reshape these spectra 300 into the audio data tensor 210 (i.e. the pitch / octave spectrum) and process the audio data tensor 210 with the plurality of different 3D convolutional kernel tensors 215a-m. In an embodiment, the convolutions may be applied with padding so that the last three dimensions of convolutions’ input and output have the same size. As already described above, the outcome of the convolution with the plurality of kernel tensors 215a-m is a set of m feature maps, i.e. output values 220 that may be aggregated by the processing circuitry 101 of the data processing apparatus 100 into the output value tensor 230 using, for instance, either concatenation, i.e. stacking, summation and / or averaging.

[0065] The processing stages of figure 5 implemented by the data processing apparatus 100 according to an embodiment may be divided into the following three main parts: (a) converting the one or more log-frequency spectra 300 into the audio data tensor 210, i.e. the pitch / octave spectrum; (b) applying the 3D convolutional kernel tensors 215a-m to the audio data tensor 210; and (c) aggregating of the output values 220 of the convolutions. More detailed embodiments of the data processing apparatus 100 for implementing these three main parts are described in the following.

[0066] As already described above, for a given numbers of bins per octave p, in an embodiment the result of should be an integer number of octaves. As illustrated in figure 6, the processing circuitry 101 of the data processing apparatus 100 may be configured in a processing block 601 to check whether this is the case. If this is not the case, the processing circuitry 101 of the data processing apparatus 100 is configured to pad (processing block 603b of figure 6) or trim (processing block 603a of figure 6) the frequency dimension of the one or more spectra 300 for generating the audio data tensor 210 until this condition is met. In an embodiment, this may be directly applied to the one or more log-frequency spectra 300, selecting the closest number satisfying that condition: F^ins. As already mentioned above and illustrated in processing block 603a of figure 6, in an embodiment the processing circuitry 111 of the data processing apparatus 100 is configured to trim the one or more input spectra 300. Trimming the input will result in a spectrogram 300 with less frequency bins, discarding sets of contiguous bins from the frequency axis. These may be sets of frequency bins that are either at the higher or lower extremes of the spectra 300, depending on whether lower of higher frequencies are the most important for the task at hand. In an embodiment, the total number of bins to trim may be Fbins- Fbins, and the trimmed bins can be a combination of higher and lower frequency bins. For instance, for Fbins- Fbins= 5, a possibility is to discard the highest 3 and lowest 2 frequency bins from the spectrum 300.

[0067] In an alternative or additional embodiment, the processing circuitry 101 of the data processing apparatus 100 is configured to pad the one or more spectra 300, as illustrated by the processing block 603b of figure 6. As will be appreciated, padding results in a bigger spectrogram 300 along the frequency axis, by adding values at the top or lower extremes of the spectrogram 300. The kind of padding used could be a set of constant values, mirrored values of the input, circular padding, other values from the spectrogram, or a combination thereof. The total number of bins to add is Fbins- Fbins, and these can be a combination of higher and lower frequency bins. For instance, for Fbins- Fbins= 5, a possibility is to add 3 bins at the top of the spectrogram 300 with zeros, and 2 mirrored frequency bins at the bottom.

[0068] As illustrated in figure 7, in an embodiment the kernel tensors 215a-m may have weights along 5 dimensions, namely kn-. Cinx Coutx Dnx Hnx Wn; n e [l, m] . In an embodiment, the m kernels may differ in size with respect to the dimensions D, W, H. As already described above, in an embodiment, the m kernels 215a-m may be dilated along the H dimension for following the distance in bins of different music intervals as shown in the table of figure 9.

[0069] In an embodiment, the kernel tensors 215a-m may have any D and W size, but the dimension H may be restricted to have size either 1 or 2. Since the octave information is encoded through the D dimension, kernel tensors 215a-m capturing only octave information may have H = 1, while all the other kernel tensors 215a-m may have H = 2.

[0070] In the following an exemplary embodiment of the data processing apparatus 100 according to an embodiment for capturing the first 6 harmonics of audio data will be described. As will be appreciated from the table in figure 9, the first 6 harmonics (indicated by the numbers in the table) contain octave, fifth and major third intervals across three octaves. Three different kinds of intervals require three different kinds of kernel tensors 215a-m. The kernel tensor 215a-m capturing octaves has no dilation across the H dimension as H = 1, and the kernel tensors 215a-m capturing fifths and major thirds have dilations N * 7 and N * 4 respectively. All three kernel tensors 215a-m may have / ) = 3, since as already mentioned, these harmonics are contained within the first three octaves.

[0071] As already described above, in an embodiment the kernel tensors 215a-m may apply only to a single time frame at a time. In other words, in the embodiment illustrated in figure 7 this corresponds to the case W = 1. In this particular case, the audio data tensor 210, i.e. pitch / octave spectrum B x Cinx o x p x T may be reshaped by the processing circuitry 101 of the data processing apparatus 100 into (B * T) x Cinx o x p (meaning that now the different time frames are stacked along the batch dimension). The kernel tensors 215a-m may have then the shape Cinx Coutx Dnx Hn.

[0072] As already described above and illustrated in figure 8, after obtaining the m sets of output values 220 for each of the m kernel tensors 215a-m the processing circuitry 101 of the data processing apparatus 100 may aggregate these output values 220 into an output value tensor 230. To this end, in an embodiment, the processing circuitry 101 of the data processing apparatus 100 may be configured to sum or concatenate, i.e. stack the output values 220. In case of a summation, the output value tensor 230 may have the same dimensions as the audio data tensor 210, i.e. the pitch / octave spectrum. If the outputs are concatenated, i.e. stacked , the channel dimension may be of size m * Coutso that that resulting output value tensor 230 has one additional dimension (in comparison with the summation embodiment), and further downstream processing layers may be configured to combine the information from the convolutions’ outputs 220.

[0073] Figure 10 is a flow diagram illustrating a data processing method 1000 for processing audio data according to an embodiment. The data processing method 1000 comprises a step 1001 of generating based on one or more spectra 300 of the audio data an audio data tensor 210, wherein, as already described above, the audio data tensor 210 comprises a plurality of spectral intensity values for a plurality of frequency ranges defined by a first dimension of the audio data tensor 210 and for a plurality of logarithmic sub-frequencies within the respective frequency range defined by a second dimension of the audio data tensor 210. Moreover, the data processing method 1000 comprises a step 1003 of implementing a neural network with at least one convolutional layer configured to generate a one or more pluralities of output values 220 by convolving the plurality of spectral intensity values of the audio data tensor 210 with one or more kernel tensors 215a-m, wherein the one or more kernel tensors 215a-m comprise a plurality of kernel weights for at least two of the plurality of frequency ranges defined by the first dimension of the audio data tensor 210 and for at least two of the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor 210.

[0074] The audio data processing method 1000 can be performed by the audio data processing apparatus 110 according to an embodiment. Thus, further features of the audio data processing method 1000 result directly from the functionality of the audio data processing apparatus 110 as well as its different embodiments described above and below.

[0075] The person skilled in the art will understand that the "blocks" ("units") of the various figures (method and apparatus) represent or describe functionalities of embodiments of the present disclosure (rather than necessarily individual "units" in hardware or software) and thus describe equally functions or features of apparatus embodiments as well as method embodiments (unit = step).

[0076] In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners.

[0077] For example, the described embodiment of an apparatus is merely exemplary. For example, the unit division is merely logical function division and may be another division in an actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0078] The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

[0079] In addition, functional units in the embodiments of the invention may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

Claims

CLAIMS1. A data processing apparatus (100) for processing audio data, wherein the data processing apparatus (100) comprises: a processing circuitry (101) configured to generate based on one or more spectra (300) of the audio data an audio data tensor (210), wherein the audio data tensor (210) comprises a plurality of spectral intensity values for a plurality of frequency ranges defined by a first dimension of the audio data tensor (210) and for a plurality of logarithmic sub-frequencies within the respective frequency range defined by a second dimension of the audio data tensor (210); wherein the processing circuitry (101) is further configured to implement a neural network with at least one convolutional layer configured to generate one or more pluralities of output values (220) by convolving the plurality of spectral intensity values of the audio data tensor (210) with one or more kernel tensors (215a-m), wherein the one or more kernel tensors (215a-m) comprise a plurality of kernel weights for at least two of the plurality of frequency ranges defined by the first dimension of the audio data tensor (210) and for at least two of the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor (210).

2. The data processing apparatus (100) of claim 1 , wherein the audio data tensor (210) comprises a further plurality of spectral intensity values for a plurality of time intervals defined by a third dimension of the audio data tensor (210).

3. The data processing apparatus (100) of claim 2, wherein the one or more kernel tensors (215a-m) comprise a further plurality of kernel weights for at least two of the plurality of time intervals defined by the third dimension of the audio data tensor (210).

4. The data processing apparatus (100) of any one of the preceding claims, wherein the plurality of frequency ranges defined by the first dimension of the audio data tensor (210) comprise a plurality of octaves and the plurality of logarithmic sub-frequencies within the respective frequency range comprise a plurality of pitches within the respective octave.

5. The data processing apparatus (100) of any one of the preceding claims, wherein the processing circuitry (101) is configured to generate the audio data tensor (210) based on the one or more spectra (300) of the audio data by padding and / or trimming the one or more spectra (300) of the audio data.

6. The data processing apparatus (100) of any one of the preceding claims, wherein the one or more kernel tensors (215a-m) comprise a plurality of kernel tensors (215a-m) and wherein each kernel tensor (215a-m) has a different dilation factor for the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor (210).

7. The data processing apparatus (100) of claim 6, wherein the processing circuitry (101) is further configured to combine the one or more pluralities of output values into an output value tensor (230).

8. The data processing apparatus (100) of claim 6 or 7, wherein at least one kernel tensor (215a-m) of the plurality of kernel tensors (215a-m) has a dilation factor for the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor (210) corresponding to a perfect fifth music interval.

9. The data processing apparatus (100) of any one of claims 6 to 8, wherein at least one kernel tensor (215a-m) of the plurality of kernel tensors (215a-m) has a dilation factor for the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor (210) corresponding to a major third music interval.

10. The data processing apparatus (100) of any one of claims 6 to 9, wherein at least one kernel tensor (215a-m) of the plurality of kernel tensors (215a-m) has a dilation factor for the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor (210) corresponding to a minor seventh music interval.11 . The data processing apparatus (100) of any one of the preceding claims, wherein the data processing apparatus (100) further comprises a communication interface (103) configured to receive the audio data and / or the one or more spectra (300) of the audio data.

12. The data processing apparatus (100) of any one of the preceding claims, wherein the data processing apparatus (100) comprises one or more microphones (106) configured to acquire the audio data.

13. The data processing apparatus (100) of any one of the preceding claims, wherein the processing circuitry (101) is further configured to generate the one or more spectra (300) of the audio data based on the audio data.

14. A data processing method (1000) for processing audio data, wherein the data processing method (1000) comprises: generating (1001) based on one or more spectra (300) of the audio data an audio data tensor (210), wherein the audio data tensor (210) comprises a plurality of spectral intensity values for a plurality of frequency ranges defined by a first dimension of the audio data tensor (210) and for a plurality of logarithmic sub-frequencies within the respective frequency range defined by a second dimension of the audio data tensor (210); and implementing (1003) a neural network with at least one convolutional layer configured to generate a one or more pluralities of output values (220) by convolving the plurality of spectral intensity values of the audio data tensor (210) with one or more kernel tensors (215a-m), wherein the one or more kernel tensors (215a-m) comprise a plurality of kernel weights for at least two of the plurality of frequency ranges defined by the first dimension of the audio data tensor (210) and for at least two of the plurality of logarithmic sub-frequencies defined by the second dimension of the audio data tensor (210).

15. A computer program product comprising a computer-readable storage medium for storing program code which causes a computer or a processor to perform the method (1000) of claim 14, when the program code is executed by the computer or the processor.

Images