Systems and methods for training a transformer-in-transformer based neural network model for audio data

Abstract

Disclosed herein are devices, systems, and methods related to causing a device to generate music information for audio data using a transformer-based neural network model having multiple levels of transformers for audio analysis using a spectral and temporal transformer. The processor generates a time-frequency representation of obtained audio data to be applied as input to the transformer-based neural network model; determines a spectral embedding and a first temporal embedding of the audio data based on the time-frequency representation of the audio data; determines each vector of a second frequency class tag (FCT) by passing each vector of the first FCT in the spectral embedding through a spectral transformer; determines a second temporal embedding by adding a linear projection of the second FCT to the first temporal embedding; determines a third temporal embedding by passing the second temporal embedding through a temporal transformer; and generates the music information based on the third temporal embedding.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Application No. 17 / 502,863, filed October 15, 2021, entitled "System and method for training a transformer-in-transformer based neural network model for audio data," the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to machine learning, and more particularly to machine learning methods and systems based on transformer architectures. Background Technology

[0004] In the field of machine learning, the transformers disclosed by A. Vaswani et al. in their paper "Attention is All You Need," presented at the 31st Conference on Neural Information Processing Systems (December 6, 2017), are used in fields such as natural language processing and computer vision. More recently, K. Han et al. proposed a transformer-in-transformer (TNT) architecture in the arXiv preprint arXiv:2103.00112,2021 (July 5, 2021), "Transformer in transformer," where local and global information are modeled, allowing sentence position encoding to maintain global spatial information while word position encoding preserves local relative positions. However, such multi-level transformer architectures have not yet been proposed or developed for applications such as music information retrieval for audio data recognition. Similarly, further development is needed in the field of transformers for audio data recognition. Summary of the Invention

[0005] This document discloses apparatus, systems, and methods related to generating music information from audio data using a transformer-based neural network model, the neural network model having a multi-level transformer for audio analysis, employing a spectral transformer and a time transformer. For example, the apparatus or method implemented using the apparatus may include at least one processor and at least one memory, the memory including computer program code for one or more programs, the memory and computer program code being configured, together with the processor, to enable the apparatus to train the transformer-based neural network model. The apparatus may be configured to train a multi-level transformer.

[0006] In some examples, the apparatus includes at least one processor and a non-transitory computer-readable medium storing computer program code therein, the computer program code including instructions for one or more programs that, when executed by the processor, cause the processor to perform the following steps: obtaining audio data; generating a time-frequency representation of the audio data to be applied as input to a transformer-based neural network model, the transformer-based neural network model including a transformer-in-transformer module including a spectrum transformer and a time transformer; determining a spectral embedding and a first time embedding of the audio data based on the time-frequency representation of the audio data, the spectral embedding including a first frequency category label (FCT); determining each vector of a second FCT by passing each vector of the first FCT in the spectral embedding through the spectrum transformer; determining a second time embedding by adding a linear projection of the second FCT to the first time embedding; determining a third time embedding by passing the second time embedding through the time transformer; and generating musical information of the audio data based on the third time embedding.

[0007] In some examples, the spectral embedding is determined by generating a first FCT to include at least one spectral feature from a frequency range and a frequency position code (FPE) to include at least one frequency position of the first FCT. In some examples, each of the spectrum transformer and time transformer includes multiple encoder layers, each encoder layer including a multi-head self-attention module, a feedforward network module, and a layer normalization module. In some examples, each of the spectrum transformer and time transformer includes multiple decoder layers configured to receive the output from one of the encoder layers, each decoder layer including a multi-head self-attention module, a feedforward network module, a layer normalization module, and an encoder-decoder attention module.

[0008] In some examples, the spectral embeddings are matrices with matrix dimensions determined based on the number of frequency bins and channels used by the converter-in-converter module, and the number of spectral embeddings is determined by the number of time steps used by the converter-in-converter module. In some examples, the temporal embeddings are vectors with a length determined by the number of features used by the converter-in-converter module, and the number of temporal embeddings is determined by the number of time steps used by the converter-in-converter module.

[0009] In some examples, the transformer-based neural network model includes multiple transformer-in-transformer modules arranged in a stacked configuration, such that the temporal embedding is updated through each of the multiple transformer-in-transformer modules. In some examples, the spectrum transformer and the time transformer are arranged hierarchically, such that the spectrum transformer is configured to generate local musical information of the audio data, and the time transformer is configured to generate global musical information of the audio data.

[0010] According to another embodiment, a method implemented by at least one processor is disclosed, wherein the method includes the following steps: obtaining audio data; generating a time-frequency representation of the audio data, the audio data to be applied as input to a transformer-based neural network model, the transformer-based neural network model including a transformer-in-transformer module, the module including a spectrum transformer and a time transformer; determining a spectrum embedding and a first time embedding of the audio data based on the time-frequency representation of the audio data, the spectrum embedding including a first frequency category label (FCT); determining each vector of a second FCT by passing each vector of the first FCT in the spectrum embedding through the spectrum transformer; determining a second time embedding by adding a linear projection of the second FCT to the first time embedding; determining a third time embedding by passing the second time embedding through the time transformer; and generating music information of the audio data based on the third time embedding.

[0011] In some examples, the method further includes the steps of determining the spectral embedding by generating a first FCT to include at least one spectral feature in a frequency range and generating a frequency position code (FPE) to include at least one frequency position of the first FCT. In some examples, each of the spectrum transformer and time transformer includes multiple encoder layers, each encoder layer including a multi-head self-attention module, a feedforward network module, and a layer normalization module. In some examples, each of the spectrum transformer and time transformer includes multiple decoder layers configured to receive output from one of the encoder layers, each decoder layer including a multi-head self-attention module, a feedforward network module, a layer normalization module, and an encoder-decoder attention module.

[0012] In some examples, the spectral embeddings are matrices with matrix dimensions determined based on the number of frequency ranges and channels used by the converter-in-converter module, and the number of spectral embeddings is determined by the number of time steps used by the converter-in-converter module. In some examples, the temporal embeddings are vectors with a length determined by the number of features used by the converter-in-converter module, and the number of temporal embeddings is determined by the number of time steps used by the converter-in-converter module.

[0013] In some examples, the transformer-based neural network model includes multiple transformer-in-transformer modules arranged in a stacked configuration, such that the temporal embedding is updated through each of the multiple transformer-in-transformer modules. In some examples, the spectrum transformer and the time transformer are arranged hierarchically, such that the spectrum transformer is configured to generate local musical information of the audio data, and the time transformer is configured to generate global musical information of the audio data. Attached Figure Description

[0014] These embodiments will be more readily understood in conjunction with the following figures and in light of the following description, wherein the same reference numerals denote the same elements, and in the figures:

[0015] Figure 1 A block diagram of an exemplary transformer-based neural network model based on the examples disclosed herein is shown.

[0016] Figure 2 A block diagram of an exemplary transformer-based neural network model based on the examples disclosed herein is shown.

[0017] Figure 3 A block diagram of an exemplary location-encoded block based on the examples disclosed herein is shown.

[0018] Figure 4 A block diagram of an exemplary spectrum-time converter-in-converter block according to the examples disclosed herein is shown.

[0019] Figure 5 A data flow diagram of each layer of an exemplary spectrum-time converter-in-converter block is shown, based on the examples disclosed herein.

[0020] Figure 6 A block diagram of an exemplary computing device and database for implementing a transformer-based neural network model is shown, based on examples disclosed herein.

[0021] Figure 7 Block diagrams of exemplary spectrum converter blocks and time converter blocks according to the examples disclosed herein are shown.

[0022] Figure 8 A flowchart is shown illustrating an exemplary method for implementing a transformer-based neural network model according to the examples disclosed herein. Detailed Implementation

[0023] In short, the system and method comprise a transformer-in-transformer (TNT) architecture that implements a spectral transformer that extracts frequency-related features into the frequency category markers (FCTs) of each frame of audio data. The FCTs are then linearly projected and added to a temporal embedding that aggregates useful information from the FCTs. The TNT architecture also implements a temporal transformer that processes the temporal embeddings to exchange information across the time axis. This architecture, which implements both the spectral and temporal transformers, is referred to herein as spectral-temporal TNT. Multiple such TNT blocks can be stacked to construct a spectral-temporal TNT model architecture, thereby learning representations of audio data such as music signals to perform tasks such as music information retrieval (MIR) research and analysis, including but not limited to music tagging, vocal melody extraction, and chord recognition.

[0024] In MIR analysis, the time axis is represented as the sequence axis, and the frequency axis is represented as the characteristic axis. (Reference) Figure 1 Based on the examples disclosed herein, an exemplary transformer-based neural network model 100 is illustrated. For example, audio data such as music clips, audio signals, and / or speech recordings are input via input block 102. Time-frequency representation block 104 is any suitable module, such as a microprocessor, processor, state machine, etc., capable of generating a time-frequency representation of the audio data (also known as the input time-frequency representation), which is a view of the audio signal represented in time and frequency, as known in the art. Convolution block 106 is any suitable module capable of processing the input time-frequency representation using a stack of convolutional layers for local feature aggregation, as known in the art.

[0025] The position encoding block 108 is any suitable module that can add positional information to the input time-frequency representation after processing by the convolutional block 106. About Figure 2 and Figure 3 The details of how location information is added are explained. The resulting data, i.e., the input time-frequency representation with location information added, is fed into the stack of the spectrum-time TNT block 110 or such TNT block. (Regarding...) Figure 4 , Figure 5 and Figure 8 The details of how each spectrum-time TNT block processes the data are explained. Output block 112 is any suitable module that will project the final embedding into the desired dimension for different tasks.

[0026] Figure 2 The diagram shows... Figure 1The data flow between blocks is introduced, and more specifically, the functionality of the position encoding block 108 according to an example of the neural network model 100 disclosed herein is illustrated. Initially, raw audio data (“audio data”) is input into the time-frequency representation block 104 to generate an input time-frequency representation (S). Representation S is referred to as... The matrix is ​​denoted by , where S is a three-dimensional matrix with dimensions T, F, and K, where T is the number of time steps, F is the number of frequency intervals, and K is the number of channels. This indicates that S is passed to the stack of convolutional layers in convolutional block 106, such that the representation after convolutional block 106 can be referred to as . Where T', F', and K' are the frequency range, time step, and number of channels, respectively.

[0027] about Figure 2 and Figure 3 , Figure 2 and Figure 3 Not only does it illustrate the data flow within position coding block 108, but it also illustrates the dimensions of each vector or matrix generated within position coding block 108, and the Frequency Class Label (FCT, also represented as c). t ) is a learnable embedding vector initialized with all zeros to serve as placeholders, and is defined as c t =0 1×K’ That is, a zero vector of dimension K'. For each time step, the FCT vector is generated by FCT generation block 200 based on the determined K' value. The input data at each time step t is referred to as... Furthermore, each element in the FCT vector is concatenated with the input data using concatenation operator 204 at the matching time step, i.e., S” t =Concat[c t ,S' t ], where S” refers to the FCT concatenation representation of S'. This concatenation will c t Each element in the vector is applied to the corresponding S' t At the end of the matrix, this changes the dimension of the matrix, resulting in S”. t The matrix has dimensions F'+1 multiplied by K'.

[0028] Frequency Position Embedding (FPE, also denoted as E) φ The FPE matrix is ​​a learnable matrix used to apply frequency position encoding to the representation and is generated by FPE generation block 202. The FPE matrix is... Refers to the element-wise adder 206 using "S". t and E φ Implement element-wise addition, where the result is referred to as (in (Referring to element-wise addition). This refers to the combined three-dimensional matrix of all time steps t, i.e., a matrix with dimensions T', F'+1, and K'. This is the output of position-coded block 108. In the resulting representation matrix... In the example, the FCT vector is composed of... Common reference, which allows for the representation of matrices Information such as pitch and timbre from audio data is carried over to subsequent attention layers. For example, pitch in a signal can cause high energy in a specific frequency range, and positional encoding makes each element in the FCT vector aware of its frequency location.

[0029] Figure 4 The diagram illustrates the encoded portion of an exemplary spectral-temporal TNT block 110 according to the examples disclosed herein. TNT block 110 includes two data streams: a temporal embedding 400 and a spectral embedding 402. These two data streams are processed by two transformer encoders, or more specifically, the temporal embedding 400 is processed by a temporal transformer encoder 414, and the spectral embedding is processed by a spectral transformer encoder 408. Linear projection blocks (or layers) 404 and 410 act as a “bridge” between the two data streams, and the temporal embedding 400 also includes an adder 412. The spectral embedding 402 also includes another adder 406. In the following description of TNT block 110, the symbol 'l' is introduced to specify the layer indexes for the two embeddings.

[0030] Regarding the time-embedded 400 data stream, E l Used to refer to the temporal embedding matrix, which is a combination of individual temporal embedding vectors at layer l, such that E l =[e l 1,e l 2,…,e l T’ ],in That is, each e l t E is a temporal embedding vector with dimension D at time t, where D is the number of features. l It is a learnable temporal embedding matrix, which is randomly initialized before entering the first spectral-temporal TNT block. As the time embedding matrix passes through each subsequent layer, the learnable matrix E l It has been gradually improved.

[0031] In the following example, the FCT vector is located in the first frequency range of the spectral embedding matrix, i.e. Initialization of the first spectrum-time TNT block Matrix (or ) is the output obtained from position-coded block 108, previously in Figure 3China is referred to as As mentioned above, spectral embedding includes FCT vectors, which help to aggregate useful local spectral information. As a general notation, It can be written as: Where l = 0, 1, ..., L; It is the FCT vector of the l-th layer at time step t; and This is the spectral data at time step t. The spectral embedding can then interact with the temporal embedding via the FCT vector, thus allowing local spectral features to be processed in a temporal, global manner.

[0032] For example, the learnable matrix E l-1 The time embedding vector, i.e., e l-1 1,e l-1 2,…,e l-1 T’ Each vector in the vector is transformed from dimension d to dimension K' by a linear projection layer 404. This allows the projection vector of dimension K' to be combined with the spectral embedding matrix using an adder 406. The first frequency interval is summed, which is the location of the FCT vector. The result of adding the projection vector to the spectral embedding matrix is ​​referred to as... The resulting matrix The input is fed into the spectrum converter encoder 408, which outputs a matrix. This matrix can be used as the input spectral embedding for the next layer.

[0033] Then, output the matrix. After passing through the linear projection layer 410, the transformed output matrix of the linear projection layer... Each of the FCT vectors, i.e., located in the output spectral embedding matrix The vector in the first frequency interval is transformed from dimension K' to D. Then, adder 412 is used to combine the linearly projected FCT vector with the time embedding vector e. l-1 1,e l-1 2,…,e l-1 T’ Addition. The vectors to be added (e) l 1,e l 2,…,e l T’ The input is fed into the time converter encoder 414 to obtain matrix E. l This matrix can be used for the input temporal embedding of the next layer.

[0034] Figure 5The diagram illustrates the components and data flow from one transformer layer (l-1) to the next layer (l) in each layer of the transformer encoder 500. In the following text, X is used to represent the temporal or spectral embedding. The transformer encoder 500 includes a layer normalization (LN) component or module 506, a multi-head self-attention (MHSA) component or module 508, a feedforward network (FFN) component or module 510, and two adders 502 and 504. Self-attention requires three inputs: Q (query), K (key), and V (value). These inputs are defined as matrices with the following properties: and Where T is the number of time steps, and d q It is the number of features of Q, d k d is the number of features of K, and d v This is the number of features of V. The output is a weighted sum of similarity values ​​between the query and the keyword at the corresponding time step, as defined below:

[0035]

[0036] The MHSA module 508 is an extension of self-attention, whereby the three inputs Q, K, and V are divided into h heads along their feature dimensions, and multiple self-attention operations are performed in parallel, each on one of the heads. The outputs of the heads are then concatenated and linearly projected onto the final output. The FFN module 510 has two linear layers with a Gaussian Error Linear Unit (GELU) activation function between them. In some examples, pre-normalized residual units are also implemented to stabilize the training of the model.

[0037] Generally speaking, the operation of the converter encoder 500 causes X l =Enc(X l-1 The Enc(·) operation is performed as follows. In the first part of encoder 500, the temporal embedding matrix or vector X... l-1 The matrix or vector X passes through layer normalization module 506 and then through multi-head self-attention module 508. The resulting matrix or vector from multi-head self-attention module 508 is added to the original matrix or vector X. l-1 The result can be referred to as X' l-1 In the next part of encoder 500, the resulting matrix or vector X' l-1 After passing through the layer normalization module 506, and subsequently the feedforward network module 510, the resulting matrix or vector from the feedforward network module 510 is then added to the original matrix or vector X'. l-1 And the final result is expressed as a vector or matrix X. l The output is in the form of a function, so that it can be input into the next transformer layer.

[0038] In some examples, multiple spectrum-time TNT blocks 110 are stacked to form a spectrum-time TNT module. For example, there may be three TNT blocks 110 in one such TNT module. The module can be started from the initial spectrum embedding matrix of the first TNT block. and the initial time embedding matrix E 0 Begin. (As...) Figure 4 As shown, there are four steps for each TNT block.

[0039] In the first step, a relevant temporal embedding vector e is added by using a linear projection layer 404. l-1 t Linear projection, update FCT vector in Each of them. This operation is performed by It means that, among them It comes from the previous FCT vector The updated FCT vector, and the Liner(·) operation represents the shared linear layer, i.e., the linear projection layer 404.

[0040] In the second step, the spectral embedding matrix is... After passing through the spectrum converter encoder 408, it is defined as The spectral embedding matrix includes the updated FCT vectors ranging from t=1 to t=T' in the first frequency interval or first row.

[0041] In the third step, FCT vector in Each element is linearly projected and added back to the corresponding temporal embedding vector. Make in Refers to the time embedding matrix being updated The updated temporal embedding vector in the middle.

[0042] Finally, in the fourth step, the updated temporal embedding matrix is... Instead of the time embedding matrix E l-1 and spectral embedding matrix The sum, then updated using the time transformer encoder 414, is represented by the TempEnc() function, making This operation helps establish relationships along the time axis, thus benefiting the performance of transformer-based neural network models by reducing the number of parameters. Furthermore, the time transformer does not need to access information for every frequency range, but only the important frequency ranges within each spectral embedding matrix that are focused by the FCT vector.

[0043] Output block 112 receives the final output of TNT block 110, referred to as E. 3 , which is the temporal embedding matrix from the third TNT block, which is the final TNT block in the TNT module. Although the number three (3) is depicted, it should be understood that, depending on the amount of data to be learned, there may be any suitable number of TNT blocks, such as more or less than three TNT blocks.

[0044] Depending on the task performed using this type of output, output block 112 may require different outputs. For example, in frame-by-frame prediction tasks such as vocal melody extraction and chord recognition, each temporal embedding vector e 3 t It is fed into a shared fully connected layer with a sigmoid or SoftMax function for the final output. For example, in a song prediction task such as music tagging, the output block 112 is the startup-time category tag vector ε. l , where l = 0, and it is in E l Connect them at the front end to form another matrix Make Note the time category label vector ε l There is no associated FCT vector in the spectral embedding matrix because the time category label vector ε l The operation is used to aggregate the time embedding vectors along the time axis. Finally, ε represents the time category tag vector after the third TNT block. 3 The vector is fed into a fully connected layer, followed by a sigmoid layer, to obtain the probability output.

[0045] Figure 6 An exemplary computing system 600 implementing the spectrum-time TNT block disclosed herein is illustrated. System 600 includes a computing device 602, such as a computer or intelligent device capable of performing the computations required to train a TNT-based neural network model for audio data. The computing device 602 has a processor 604 and a memory unit 606, and is also operably coupled to a database 616, such as a remote data server, via a connector 614, which includes wired or wireless data communication equipment, such as a cloud network for cloud computing capabilities.

[0046] In processor 604, there are modules capable of executing each of the previously disclosed blocks 102, 104, 106, 108, 110, and 112. These modules may be implemented in a computer program, software, or firmware incorporated into a non-transitory computer-readable storage medium, such as memory unit 606, for execution by processor 604. Furthermore, in each spectrum TNT block 110, there are spectrum converter blocks 608, time converter blocks 610, and linear projection blocks 612, such that multiple spectrum TNT blocks 110 may include multiple individually operable spectrum converters 608, time converters 610, and linear projection blocks 612 to implement the multi-stage converter architecture disclosed herein.

[0047] Figure 7 The exemplary spectrum converter block 608 and exemplary time converter block 610 disclosed herein are illustrated. As previously stated, each converter has multiple encoders and multiple decoders. In this figure, for simplicity, only one of each encoder and decoder is shown; however, it should be understood that such encoders and decoders can be distributed within the converter in any suitable configuration, such as serial or parallel distribution, as known in the art. For example, each encoder 408 of the spectrum converter 608 includes an implementation... Figure 5 The multi-head self-attention block 508, feedforward network block 510, and layer normalization block 506 required for the data stream shown are also implemented in each encoder 414 of the time transformer 610 to implement the same blocks.

[0048] The decoder 700 of the spectrum converter block 608 and the decoder 702 of the time converter block 610 also have similar component blocks, mainly multi-head self-attention block 508, feedforward network block 510, layer normalization block 506 and encoder-decoder attention block 704, which helps decoder 700 or 702 to pay attention to the appropriate matrix of each encoder output.

[0049] Figure 8An exemplary method or process 800 is illustrated, in which a processor, when implementing the spectrum-time TNT block disclosed herein, follows this method or process to perform audio data analysis and processing using a TNT-based neural network model to obtain information about the audio data (e.g., music information or sound identification information), as explained herein. In step 802, the processor obtains audio data to be analyzed and processed. In step 804, the processor generates a time-frequency representation of the audio data to be applied as input to a transformer-based neural network model. The transformer-based neural network model includes a transformer-in-transformer module that includes a spectrum transformer and a time transformer as disclosed herein. In step 806, the processor determines a spectrum embedding and a first time embedding of the audio data based on the time-frequency representation of the audio data. The spectrum embedding includes a first frequency class label (FCT).

[0050] In step 808, the processor determines each vector of the second FCT by passing each vector of the first FCT in the spectral embedding through a spectral transformer. In step 810, the processor determines the second temporal embedding by adding a linear projection of the second FCT to the first temporal embedding. In step 812, the processor determines the third temporal embedding by passing the second temporal embedding through a temporal transformer. In step 814, the processor generates music information of the audio data based on the third temporal embedding.

[0051] In some examples, method 800 may involve data flow within a single spectral TNT block, and it should be understood that a TNT-based neural network model may have multiple such TNT blocks that are functionally coupled or stacked together, for example, serially, such that the output of the first TNT block is used as the input to subsequent TNT blocks to improve the efficiency and power of training the model based on the training dataset in the database.

[0052] In some examples, each of the spectrum transformer and time transformer includes multiple encoder layers, each encoder layer including a multi-head self-attention module, a feedforward network module, and a layer normalization module. Each of the spectrum transformer and time transformer may also include multiple decoder layers configured to receive an output matrix from one of the encoder layers, each decoder layer including a multi-head self-attention module, a feedforward network module, a layer normalization module, and an encoder-decoder attention module.

[0053] Additional steps may be implemented in the method 800 disclosed herein. For example, the processor may determine the dimension of the spectral embedding matrix based on the number of frequency ranges and channels used by the multi-stage converter, and further determine the number of spectral embedding matrices based on the number of time steps used by the multi-stage converter. For example, the processor may determine the vector length of the temporal embedding vector based on the number of features used by the multi-stage converter, and further determine the number of temporal embedding vectors based on the number of time steps used by the multi-stage converter. The spectral converter and the temporal converter may be arranged hierarchically, such that the spectral (lower-level) converter learns local information of the audio data, while the temporal (higher-level) converter learns global information of the audio data.

[0054] In some examples, the positional coding block is operatively coupled to a multi-stage transformer such that the concatenator of the positional coding block concatenates the FCT vector with the convolutional time-frequency representation of the audio data, and the element-wise adder of the positional coding block adds the FPE matrix with the convolutional time-frequency representation of the audio data.

[0055] Using multi-stage transformers to implement such methods or processing devices for training transformer-based neural network models offers numerous advantages. For example, multi-stage transformers can learn representations of audio data such as music or vocal signals and exhibit improved performance in music tagging, vocal melody extraction, and chord recognition. In some examples, because multi-stage transformers are configured such that only important local information is passed to the time transformer via the FCT, they can learn more efficient models using smaller datasets, significantly reducing the dimensionality of the data stream compared to other transformer-based models known in the art for learning audio data. This reduction in data stream dimensionality, due to the reduced workload, facilitates more efficient machine learning.

[0056] Although the features and elements have been described above in specific combinations, each feature or element may be used alone without other features and elements, or in various combinations with or without other features and elements. The methods provided can be implemented in general-purpose computers, processors, or processor cores. Suitable processors, for example, include general-purpose processors, special-purpose processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate array (FPGA) circuits, any other type of integrated circuit (IC), and / or state machines. Such processors can be manufactured by configuring a manufacturing process using processed hardware description language (HDL) instructions and other intermediate data, including netlists (such instructions can be stored on a computer-readable medium). The result of such processing can be a mask, which is then used in a semiconductor manufacturing process to manufacture processors of various aspects of the implementation examples.

[0057] The methods or flowcharts provided herein can be implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROMs and digital versatile discs (DVDs).

[0058] In the foregoing detailed description of the various examples, reference has been made to the accompanying drawings, which form a part thereof, illustrating specific preferred examples in which the invention may be practiced. These examples have been described in sufficient detail to enable those skilled in the art to practice the invention, and it should be understood that other examples may be utilized, and logical, mechanical, and electrical changes may be made without departing from the scope of the invention. Furthermore, those skilled in the art can readily construct examples incorporating many other variations of the teachings of this disclosure. Therefore, the invention is not intended to be limited to the specific forms set forth herein, but rather, it is intended to cover such substitutions, modifications, and equivalents that are reasonably included within the scope of the invention. Accordingly, the foregoing detailed description should not be considered limiting, and the scope of the invention is defined only by the appended claims. The detailed description of the embodiments and examples described therein is for illustrative and descriptive purposes only and not for limiting purposes. For example, the described operations are performed in any suitable order or manner. Therefore, it is contemplated that the invention covers any and all modifications, variations, or equivalents falling within the scope of the basic principles disclosed above and claimed herein.

[0059] The detailed descriptions and examples described above are for illustrative and descriptive purposes only, and not for limiting purposes.

Claims

1. An apparatus for generating music information, comprising: At least one processor and a non-transitory computer-readable medium therein storing computer program code, the computer program code including instructions for one or more programs, the instructions, when executed by the processor, causing the processor to: Obtain audio data; The audio data is used to generate a time-frequency representation of the audio data, which is to be used as input to a transformer-based neural network model. The transformer-based neural network model includes a transformer-in-transformer module, which includes a spectrum transformer and a time transformer. Each of the spectrum transformer and the time transformer includes multiple encoder layers and multiple decoder layers, respectively. Based on the time-frequency representation of the audio data, a spectral embedding and a first time embedding of the audio data are determined, wherein the spectral embedding includes a first frequency category label (FCT). Each vector of the second FCT is determined by passing each vector of the first FCT in the spectrum embedding through the spectrum transformer; The second temporal embedding is determined by adding a linear projection of the second FCT to the first temporal embedding; The third time embedding is determined by passing the second time embedding through the time transformer; as well as Music information is generated based on the third time embedding of the audio data.

2. The apparatus of claim 1, wherein the spectral embedding is determined by generating at least one spectral feature from a frequency range and a frequency position code (FPE) to include at least one frequency position of the first FCT.

3. The apparatus of claim 1, wherein each encoder layer comprises a multi-head self-attention module, a feedforward network module, and a layer normalization module.

4. The apparatus of claim 3, wherein the plurality of decoder layers are configured to receive output from one of the encoder layers, each decoder layer including a multi-head self-attention module, a feedforward network module, a layer normalization module, and an encoder-decoder attention module.

5. The apparatus of claim 1, wherein the spectrum embedding is a matrix having a matrix dimension determined based on the number of frequency ranges and channels used by the converter-in-converter module, and the number of spectrum embeddings is determined by the number of time steps used by the converter-in-converter module.

6. The apparatus of claim 1, wherein the time embedding is a vector having a vector length determined by the number of features employed by the converter-in-converter module, and the number of time embeddings is determined by the number of time steps employed by the converter-in-converter module.

7. The apparatus of claim 1, wherein the transformer-based neural network model comprises a plurality of transformer-in-transformer modules configured in a stacked manner, such that the temporal embedding is updated by each of the plurality of transformer-in-transformer modules.

8. The apparatus of claim 1, wherein the spectrum converter and the time converter are arranged in layers such that the spectrum converter is configured to generate local music information of the audio data, and the time converter is configured to generate global music information of the audio data.

9. A method implemented by at least one processor, comprising: Obtain audio data; The audio data is used to generate a time-frequency representation of audio data, which is then applied as input to a transformer-based neural network model. The transformer-based neural network model includes a transformer-in-transformer module, which includes a spectrum transformer and a time transformer. Each of the spectrum transformer and the time transformer includes multiple encoder layers and multiple decoder layers, respectively. Based on the time-frequency representation of the audio data, a spectral embedding and a first time embedding of the audio data are determined, wherein the spectral embedding includes a first frequency category label (FCT). Each vector of the second FCT is determined by passing each vector of the first FCT in the spectrum embedding through the spectrum transformer; The second temporal embedding is determined by adding a linear projection of the second FCT to the first temporal embedding; The third time embedding is determined by passing the second time embedding through the time transformer; as well as Music information is generated based on the third time embedding of the audio data.

10. The method of claim 9, further comprising: The spectral embedding is determined by generating the first FCT to include at least one spectral feature from a frequency range and generating a frequency position code (FPE) to include at least one frequency position of the first FCT.

11. The method of claim 9, wherein each encoder layer comprises a multi-head self-attention module, a feedforward network module, and a layer normalization module.

12. The method of claim 11, wherein the plurality of decoder layers are configured to receive the output of one of the encoder layers, each decoder layer including a multi-head self-attention module, a feedforward network module, a layer normalization module, and an encoder-decoder attention module.

13. The method of claim 9, wherein the spectrum embedding is a matrix having a matrix dimension determined based on the number of frequency ranges and channels used by the converter-in-converter module, and the number of spectrum embeddings is determined by the number of time steps used by the converter-in-converter module.

14. The method of claim 9, wherein the time embedding is a vector having a vector length determined by the number of features employed by the converter-in-converter module, and the number of time embeddings is determined by the number of time steps employed by the converter-in-converter module.

15. The method of claim 9, wherein the transformer-based neural network model comprises a plurality of transformer-in-transformer modules configured in a stacked manner, such that the temporal embedding is updated through each of the plurality of transformer-in-transformer modules.

16. The method of claim 9, wherein the spectrum converter and the time converter are arranged in layers such that the spectrum converter is configured to generate local music information of the audio data, and the time converter is configured to generate global music information of the audio data.

Images