Multi-instrument identification method and system based on time convolution network

Abstract

The application belongs to the technical field of music information retrieval, and specifically discloses a multi-instrument identification method and system based on a time convolution network, which comprises the following steps: acquiring an audio data set and performing preprocessing; constructing an identification model based on a convolution neural network and a time convolution network; converting the preprocessed audio data set into a Mel spectrogram and a Wave graph respectively, and inputting the two graphs into the convolution neural network CNN and the time convolution network TCN of the identification model respectively to train the model; collecting audio data to be identified, inputting the preprocessed audio data into the trained identification model, and outputting an audio data identification result. The technical scheme is used to use the time convolution network to improve the performance in the multi-instrument identification task and obtain more accurate and reliable multi-instrument classification results.

Description

Technical Field

[0001] This invention belongs to the field of music information retrieval technology, and relates to a multi-instrument recognition method and system based on temporal convolutional networks. Background Technology

[0002] Since the late 20th century, with the widespread adoption of the internet globally, the rapid dissemination, downloading, and enjoyment of digital music on online platforms has spurred the rise of Music Information Retrieval (MIR) technology. Initially, MIR focused primarily on music search; however, with the evolution of technology, this field has expanded. Today, MIR more broadly encompasses computational methods for in-depth content analysis and understanding of digital music.

[0003] As an interdisciplinary research field, MIR (Music Information Retrieval) is broadly defined as the extraction of information from music and related applications. The concept of music typically involves audio content, and its definition can be extended to include a variety of music-related information such as lyrics, music descriptions, and user listening history.

[0004] Instrument recognition, as a crucial means of acquiring advanced music information, has become a hot research topic in the field of music information retrieval (MIR). The high level of attention and emphasis placed on instrument recognition by academia and the arts reflects its importance in music information retrieval. The core objective of this field is to enable computers to deeply understand and analyze instrument sounds through intelligent computing, thereby achieving accurate identification of different instruments. Further research in this area will not only have a profound impact on music information retrieval but also provide abundant possibilities for the automated processing and innovation of digital music.

[0005] In music information retrieval and related fields, finding suitable audio feature extraction schemes is crucial for instrument recognition systems to operate effectively in diverse and complex musical environments and achieve their intended tasks and goals. This allows for the acquisition of information related to instrument timbre, thereby improving the performance of the instrument recognition system. Currently, common instrument recognition methods are mainly divided into two categories: shallow instrument recognition methods based on machine learning and instrument recognition methods based on deep learning.

[0006] In early research on musical instrument recognition, researchers primarily employed machine learning-based methods such as Support Vector Machines (SVM) and K-Nearest Neighbors (KNN). These methods aimed to identify musical instrument types by modeling and classifying pre-extracted audio features. In 1999, Janet Marques and Pedro J. Moreno proposed a musical instrument classification study based on Gaussian Mixture Model (GMM) and SVM. They used three different feature representations, including Linear Predictive Coefficients (LPC), cepstral frequency, and Mel-Cepstral Frequency, combined with an SVM classifier for instrument classification. In experiments, the overall error rate reached 30%. On the other hand, Krishna proposed Line Spectral Frequencies (LSF) as a feature for musical instrument recognition and evaluated the proposed system using MFCC and LPCC features. They employed Gaussian Mixture Model and K-Nearest Neighbors classifiers for classification. KNN can asymptotically approximate ideal performance and provides arbitrarily complex class boundaries in the feature space, making it a simple and intuitive classifier. However, a drawback of KNN is that all training patterns must be stored for testing, which can impose a computational and storage burden when dealing with large numbers of classes and data.

[0007] With the development of deep learning neural networks, artificial neural networks can be seen as an organic combination of feature extractors and classifiers, capable of directly extracting features from raw audio data or spectrograms and achieving effective recognition, often outperforming shallow machine learning methods. As input to Convolutional Neural Networks (CNNs), spectrograms are obtained by performing Fourier transforms on audio signals, a process that may lead to some information loss. Particularly during the discretization of spectrograms, certain details and temporal information may be lost. Spectrograms lose the phase information of the audio signal, which is crucial for audio processing in some applications. Phase information contains the temporal sequence of the audio signal, while spectrograms only contain amplitude information. Since the characteristics of audio signals may change over time, and spectrograms typically divide the entire audio signal into fixed time segments, this may lead to insufficient modeling of non-stationary audio variations. Therefore, in deep learning applications, the information loss introduced by spectrograms needs careful consideration, especially for audio processing tasks sensitive to temporal and phase information.

[0008] Recent research has explored training with multi-labeled polyphonic music to achieve multi-instrument recognition. While this approach offers new possibilities for expanding instrument recognition tasks, multi-instrument recognition faces numerous challenges. The overlap of instruments during performance and the imbalance between different instrument data make multi-instrument recognition exceptionally difficult. Summary of the Invention

[0009] The purpose of this invention is to provide a multi-instrument recognition method and system based on temporal convolutional networks, which improves the performance of multi-instrument recognition tasks and obtains more accurate and reliable multi-instrument recognition results.

[0010] To achieve the above objectives, the basic solution of this invention is: a multi-instrument recognition method based on temporal convolutional networks, comprising the following steps:

[0011] Obtain the audio dataset and perform preprocessing;

[0012] A recognition model is constructed based on convolutional neural networks and temporal convolutional networks;

[0013] The preprocessed audio dataset is converted into Mel spectrograms and Wave graphs, respectively, and then input into the convolutional neural network (CNN) and temporal convolutional network (TCN) of the recognition model to train the recognition model.

[0014] The system collects audio data to be recognized, preprocesses it, inputs it into the trained recognition model, and outputs the audio data recognition results.

[0015] The working principle and beneficial effects of this basic scheme are as follows: This technical scheme proposes a network structure that integrates TCN and CNN, uses a temporal convolutional network (TCN) to train audio data, focuses on capturing the temporal sequence features of instrument audio, overcomes the challenges of overlap and data imbalance in multi-instrument recognition, so as to more comprehensively reflect the evolution of audio signals over time and provide more accurate and reliable multi-instrument classification performance.

[0016] This invention fully utilizes the modeling capabilities of TCN for temporal relationships, enabling the model to more comprehensively understand and process the interaction relationships between different instruments in polyphonic music. This results in more significant achievements in multi-instrument recognition tasks, overcomes the information loss problem that traditional CNNs may face when processing audio, improves the overall performance of the model, and obtains more accurate and reliable multi-instrument recognition results.

[0017] Furthermore, the method for preprocessing the audio dataset is as follows:

[0018] Remove silence: Cut off the silent portions at the beginning and end of the audio data;

[0019] Two-channel conversion: Stereo audio is converted to mono by taking the average value of the left and right channels;

[0020] Sampling rate adjustment: Convert the audio sampling rate to 22050Hz.

[0021] The audio data is preprocessed to ensure that the audio data input to the model has a consistent format and quality.

[0022] Furthermore, the recognition model includes a convolutional neural network, a temporal convolutional network, and a feature vector fusion layer, a global average pooling layer, and a fully connected layer connected in sequence.

[0023] The outputs of the convolutional neural network and the temporal convolutional network are connected to the feature vector fusion layer, and after passing through the global average pooling layer and the fully connected layer, they are input to the classification layer and subjected to a DropOut operation of 0.3.

[0024] The recognition model combines TCN and CNN, has a simple structure, and can learn features and patterns in the data to improve its performance in classifying audio data or other tasks.

[0025] Furthermore, the preprocessed audio dataset is converted into a Mel spectrogram, and the specific steps are as follows:

[0026] In the preprocessed audio dataset, audio signal samples are truncated to a preset number of seconds and then processed into frames, where the frame shift is a preset number of sampling points.

[0027] The audio sampling rate is 22.05kHz, therefore, the extracted audio samples yield N = 130 frames. Mel spectrogram extraction is performed on each frame, specifically as follows:

[0028] The original waveform of the audio data is converted into the amplitude of each frequency component by fast Fourier transform, and then the frequency is mapped to the Mel frequency.

[0029] The frequency is input into a Mel filter bank that mimics the response of the human auditory system, the frequency is converted into a logarithmic scale, and the amplitude is converted into decibels to generate a Mel spectrum, the length of which is selected.

[0030] The audio data, originally in Hz frequency, is converted to mel frequency. The conversion relationship is as follows:

[0031]

[0032] Where f is the Hz frequency and M(f) is the Mel frequency.

[0033] Mel spectrograms better reflect the nonlinear characteristics of the human auditory system. Converting preprocessed audio datasets into Mel spectrograms facilitates subsequent use.

[0034] Furthermore, the convolutional neural network includes three residual convolutional blocks, each of which includes a first convolutional layer, a first batch normalization (BN) layer, a second convolutional layer, a second batch normalization (BN) layer, and a max pooling layer connected in sequence.

[0035] The max pooling layer uses a 3×3 receptive field and a filter with a fixed step size of 1. It uses a 3×3 max pooling layer with a step size of 1 to process data through spatial abstraction.

[0036] The number of convolutional layer filters in the three residual convolutional blocks increases in multiples of 2, that is, the number of filters in each residual convolutional block is 32, 64, and 128, respectively. The change in the number of filters helps to gradually improve the network's ability to learn more abstract and complex features.

[0037] The Mel spectrogram is input into a convolutional neural network, and convolution yields the feature F. cnn .

[0038] Convolutional neural networks have a simple structure and can learn the frequency domain features of audio data.

[0039] Furthermore, the temporal convolutional network includes three temporal blocks, each of which includes two dilated convolutional layers, two weight normalization layers, a ReLU activation layer, and a dropout layer connected in sequence.

[0040] The Wave graph of the audio data is sequentially processed through three temporal convolutional blocks. Within each temporal convolutional block, the Wave graph is processed by two dilated convolutional layers and a weight normalization layer. The result is then added to the original input data, followed by a ReLU activation layer, a dropout layer, and finally, the output feature F of the temporal convolutional block. tcn ;

[0041] F tcn and the features F output by the convolutional neural network cnn Feature fusion is performed on the channels to obtain the final output feature F.

[0042] The design of temporal convolutional networks focuses on modeling the temporal relationship of input data. By directly utilizing audio waveforms, it can more comprehensively capture the temporal characteristics of audio signals and emphasize the temporal information in audio data.

[0043] Multi-label tasks typically involve features at different time scales because different labels may evolve at different rates over time. Dilated convolutions in TCNs allow the network to effectively learn features at different time scales, thus better adapting to the temporal patterns of multi-label tasks.

[0044] Furthermore, the method for optimizing temporal convolutional networks is as follows:

[0045] For a one-dimensional sequence input x∈R n Given a convolution kernel f:{0,...,k-1}→R, the dilation convolution operation on the elements s of the sequence is defined as:

[0046]

[0047] Where d represents the dilation factor, k represents the scale of the convolution kernel, and sd·i represents the direction in which the learning process can understand and capture data trends and patterns from past time points; R n This vector contains n elements drawn from the real number field; i represents the number of hidden layers; DC(S) represents the result of dilated convolution of element s; (x* d f)(s) represents the convolution of input x and function df with expansion factor d at position s, and f(i) represents the value of function f at position i;

[0048] When using dilated convolution, the dilation factor increases exponentially with increasing network depth;

[0049] A temporal convolutional network contains a branch with a series of transformations F, including dilated causal convolution, weight normalization, ReLU activation, and other processes. The output of this branch is the sequence input x∈R. n Add them together:

[0050] o = Activation(x + F(x))

[0051] Where Activation is the activation function, o represents the output after passing through the activation function, and F(X) represents the output obtained after processing the input x by the F transform;

[0052] This enables each hidden layer to effectively learn the local transformations of equivalent mappings without having to directly model the entire transformation process.

[0053] The increased receptive field allows the network to better review the past, thus learning information in the time dimension more comprehensively.

[0054] The present invention also provides a multi-instrument recognition system based on a temporal convolutional network, including a data acquisition module and a processing module. The data acquisition module is used to acquire audio data, and the output end of the data acquisition module is connected to the input end of the processing module. The processing module executes the method described in this invention to perform multi-instrument recognition.

[0055] This system integrates a temporal convolutional network (TCN) and a convolutional neural network (CNN) to capture the temporal sequence features of musical instrument audio, thereby more comprehensively reflecting the evolution of audio signals over time and improving recognition accuracy. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the multi-instrument recognition method based on temporal convolutional networks of the present invention.

[0057] Figure 2 This is a schematic diagram of the Mel spectrogram conversion process of the multi-instrument recognition method based on temporal convolutional networks of the present invention;

[0058] Figure 3 This is a schematic diagram of the convolutional neural network structure of the multi-instrument recognition method based on temporal convolutional networks of this invention;

[0059] Figure 4 This is a schematic diagram of the temporal convolutional network structure of the multi-instrument recognition method based on temporal convolutional networks of the present invention. Detailed Implementation

[0060] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0061] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0062] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0063] This invention discloses a multi-instrument recognition method based on temporal convolutional networks. It deeply analyzes the multi-instrument recognition problem in polyphonic music. To address the problem that using spectrograms as input in convolutional neural networks (CNNs) may lead to the loss of phase and temporal information of audio signals, a network structure that integrates temporal convolutional networks (TCNs) and CNNs is proposed, enabling the model to learn and utilize the temporal and frequency features of audio data more comprehensively.

[0064] This invention combines the time-series processing capabilities of TCN (Transmission Channel Network) to extract temporal information from audio data while simultaneously preserving the spectrogram's description of the audio signal in the frequency domain. While the TCN network itself does not have an explicit self-attention mechanism, through cleverly designed convolutional structures, it achieves attention to different time points in the sequence to some extent, especially by expanding the convolutional layers to achieve a wider receptive field.

[0065] like Figure 1 As shown, the multi-instrument recognition method based on temporal convolutional networks includes the following steps:

[0066] Acquire the audio dataset and perform preprocessing, including but not limited to removing silence, stereo conversion, and sampling rate adjustment, to ensure that the audio data input to the model has a consistent format and quality.

[0067] A recognition model is constructed based on convolutional neural networks and temporal convolutional networks (TCNs). Preferably, the recognition model includes a convolutional neural network, a temporal convolutional network, and a feature vector fusion layer, a global average pooling layer, and a fully connected layer connected sequentially. The outputs of the convolutional neural network and the temporal convolutional network are connected to the feature vector fusion layer. After passing through the global average pooling layer and the fully connected layer, the input is fed into the classification layer, where a 0.3 DropOut operation is performed (DropOut is a regularization technique used in deep learning, whose main functions include preventing overfitting, enhancing the generalization ability of the network, reducing coupling between neurons, and reducing the complexity of the network) to prevent feature vectors from directly entering the fully connected layer and causing overfitting on a large amount of retained spatial information.

[0068] The preprocessed audio dataset is converted into a Mel spectrogram and a Wave graph (audio waveform graph), respectively, and then input into the Convolutional Neural Network (CNN) and Temporal Convolutional Network (TCN) of the recognition model to train the recognition model. During the training process, the recognition model learns the features and patterns in the audio data to improve its performance in audio data classification or other tasks.

[0069] The system collects audio data to be recognized, preprocesses it, inputs it into the trained recognition model, and outputs the audio data recognition results.

[0070] In a preferred embodiment of the present invention, the method for preprocessing the audio dataset is as follows:

[0071] Remove silence: Cut off the silent parts at the beginning and end of the audio data to reduce unnecessary background noise interference with the model;

[0072] Two-channel conversion: Stereo audio is converted to mono by taking the average of the left and right channels. Using mono can reduce the complexity of processing.

[0073] Sampling rate adjustment: Converting the sampling rate of the audio data from 44100Hz to 22050Hz allows the use of frequencies up to 11025Hz, i.e., the Nyquist frequency. This is sufficient to cover most of the harmonics produced by instruments, while eliminating noise that may be included in frequencies above this range. This ensures the consistency of the input data, helps avoid problems caused by different sampling rates, and makes the model easier to process.

[0074] In a preferred embodiment of the present invention, the Mel spectrogram is a commonly used representation method in speech signal processing and audio analysis, used to extract spectral information from audio signals. Compared with other spectrograms, the Mel spectrogram is more consistent with the nonlinear characteristics of the human auditory system.

[0075] like Figure 2 As shown, the preprocessed audio dataset is converted into a Mel spectrogram. The specific steps are as follows:

[0076] In the preprocessed audio dataset, audio signal samples are truncated to a preset number of seconds (e.g., 3 seconds, which can be selected according to your dataset and network input settings) and then processed into frames, with the frame shift being a preset number of sampling points (e.g., 512, which can be selected according to network input settings).

[0077] The audio sampling rate is 22.05kHz, so the extracted audio sample (e.g., 3 seconds) yields N = 130 frames. The Mel spectrogram is extracted from each frame, specifically as follows:

[0078] The original waveform of the audio data is converted into the amplitude of each frequency component by fast Fourier transform, and then the frequency is mapped to the Mel frequency.

[0079] The frequency is input into the Mel filter bank that mimics the response of the human auditory system, the frequency is converted into a logarithmic scale and the amplitude is converted into decibels, generating a Mel spectrum. The length of the spectrum is selected (e.g., 128), which can fully preserve the harmonic characteristics of the music while significantly reducing the dimensionality of the input data.

[0080] Converting audio data originally in Hz frequency to mel frequency more closely resembles the characteristics of the human auditory system. The conversion relationship is as follows:

[0081]

[0082] Where f is the Hz frequency and M(f) is the Mel frequency.

[0083] In a preferred embodiment of the present invention, the CNN network is considered as a fusion of the feature extractor and classifier in the recognition model structure, and it is built based on ResNet. Figure 3 As shown, a convolutional neural network (CNN) includes three residual convolutional blocks, each of which comprises a first convolutional layer, a first batch normalization (BN) layer, a second convolutional layer, a second batch normalization (BN) layer, and a max pooling layer connected in sequence.

[0084] The max pooling layer uses a 3×3 receptive field and a filter with a fixed stride size of 1. It uses a 3×3 max pooling layer with a stride size of 1 to process data in a spatial abstraction manner to prevent information loss and perform zero padding.

[0085] The number of convolutional layer filters in the three residual convolutional blocks increases in multiples of 2, that is, the number of filters in each residual convolutional block is 32, 64, and 128, respectively. The change in the number of filters helps to gradually improve the network's ability to learn more abstract and complex features. The earlier convolutional layers may mainly capture low-level features such as edges and textures, while the deeper convolutional layers can learn higher-level and semantically richer features.

[0086] The Mel spectrogram is input into a convolutional neural network, and convolution yields the feature F. cnn .

[0087] In a preferred embodiment of this invention, TCN, a deep learning structure capable of effectively capturing temporal information, is introduced to better adapt to complex multi-instrument audio data. By employing TCN, this method aims to overcome challenges such as overlap and data imbalance in multi-instrument recognition, providing more accurate and reliable multi-instrument classification performance. By fully utilizing TCN's ability to model temporal relationships, the model can more comprehensively understand and process the interaction relationships between different instruments in polyphonic music, thereby achieving more significant results in multi-instrument recognition tasks.

[0088] like Figure 4 As shown, the temporal convolutional network includes three temporal blocks, each of which includes two dilated convolutional layers, two weight normalization layers, a ReLU activation layer, and a dropout layer connected in sequence.

[0089] The Wave graph of the audio data is sequentially processed through three temporal convolutional blocks. Within each temporal convolutional block, the Wave graph is processed by two dilated convolutional layers and a weight normalization layer. The result is then added to the original input data, followed by a ReLU activation layer, a dropout layer, and finally, the output feature F of the temporal convolutional block. tcn ;

[0090] F tcn and the features F output by the convolutional neural network cnn Feature fusion is performed on the channels to obtain the final output feature F. Modeling long-term dependencies has enabled TCN to achieve good results in multiple time series tasks. At the same time, this feature fusion method enables comprehensive and effective feature extraction from audio data, which can cope with the complex interaction between time domain and frequency domain information in audio processing tasks.

[0091] Temporal convolutional networks, using audio waveforms, can more comprehensively capture the temporal characteristics of audio signals, emphasizing temporal information in audio data. This enhances the ability to learn long-term dependencies and temporal patterns, thereby more effectively extracting and utilizing the temporal structure of audio data.

[0092] The application of dilated convolution in TCN enables the model to achieve an exponentially larger receptive field, meaning that its receptive field grows exponentially with the depth of the convolutional layer. This implies that TCN can effectively capture long-term historical information when processing temporal data, surpassing traditional convolutional structures with linearly growing receptive fields.

[0093] To mitigate the risk of overfitting during training, a dropout layer is introduced after the activation layer for regularization. By randomly removing a subset of neurons and setting the entire channel to zero after each training iteration, the dropout layer helps prevent the network from becoming overly reliant on specific neurons, thereby improving the model's generalization performance. This strategy makes the network more robust and adaptable to diverse input data, providing an effective regularization method for training deep networks.

[0094] In a preferred embodiment of the present invention, the method for optimizing the temporal convolutional network is as follows:

[0095] The exponential receptive field acquisition method improves the network's ability to model long-range contexts while maintaining a relatively low number of model parameters, thereby reducing the complexity of the TCN model.

[0096] For a one-dimensional sequence input x∈R n Given a convolution kernel f:{0,...,k-1}→R, the dilation convolution operation on the elements s of the sequence is defined as:

[0097]

[0098] Where d represents the dilation factor, k represents the scale of the convolution kernel, and sd·i represents the direction in which the learning process can understand and capture data trends and patterns from past time points; R n This vector contains n elements, where i represents the number of hidden layers, and DC(S) represents the result of dilated convolution of element s. d f)(s) represents the convolution of input x and function df with expansion factor d at position s, and f(i) represents the value of function f at position i;

[0099] Dilation is equivalent to introducing a gap with a fixed stride between two adjacent elements of the convolution kernel, allowing the kernel to skip intermediate elements during operation, thus affecting the receptive field of the convolution operation. When a large dilation factor is used, i.e., d is large, a single feature map unit in the output layer can represent information from the input over a large time range prior to that feature time step, effectively expanding the receptive field of the convolutional network.

[0100] When using dilated convolutions, the dilation factor grows exponentially with increasing network depth; this ensures that the convolutional kernel can effectively capture information from each input, enabling deeper networks to represent longer historical information and allowing the model to more effectively capture and utilize long-term dependencies in time-series data.

[0101] In TCN, the receptive field is determined by three hyperparameters: network depth, kernel size, and dilation factor. By adjusting these three hyperparameters, deeper TCN models with larger receptive fields can be constructed. Time-series data is data that changes continuously over time; therefore, the stability of TCN becomes crucial when training on such data.

[0102] A temporal convolutional network contains a branch with a series of transformations F (transformation sequence F: a series of transformations or layers within the residual block that process the input x), including dilated causal convolution, weight normalization, ReLU activation, and other processes. The output of this branch is the sequence input x∈R. n Add them together:

[0103] o = Activation(x + F(x))

[0104] Where Activation is the activation function, o represents the output after passing through the activation function, and F(X) represents the output obtained after processing the input x by the F transform;

[0105] This allows each hidden layer (i represents the number of hidden layers; this network has 3 hidden layers) to effectively learn the local transformations of equivalent mappings without directly modeling the entire transformation process. This is significant for deep networks because it improves network efficiency and stability.

[0106] The present invention also provides a multi-instrument recognition system based on a temporal convolutional network, including a data acquisition module and a processing module. The data acquisition module is used to acquire audio data, and the output end of the data acquisition module is electrically connected to the input end of the processing module. The processing module executes the method described in this invention to perform multi-instrument recognition.

[0107] This invention uses Temporal Convolutional Networks (TCNs) to capture long-term dependencies in temporal data. In instrument audio, certain features may exhibit long temporal patterns. The structure of TCNs enables them to effectively capture these long-term temporal relationships, contributing to improved accuracy in instrument recognition.

[0108] The features obtained from the TCN sub-network module and the CNN sub-network module are fused into vectors. This design aims to address the issue that CNNs may overlook the temporal features of audio, while preserving the uniform frequency distribution of the Mel spectrogram. Through feature synthesis, the model can more comprehensively capture and utilize the temporal and frequency features of audio data, improving instrument recognition performance.

[0109] Multi-label tasks typically involve features at different time scales because different labels may evolve at different rates over time. Dilated convolutions in TCNs allow the network to effectively learn features at different time scales, thus better adapting to the temporal patterns of multi-label tasks.

[0110] Temporal Convolutional Networks (TCNs) are generally considered more lightweight than Transformer models. TCNs have a relatively simple structure and fewer parameters, making them suitable for lightweight deployment in resource-constrained environments. By employing dilated convolutions, TCNs effectively capture long-term dependencies while avoiding the complexity of self-attention mechanisms. This makes TCNs perform better in scenarios where model size and computational complexity are limited.

[0111] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-instrument recognition method based on temporal convolutional networks, characterized in that, Includes the following steps: Obtain the audio dataset and perform preprocessing; A recognition model is constructed based on convolutional neural networks and temporal convolutional networks. The recognition model includes a convolutional neural network, a temporal convolutional network, and a feature vector fusion layer, a global average pooling layer, and a fully connected layer connected in sequence. The preprocessed audio dataset is converted into Mel spectrograms and Wave graphs, respectively, and then input into the convolutional neural network (CNN) and temporal convolutional network (TCN) of the recognition model to train the recognition model. Collect audio data to be recognized, preprocess it, input it into the trained recognition model, and output the audio data recognition result. The convolutional neural network includes three residual convolutional blocks, each of which comprises a first convolutional layer, a first batch normalization (BN) layer, a second convolutional layer, a second batch normalization (BN) layer, and a max pooling layer connected in sequence. The temporal convolutional network includes three temporal convolutional blocks, each of which includes two dilated convolutional layers, two weight normalization layers, a ReLU activation layer, and a dropout layer connected in sequence. The Wave graph of the audio data is sequentially processed through three temporal convolutional blocks. Within each temporal convolutional block, the Wave graph is processed by two dilated convolutional layers and a weight normalization layer. The result is then added to the original input data, followed by a ReLU activation layer, a dropout layer, and finally, the output feature F of the temporal convolutional block. tcn ; F tcn and the features F output by the convolutional neural network cnn Feature fusion is performed on the channels to obtain the final output feature F. 最终 ; The methods for optimizing temporal convolutional networks are as follows: For a one-dimensional sequence input and a convolution kernel Elements in the sequence The dilated convolution operation on is defined as: ， Where d represents the dilation factor, k represents the scale of the convolution kernel, and sd·i represents the direction in which the learning process can understand and capture data trends and patterns from past time points; R n This indicates that the elements of this vector are taken from the real number field and there are n elements; i represents the number of hidden layers. This represents the result of element s after dilation and convolution. Indicates input The convolution of the function df with the dilation factor d at position s, This represents the value of function f at position i; When using dilated convolution, the dilation factor increases exponentially with increasing network depth; A temporal convolutional network contains a branch with a series of transformations F, the output of which is the sequence input. Add them together: ， Where Activation is the activation function, and o represents the output after passing through the activation function. Indicates the F-transform with respect to the input The output obtained after processing; This enables each hidden layer to effectively learn the local transformations of equivalent mappings without having to directly model the entire transformation process.

2. The multi-instrument recognition method based on temporal convolutional networks as described in claim 1, characterized in that, The method for preprocessing audio datasets is as follows: Remove silence: Cut off the silent portions at the beginning and end of the audio data; Two-channel conversion: Stereo audio is converted to mono by taking the average value of the left and right channels; Sampling rate adjustment: Convert the audio sampling rate to 22050Hz.

3. The multi-instrument recognition method based on temporal convolutional networks as described in claim 1, characterized in that, The outputs of the convolutional neural network and the temporal convolutional network are connected to the feature vector fusion layer, and after passing through the global average pooling layer and the fully connected layer, they are input to the classification layer and subjected to a DropOut operation of 0.

3.

4. The multi-instrument recognition method based on temporal convolutional networks as described in claim 1, characterized in that, The preprocessed audio dataset is converted into a Mel spectrogram. The specific steps are as follows: In the preprocessed audio dataset, audio signal samples are truncated to a preset number of seconds and then processed into frames, where the frame shift is a preset number of sampling points. The audio sampling rate is 22.05kHz, therefore, the number of frames N is obtained from the extracted audio samples. Mel spectrogram extraction is performed on each frame, specifically as follows: The original waveform of the audio data is converted into the amplitude of each frequency component by fast Fourier transform, and then the frequency is mapped to the Mel frequency. The frequency is input into a Mel filter bank that mimics the response of the human auditory system, the frequency is converted into a logarithmic scale, and the amplitude is converted into decibels to generate a Mel spectrum, the length of which is selected. The audio data, originally in Hz frequency, is converted to mel frequency. The conversion relationship is as follows: ， Where f is the Hz frequency and M (f) is the Mel frequency.

5. The multi-instrument recognition method based on temporal convolutional networks as described in claim 1, characterized in that, The max pooling layer uses a 3×3 receptive field and a filter with a fixed step size of 1. It uses a 3×3 max pooling layer with a step size of 1 to process data through spatial abstraction. The number of convolutional layer filters in the three residual convolutional blocks increases in multiples of 2, that is, the number of filters in each residual convolutional block is 32, 64, and 128, respectively. The change in the number of filters helps to gradually improve the network's ability to learn more abstract and complex features. The Mel spectrogram is input into a convolutional neural network, and convolution yields the feature F. cnn .

6. A multi-instrument recognition system based on temporal convolutional networks, characterized in that, It includes a data acquisition module and a processing module. The data acquisition module is used to acquire audio data. The output end of the data acquisition module is connected to the input end of the processing module. The processing module performs the method described in any one of claims 1-5 to perform multi-instrument recognition.

Images