Multi-task music notation method fusing pitch and timbre characteristics and related device

Abstract

The invention provides a multi-task music score recording method and device fusing pitch and timbre features, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-recognized audio signal; inputting the to-be-recognized audio signal into the multi-task learning model framework, and outputting a multi-musical-instrument music notation result; wherein the shared feature extraction module is used for extracting multi-scale high-level semantic features of an audio signal to be recognized, the two auxiliary task branches are respectively used for extracting pitch features and timbre features based on the multi-scale high-level semantic features, and the feature fusion module is used for performing deep fusion on the pitch features and the timbre features. According to the multi-musical-instrument music score recording method, on the basis of a multi-task learning framework, two auxiliary tasks of pitch estimation and musical instrument classification are introduced, frequency distribution and timbre modes of audios are modeled respectively, pitch and timbre semantic representations are extracted, and the audio frequency distribution and timbre semantic representations are explicitly injected into multi-musical-instrument music score recording tasks through a fusion mechanism. And the multi-instrument multi-tone music automatic notation performance is improved.

Description

Technical Field

[0001] This invention relates to the field of computer auditory and music information retrieval technology, and in particular to a multi-task music notation method, apparatus, electronic device and storage medium that integrates pitch and timbre features. Background Technology

[0002] Automatic music transcription (AMT) is the process of converting musical audio signals into symbolic musical representations, such as numbered musical notation, staff notation, piano score, and Musical Instrument Digital Interface (MIDI). It is a core task in the field of Music Information Retrieval (MIR). Although AMT technology has made some progress in recent years, existing methods are mostly designed for single-instrument or simple melodic scenarios, and still struggle to effectively model and analyze polyphonic music with multiple voices. Current multi-instrument music notation still heavily relies on manual track-by-track annotation, a complex, labor-intensive, and extremely time-consuming process. This severely limits the construction of large-scale, high-quality music datasets and hinders the development of data-driven intelligent music analysis and processing technologies. In related technologies, multi-task learning has been introduced into the field of music notation. Multi-task learning-based multi-instrument automatic music notation methods typically use multi-instrument time-spectrum diagrams as input, modeling only single features such as pitch or timbre, while ignoring their collaborative relationship. For example, while multi-channel enhancement structures can enhance spectral features, they do not explicitly model instrument category information, which makes pitch recognition in multi-instrument scenarios susceptible to interference from timbre differences and inaccurate recognition of the pitch activities of various instruments in the input audio signal. Summary of the Invention

[0003] This invention provides a multi-task music notation method, device, electronic device, and storage medium that integrates pitch and timbre features. This addresses the shortcomings of traditional automatic music notation techniques in the automatic transcription of multi-instrument polyphonic music, where there are significant differences in harmonic structure between instruments at the same pitch, the performance range is limited, and relying solely on pitch or timbre information can easily lead to recognition errors.

[0004] This invention provides a multi-task music notation method that integrates pitch and timbre features, including: Acquire the audio signal to be recognized; The audio signal to be identified is input into a multi-task learning model framework, and the multi-instrument music notation results are output. The multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification. The shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified. The two auxiliary task branches are used to extract pitch features and timbre features based on the multi-scale high-level semantic features, respectively. The feature fusion module is used to perform deep fusion of the pitch features and the timbre features. The main task branch is used to output multi-instrument music notation results based on the fused features.

[0005] According to the multi-task music notation method integrating pitch and timbre features provided by the present invention, the shared feature extraction module is constructed based on a U-Net encoder. The shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified, including: Multi-scale high-level semantic features of the spectrogram corresponding to the audio signal to be identified are extracted by multi-layer two-dimensional convolution and pooling operations.

[0006] According to the multi-task music notation method that integrates pitch and timbre features provided by the present invention, the method for obtaining the spectrogram corresponding to the audio signal to be identified includes: The audio signal to be identified is converted into a mono format and resampled to a preset sampling rate; Based on the mono format audio with the preset sampling rate, the spectrogram corresponding to the audio signal to be identified is calculated using the constant Q transform algorithm.

[0007] According to the multi-task music notation method integrating pitch and timbre features provided by the present invention, the two auxiliary task branches are respectively used to extract pitch features and timbre features based on the multi-scale high-level semantic features, including: The multi-scale high-level semantic features are respectively input into the pitch estimation branch and the instrument classification branch; In the pitch estimation branch, pitch-related features are extracted through the first convolutional layer, and 256-channel intermediate pitch features are output; the second convolutional layer focuses on pitch discriminative features; and the first convolutional output layer maps the features to 88-dimensional pitch activation probabilities. In the instrument classification branch, timbre-related features are extracted through the third convolutional layer, and 256-channel intermediate timbre features are output. These intermediate timbre features encode both instrument category information and timbre characteristics. The discriminative features of the instrument are enhanced through the fourth convolutional layer. The features are then mapped to 11-dimensional instrument classification probabilities through the second convolutional output layer.

[0008] According to the multi-task music notation method for fusing pitch and timbre features provided by the present invention, the feature fusion module is used to perform deep fusion of the pitch features and the timbre features, including: The multi-scale high-level semantic features output by the shared feature extraction module are converted into a two-dimensional matrix as a query vector. The intermediate pitch features and intermediate timbre features output by the pitch estimation branch and the instrument classification branch are concatenated to form an auxiliary feature matrix; The auxiliary feature matrix is ​​mapped into a key matrix and a value matrix through convolution, respectively; A mask matrix is ​​constructed based on the coupling relationship between pitch and timbre; A multi-head attention mechanism is adopted, which projects the query vector, key matrix and value matrix into multiple subspaces respectively. In each attention head, the attention weights of the query vector and key matrix are calculated. After softmax normalization of the mask matrix, the weights are multiplied with the value matrix to obtain the attention features output by each head. The attention features output from each head are concatenated and fused through a linear transformation to obtain a comprehensive attention feature; The integrated attention features and the multi-scale high-level semantic features are residually connected, and the residual connection results are normalized to obtain the fused features.

[0009] According to the multi-task music notation method for fusing pitch and timbre features provided by the present invention, the main task branch is constructed based on a U-Net decoder, the U-Net decoder being symmetrical to the U-Net encoder, and the main task branch is used to output multi-instrument music notation results based on the fusion result, including: The fused features are used as direct input to the first layer of the decoder to provide an initial feature representation containing pitch-timbre coordination information; During each layer of decoding, skip connections are used to combine with the features of the corresponding coding layer to supplement detailed information at different scales on the basis of the initial feature representation. The output layer outputs multi-instrument music notation results based on the detailed information at different scales. The multi-instrument music notation results are three-dimensional tensor representations of instrument category, pitch information and time position.

[0010] According to the multi-task music notation method integrating pitch and timbre features provided by the present invention, the multi-task learning model framework is trained based on a multi-task joint loss function, including: The multi-task joint loss function includes the binary cross-entropy loss of the pitch estimation branch, the binary cross-entropy loss of the instrument classification branch, and the binary cross-entropy loss of the multi-instrument notation main branch.

[0011] The present invention also provides a multi-task music notation device that integrates pitch and timbre characteristics, comprising: The acquisition module is used to acquire the audio signal to be recognized; The output module is used to input the audio signal to be identified into the multi-task learning model framework and output the multi-instrument music notation results; The multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification. The shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified. The two auxiliary task branches are used to extract pitch features and timbre features based on the multi-scale high-level semantic features, respectively. The feature fusion module is used to perform deep fusion of the pitch features and the timbre features. The main task branch is used to output multi-instrument music notation results based on the fused features.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-task music notation method for integrating pitch and timbre features as described in any of the preceding claims.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-task music notation method for integrating pitch and timbre features as described above.

[0014] This invention provides a multi-task music notation method, apparatus, electronic device, and storage medium that integrates pitch and timbre features. The method involves acquiring an audio signal to be identified, inputting the audio signal into a multi-task learning model framework, and outputting multi-instrument music notation results. The multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches: pitch estimation and instrument classification. The shared feature extraction module extracts multi-scale high-level semantic features from the audio signal to be identified. The two auxiliary task branches extract pitch and timbre features based on the multi-scale high-level semantic features, respectively. The feature fusion module performs deep fusion of the pitch and timbre features. The main task branch outputs multi-instrument music notation results based on the fused features. This invention, based on a multi-task learning framework, introduces two auxiliary tasks: pitch estimation and instrument classification. These tasks model the frequency distribution and timbre patterns of the audio, extract semantic representations of pitch and timbre, and explicitly inject them into the multi-instrument notation task through a fusion mechanism, thereby improving the automatic notation performance of multi-instrument polyphonic music. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the multi-task music notation method that integrates pitch and timbre features provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the multi-task learning model framework provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the fusion module processing flow provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the functional structure of the multi-task music notation device that integrates pitch and timbre features provided in an embodiment of the present invention; Figure 5 This is a functional structure diagram of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0018] Figure 1 The flowchart of the multi-task music notation method that integrates pitch and timbre features provided in the embodiments of the present invention is as follows: Figure 1 As shown, the multi-task music notation method that integrates pitch and timbre features provided in this embodiment of the invention includes: Step 101: Obtain the audio signal to be recognized; Step 102: Input the audio signal to be identified into the multi-task learning model framework and output the multi-instrument music notation results; The multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification. The shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified. The two auxiliary task branches are used to extract pitch features and timbre features based on the multi-scale high-level semantic features, respectively. The feature fusion module is used to perform deep fusion of the pitch features and the timbre features. The main task branch is used to output multi-instrument music notation results based on the fused features.

[0019] This invention breaks through the limitations of traditional methods in modeling single features, and jointly models pitch and timbre information within a unified framework.

[0020] In the embodiments of the present invention, such as Figure 2 As shown, the model structure consists of a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, two lightweight auxiliary task branches (pitch estimation and instrument classification), and an output layer. The shared feature extraction module is built based on the U-Net encoder structure, and the extracted shared features are input in parallel to the two auxiliary task branches and the feature fusion module. The two auxiliary branches not only perform their respective supervised learning tasks but also provide intermediate semantic features for the fusion module of the main task. The fusion module is located between the shared feature extraction module and the main task multi-instrument transcription branch. It performs cross-task information fusion by introducing the output of the shared feature extraction module and the intermediate semantic features from the pitch estimation and instrument classification auxiliary tasks. The fused features are finally predicted by the main task multi-instrument music transcription branch, and the temporal and frequency resolution of the feature map is gradually restored through layer-by-layer deconvolution operations. In each decoding stage, the decoder is concatenated with the output of the corresponding layer of the encoder in the shared feature extraction module through skip connections.

[0021] Automatic notation technology for multi-instrument polyphonic music aims to convert musical audio into musical notation. In traditional automatic transcription tasks of multi-instrument polyphonic music, pitch information and instrument information are two key factors in building an accurate recognition system. Instruments exhibit significant differences in harmonic structure at the same pitch, and their playing range is limited. Traditional multi-task learning has been introduced into the field of music notation. However, multi-task learning-based automatic music notation methods for multi-instrument music typically use the time-spectrum diagrams of multiple instruments as input, modeling only a single feature of pitch or timbre, while ignoring the synergistic relationship between the two. Relying solely on pitch or timbre information easily leads to recognition errors.

[0022] The multi-task music notation method merging pitch and timbre features provided in this invention involves acquiring an audio signal to be identified; inputting the audio signal to be identified into a multi-task learning model framework; and outputting multi-instrument music notation results. The multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches: pitch estimation and instrument classification. The shared feature extraction module extracts multi-scale high-level semantic features from the audio signal to be identified. The two auxiliary task branches are used to extract pitch and timbre features based on the multi-scale high-level semantic features, respectively. The feature fusion module performs deep fusion of the pitch and timbre features. The main task branch outputs multi-instrument music notation results based on the fused features. This invention, based on a multi-task learning framework, introduces two auxiliary tasks: pitch estimation and instrument classification. These tasks model the frequency distribution and timbre patterns of the audio, extract semantic representations of pitch and timbre, and explicitly inject them into the multi-instrument notation task through a fusion mechanism, thereby improving the automatic notation performance of multi-instrument polyphonic music.

[0023] Based on any of the above embodiments, the shared feature extraction module is built based on the U-Net encoder, and the shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified, including: Multi-scale high-level semantic features of the spectrogram corresponding to the audio signal to be identified are extracted by multi-layer two-dimensional convolution and pooling operations.

[0024] In this embodiment of the invention, the shared feature extraction module adopts a four-layer two-dimensional convolutional network structure. This module's input It is the CQT spectrogram obtained after preprocessing, where T represents the number of spectrogram frames contained in the feature map, and F represents the number of frequency points of the spectrogram contained in the feature map. It consists of two 3×3 two-dimensional convolutional layers, a batch normalization (BatchNorm) layer, and a ReLU activation function. After each encoding stage, a 2×2 max pooling operation is used to downsample the time and frequency dimensions, achieving spatial compression and receptive field expansion. As the network depth increases, the number of convolutional channels doubles layer by layer to gradually capture higher-order features and enhance abstraction capabilities. The number of convolutional channels in each encoding stage is 64, 128, 256, and 512, respectively. The outputs are all passed to the corresponding decoder layer via skip connections to recover detailed features. Then, output The data is passed to the fusion module and two auxiliary task branches, where C is the number of channels, T is the number of frames after downsampling, and F is the number of frequency points after downsampling.

[0025] In this embodiment of the invention, the method for obtaining the spectrogram corresponding to the audio signal to be identified includes: Step 201: Convert the audio signal to be identified into a mono format and resample it to a preset sampling rate; Step 202: Based on the mono format audio with the preset sampling rate, the spectrogram corresponding to the audio signal to be identified is calculated using the constant Q transform algorithm.

[0026] This invention uses the librosa audio processing library to preprocess the original audio. First, the original mono audio signal with a sampling rate of 44.1kHz is downsampled to 16.0kHz. Then, a Constant-Q Transform (CQT) spectrogram is calculated, with the following parameters: a Hanning window is used, the window length is 512, the lowest frequency is set to 27.5Hz (corresponding to the lowest note A0 on a piano), and each octave is divided into 48 frequency points, meaning each frequency point corresponds to a quarter-tone, resulting in a total of 352 frequency points. After obtaining the complex-domain CQT spectrogram, its magnitude is extracted and converted to a dB scale. The final output CQT spectrogram is a two-dimensional matrix, with dimensions corresponding to the number of frames and the number of frequency points in the audio. To improve data loading efficiency during the training phase, all training samples are organized and stored according to the frame number dimension. In the actual training and testing process, the preprocessed and stored logarithmic CQT spectrum needs to be segmented. During training, the deep neural network proposed in this embodiment performs spectral estimation on the 2064×352 CQT spectrum each time, where 2064 represents the number of frames contained in the sample and 352 represents the number of frequency points in each frame. Similarly, during testing, the deep neural network proposed in this embodiment performs spectral prediction on the 2064×352 CQT spectrum each time.

[0027] Based on any of the above embodiments, the two auxiliary task branches are respectively used to extract pitch features and timbre features based on the multi-scale high-level semantic features, including: Step 301: Input the multi-scale high-level semantic features into the pitch estimation branch and the instrument classification branch respectively; Step 302: In the pitch estimation branch, the pitch-related features are extracted through the first convolutional layer, and the intermediate pitch features of 256 channels are output; the pitch discriminative features are focused through the second convolutional layer; and the features are mapped to 88-dimensional pitch activation probabilities through the first convolutional output layer. In this embodiment of the invention, the pitch estimation auxiliary branch is used to perform polyphony estimation on the input audio at the frame level. The pitch estimation auxiliary branch receives the output from the shared feature extraction module. Next, a two-dimensional convolutional module is first connected. This convolutional layer uses a 3×3 convolutional kernel with a stride of 1 and an output channel count of 256. It is then connected to a BatchNorm layer and a ReLU activation function to obtain the 256-channel intermediate features. The semantic features are fed into the feature fusion module to help model the timbre-pitch coupling relationship. Then, this branch continues with a 3×3 convolutional layer to further integrate local time-frequency information, followed by a BatchNorm layer and a ReLU activation function. Finally, it is connected to the output layer, where a 1×1 convolution maps the number of channels to 88, resulting in an output tensor of size 88×T. The output is , which represents the activation probability of the 88 MIDI pitches in each frame.

[0028] Step 303: In the instrument classification branch, timbre-related features are extracted through the third convolutional layer, and 256-channel intermediate timbre features are output. The intermediate timbre features encode both instrument category information and timbre characteristics. The discriminative features of the instrument are enhanced through the fourth convolutional layer. The features are mapped to 11-dimensional instrument classification probabilities through the second convolutional output layer.

[0029] In this embodiment of the invention, the instrument classification auxiliary branch is used to perform multi-instrument classification on the input audio at the frame level. This convolutional layer uses a 3×3 convolutional kernel with a stride of 1, and the number of output channels is set to 256. It is then connected to a BatchNorm layer and a ReLU activation function, resulting in 256 channels of intermediate features. This intermediate feature is identical to the intermediate feature in the pitch estimation branch. It is not only used for output prediction in this branch but also passed to the feature fusion module as a representation of timbre features to participate in the joint modeling of the main task, thus providing auxiliary information for the multi-instrument music notation task. Subsequently, this branch continues with a 3×3 convolutional layer to further integrate local time-frequency information, followed by a BatchNorm layer and a ReLU activation function. Finally, it is connected to the output layer, using a 1×1 convolution to map the channel dimension to the target output dimension of 11, representing the 11 supported instrument classes. Finally, through... The output is a two-dimensional tensor of size 11×T. , where T represents the activation probability of each type of instrument in each frame, and T represents the frame number.

[0030] Based on any of the above embodiments, the feature fusion module is used to perform deep fusion of the pitch feature and the timbre feature, including: Step 401: Convert the multi-scale high-level semantic features output by the shared feature extraction module into a two-dimensional matrix as a query vector; In this embodiment of the invention, the feature fusion module is located between the shared feature extraction module and the multi-instrument transcription branch. It integrates pitch and timbre features of the auxiliary task based on a cross-attention mechanism, aiming to enhance the main task's ability to model the coupling relationship between timbre and pitch during multi-instrument note transcription. The module's input includes encoder output features. intermediate semantic features in pitch and instrument branches , 。 Reshape the two-dimensional matrix using the three inputs. , , 。 Encoder output The query vector is generated after passing through a convolutional layer with a kernel size of 1: , in This indicates a pointwise convolution operation. Reshaped into projection dimensions In this embodiment, d represents the tap dimension corresponding to each attention tap, and its size is set to 512.

[0031] Step 402: Concatenate the intermediate pitch features and intermediate timbre features output by the pitch estimation branch and the instrument classification branch to form an auxiliary feature matrix; The features obtained by concatenating the pitch recognition branch and the instrument classification branch remain unchanged in the time and frequency dimensions, and the number of channels is the sum of the two, thus ensuring that... The features are all the same size, which facilitates subsequent fusion processing. The splicing operation is as follows: Remodeling Then, a 1×1 convolution is used to map the data to keys and values: in, , , Remodeling ， Where d is 512.

[0032] Step 403: Map the auxiliary feature matrix into a key matrix and a value matrix through convolution, respectively; Step 404: Construct a mask matrix based on the coupling relationship between pitch and timbre; Step 405: Employ a multi-head attention mechanism to project the query vector, key matrix, and value matrix into multiple subspaces. Calculate the attention weights of the query vector and key matrix in each attention head. After softmax normalization of the mask matrix, multiply the weights by the value matrix to obtain the attention features output by each head. Step 406: Concatenate the attention features output by each head and fuse them through a linear transformation to obtain the comprehensive attention features; Step 407: Perform residual connection between the integrated attention feature and the multi-scale high-level semantic feature, and normalize the residual connection result to obtain the fused feature.

[0033] In the feature fusion module, the mask matrix M is dynamically generated based on the correlation between pitch and intermediate features in the instrument branch. Specifically, each query vector... On the frequency channels, only key / value features related to their pitch or timbre are allowed to be focused on. It masks frequency channels that are irrelevant to the current task. The mask matrix sets the feature weights that are irrelevant to the current task to... This makes the attention weights at these positions zero after softmax, so they are completely excluded from the attention score calculation. This allows the model to focus on learning the coupling relationship between pitch and timbre, avoiding interference from irrelevant information. The mask matrix is ​​defined as: In the frequency domain, the coupling relationship between pitch and timbre can be determined by the following three strategies to determine the frequency relevance of the mask matrix: First, pitch-corresponding frequency, allowing only the query vector to access frequency channels related to the fundamental frequency and harmonics of the target note; second, instrument frequency band matching, masking frequency channels unrelated to the timbre of the current target instrument to enhance the selectivity of timbre features; third, task relevance, dynamically filtering frequency channels that contribute only to the current task by combining pitch estimation and intermediate features of the instrument classification branch. These three strategies work together to enable the mask matrix to precisely control the scope of attention computation, ensuring that the model focuses on frequency information related to pitch-timbre coupling.

[0034] The attention score is calculated as follows: ' Where h represents the number of attention taps, which is set to 4 in this embodiment, and d is set to 512. The multi-head attention value is calculated for all attention taps as follows: in , This indicates a matrix concatenation operation along the channel dimension of the attention tap, for any one of them... ,have: Where i represents the attention tap number.

[0035] The output undergoes concatenation and linear transformation through multi-head attention, as detailed below. Figure 3 As shown, the final fusion process is achieved through residual connections and layer normalization (LayerNorm): Fusion results It is passed to the main task decoder, where d is 512.

[0036] Based on any of the above embodiments, the main task branch is constructed based on the U-Net decoder, the U-Net decoder being symmetrical to the U-Net encoder, and the main task branch is used to output multi-instrument music notation results based on the fusion result, including: Step 501: Use the fused features as the direct input to the first layer of the decoder to provide an initial feature representation containing pitch-timbre coordination information; Step 502: During the decoding process of each layer, by combining the features of the corresponding coding layer with skip connections, detailed information at different scales is supplemented on the basis of the initial feature representation; Step 503: Output the multi-instrument music notation results at the output layer based on the detailed information at different scales. The multi-instrument music notation results are a three-dimensional tensor representation of instrument category, pitch information and time position.

[0037] In this embodiment of the invention, the main task branch is composed of a U-Net decoder, which receives the feature tensor output from the bottleneck layer fusion module. The decoder has a symmetrical structure and a shared encoder, consisting of 4 layers of deconvolution modules. Composition. Each First, an upsampling operation is performed to gradually restore the resolution in the time and frequency dimensions. The decoder includes a deconvolution operation with a kernel size of 1, a BatchNorm layer, and a ReLU activation function, and is concatenated with the outputs of each encoder layer via skip connections. The number of channels in the decoder is 512, 256, 128, and 64, decreasing layer by layer to compress the parameter size. The final output layer uses a convolutional layer with a kernel size of 1 to map the number of channels to the target dimension. This represents each instrument in each frame. The activation probability at each pitch position. Where S=11 represents the number of instrument categories, F=88 represents the number of pitches, and T is the number of frames.

[0038] In this embodiment of the invention, each task branch is responsible for extracting a specific note estimation target, namely, a pitch estimation branch, an instrument recognition branch, and a multi-instrument music notation branch. Each branch processes the input features according to its specific target: the pitch estimation branch generates a pitch activation map, the instrument recognition branch performs instrument classification, and the multi-instrument music notation branch outputs the notation result. The output results of the three branches are processed through their respective output layers. Each output layer contains a convolution operation that maps the features of each task branch to the target dimension, generating a prediction result that meets the task requirements. , , Where S represents the number of supported instrument types (e.g., set to 11), P represents the pitch dimension (covering 88 pitches from MIDI numbers 21-109), and T represents the number of frames. To adapt to multi-label binary classification tasks, the output layer uniformly uses the Sigmoid activation function to normalize the output results. Each branch predicts the frame-level pitch activation map, instrument activation map, and joint instrument-pitch spectrogram, respectively, and the output is a probability matrix, which can be binarized by setting a threshold. Here, , As input It has a sigmoid function. for: This invention introduces a multi-task learning framework, incorporating auxiliary tasks (such as pitch estimation and instrument classification) on top of the main task (multi-instrument transcription). Through collaborative optimization between tasks, the model's notation performance is significantly improved. A fusion module based on cross-attention performs deep fusion of pitch and timbre features.

[0039] Since all three tasks are essentially multi-label binary classification problems, this embodiment of the invention employs a binary cross-entropy loss function to supervise the prediction results of each task. The formula for the binary cross-entropy loss function is as follows: in, For real labels, This represents the probability value predicted by the model.

[0040] The final loss is a weighted sum of the losses from the three tasks, used to jointly optimize the overall model performance.

[0041] In this embodiment of the invention, the proposed multi-task learning model is trained and evaluated based on the publicly available MusicNet dataset. MusicNet contains 11 common Western musical instruments, including piano, violin, viola, cello, double bass, French horn, trumpet, trombone, oboe, clarinet, and bassoon, demonstrating strong diversity and representativeness. This dataset covers MIDI pitches from 0 to 127, with pitches from 21 to 109 selected for processing, covering the effective range of most instruments. The dataset provides note-level annotation information, including the start and end times of the notes, the corresponding instrument number, and the pitch number. To supervise the training and performance evaluation of the deep neural network in this embodiment, the CSV annotation files included in the MusicNet dataset are normalized. Each audio sample and its corresponding annotation file have the same unique identifier (ID), enabling accurate matching of audio and labels. First, the CSV files are structurally transformed. Based on the ID, all annotation files in the specified directory are traversed, and the note event information recorded in each file is loaded one by one. The key fields of each note event are analyzed, including start time, end time, instrument number, and pitch number. The start and end times are calculated based on the number of sampling points at a 44.1kHz sampling rate. The analysis results are stored in a dictionary format, containing the start time, end time, instrument number, and pitch number of each sample. The set of event lists for all samples forms a dictionary structure with IDs as keys and individual event lists as values. This dictionary is then saved using pickle serialization for later label alignment and training. The preprocessing steps for the labels include: traversing all labeled files in the dataset, sorting by ID name, loading the corresponding mono audio, and uniformly resampling to 16.0kHz. For each audio segment, a CQT spectrogram is calculated to determine the frame number information, which serves as the time dimension of the label matrix. Based on the number of frames and categories, three key label matrices are initialized: instrument activation labels (number of instruments × number of frames), pitch labels (number of pitches × number of frames), and multi-instrument pitch clef labels (number of instruments × number of pitches × number of frames), representing the instrument activation state, pitch activation state, and joint activation status of each frame, respectively. Then, pre-processed pickle-formatted note events are loaded, and the start and end times of each event are parsed and converted into corresponding frame indices. Within the corresponding time interval, the relevant instrument and pitch labels are set to 1, and others to 0, achieving frame-level multi-task label alignment and accurately reflecting the time, pitch, and instrument attribute distribution of notes. Finally, the generated label matrices are saved in sample order to ensure strict time alignment between labels and input features during training and testing, providing accurate supervision for multi-task joint training.

[0042] This invention provides a multi-task music notation method that integrates pitch and timbre features. It constructs a unified multi-task modeling framework, jointly optimizing three tasks: pitch estimation, instrument classification, and multi-instrument music notation. Through collaborative modeling and fusion of pitch and timbre features, it assists in automatic music notation tasks involving multiple instruments, accurately identifying the pitch activity of various instruments in the input audio. Applicable to both single-instrument and multi-instrument audio scenarios, it can accurately extract the pitch information corresponding to each instrument in complex acoustic structures. It extracts common features between tasks while preserving the individual differences of each task. Furthermore, a feature fusion mechanism is introduced to deeply integrate the intermediate semantic features of pitch and timbre, promoting knowledge complementarity and collaborative optimization between tasks. This method is suitable for complex musical environments involving both single-instrument and multi-instrument audio.

[0043] The following describes the multi-task music notation device that integrates pitch and timbre features provided by the present invention. The multi-task music notation device that integrates pitch and timbre features described below can be referred to in correspondence with the multi-task music notation method that integrates pitch and timbre features described above.

[0044] Figure 4 This is a functional structure diagram of the multi-task music notation device that integrates pitch and timbre features provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the multi-task music notation device that integrates pitch and timbre features provided in this embodiment of the invention includes: Acquisition module 401 is used to acquire the audio signal to be recognized; Output module 402 is used to input the audio signal to be identified into the multi-task learning model framework and output the multi-instrument music notation result; The multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification. The shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified. The two auxiliary task branches are used to extract pitch features and timbre features based on the multi-scale high-level semantic features, respectively. The feature fusion module is used to perform deep fusion of the pitch features and the timbre features. The main task branch is used to output multi-instrument music notation results based on the fused features.

[0045] The multi-task music notation device merging pitch and timbre features provided in this embodiment of the invention acquires an audio signal to be identified; inputs the audio signal to be identified into a multi-task learning model framework, and outputs multi-instrument music notation results; wherein, the multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification; the shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified, the two auxiliary task branches are used to extract pitch features and timbre features based on the multi-scale high-level semantic features, respectively, the feature fusion module is used to perform deep fusion of the pitch features and the timbre features, and the main task branch is used to output multi-instrument music notation results based on the fused features. This embodiment of the invention, based on a multi-task learning framework, introduces two auxiliary tasks, pitch estimation and instrument classification, to model the frequency distribution and timbre pattern of the audio, extract the semantic representations of pitch and timbre, and explicitly inject them into the multi-instrument notation task through a fusion mechanism, thereby improving the automatic notation performance of multi-instrument polyphonic music.

[0046] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The memory 530 includes a computer program, an operating system, and acquired data. The processor 510 can call the logical instructions in the memory 530 to execute a multi-task music notation method that integrates pitch and timbre features. The method includes: acquiring an audio signal to be identified; inputting the audio signal to be identified into a multi-task learning model framework and outputting a multi-instrument music notation result; wherein, the multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification; the shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified, the two auxiliary task branches are used to extract pitch features and timbre features based on the multi-scale high-level semantic features, the feature fusion module is used to perform deep fusion of the pitch features and the timbre features, and the main task branch is used to output a multi-instrument music notation result based on the fused features.

[0047] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a multi-task music notation method for fusing pitch and timbre features provided by the above-described methods. The method includes: acquiring an audio signal to be identified; inputting the audio signal to be identified into a multi-task learning model framework and outputting a multi-instrument music notation result; wherein the multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification; the shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified, the two auxiliary task branches are respectively used to extract pitch features and timbre features based on the multi-scale high-level semantic features, the feature fusion module is used to perform deep fusion of the pitch features and the timbre features, and the main task branch is used to output a multi-instrument music notation result based on the fused features.

[0049] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0050] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-task music notation method that integrates pitch and timbre characteristics, characterized in that, include: Acquire the audio signal to be recognized; The audio signal to be identified is input into a multi-task learning model framework, and the multi-instrument music notation results are output. The multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification. The shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified. The two auxiliary task branches are used to extract pitch features and timbre features based on the multi-scale high-level semantic features, respectively. The feature fusion module is used to perform deep fusion of the pitch features and the timbre features. The main task branch is used to output multi-instrument music notation results based on the fused features.

2. The multi-task music notation method integrating pitch and timbre features according to claim 1, characterized in that, The shared feature extraction module is built based on the U-Net encoder. This module is used to extract multi-scale high-level semantic features of the audio signal to be identified, including: Multi-scale high-level semantic features of the spectrogram corresponding to the audio signal to be identified are extracted by multi-layer two-dimensional convolution and pooling operations.

3. The multi-task music notation method integrating pitch and timbre features according to claim 2, characterized in that, The method for obtaining the spectrogram corresponding to the audio signal to be identified includes: The audio signal to be identified is converted into a mono format and resampled to a preset sampling rate; Based on the mono format audio with the preset sampling rate, the spectrogram corresponding to the audio signal to be identified is calculated using the constant Q transform algorithm.

4. The multi-task music notation method integrating pitch and timbre features according to claim 1, characterized in that, The two auxiliary task branches are respectively used to extract pitch features and timbre features based on the multi-scale high-level semantic features, including: The multi-scale high-level semantic features are respectively input into the pitch estimation branch and the instrument classification branch; In the pitch estimation branch, pitch-related features are extracted through the first convolutional layer, and 256-channel intermediate pitch features are output; the second convolutional layer focuses on pitch discriminative features; and the first convolutional output layer maps the features to 88-dimensional pitch activation probabilities. In the instrument classification branch, timbre-related features are extracted through the third convolutional layer, and 256-channel intermediate timbre features are output. These intermediate timbre features encode both instrument category information and timbre characteristics. The discriminative features of the instrument are enhanced through the fourth convolutional layer. The features are then mapped to 11-dimensional instrument classification probabilities through the second convolutional output layer.

5. The multi-task music notation method integrating pitch and timbre features according to claim 1, characterized in that, The feature fusion module is used to perform deep fusion of the pitch feature and the timbre feature, including: The multi-scale high-level semantic features output by the shared feature extraction module are converted into a two-dimensional matrix as a query vector. The intermediate pitch features and intermediate timbre features output by the pitch estimation branch and the instrument classification branch are concatenated to form an auxiliary feature matrix; The auxiliary feature matrix is ​​mapped into a key matrix and a value matrix through convolution, respectively; A mask matrix is ​​constructed based on the coupling relationship between pitch and timbre; A multi-head attention mechanism is adopted, which projects the query vector, key matrix and value matrix into multiple subspaces respectively. In each attention head, the attention weights of the query vector and key matrix are calculated. After softmax normalization of the mask matrix, the weights are multiplied with the value matrix to obtain the attention features output by each head. The attention features output from each head are concatenated and fused through a linear transformation to obtain a comprehensive attention feature; The integrated attention features and the multi-scale high-level semantic features are residually connected, and the residual connection results are normalized to obtain the fused features.

6. The multi-task music notation method integrating pitch and timbre features according to claim 2, characterized in that, The main task branch is constructed based on the U-Net decoder, which is symmetrical to the U-Net encoder. This main task branch is used to output multi-instrument musical notation results based on the fusion result, including: The fused features are used as direct input to the first layer of the decoder to provide an initial feature representation containing pitch-timbre coordination information; During each layer of decoding, skip connections are used to combine with the features of the corresponding coding layer to supplement detailed information at different scales on the basis of the initial feature representation. The output layer outputs multi-instrument music notation results based on the detailed information at different scales. The multi-instrument music notation results are three-dimensional tensor representations of instrument category, pitch information and time position.

7. The multi-task music notation method integrating pitch and timbre features according to claim 1, characterized in that, The multi-task learning model framework is trained based on a multi-task joint loss function, including: The multi-task joint loss function includes the binary cross-entropy loss of the pitch estimation branch, the binary cross-entropy loss of the instrument classification branch, and the binary cross-entropy loss of the multi-instrument notation main branch.

8. A multi-task music notation device that integrates pitch and timbre characteristics, characterized in that, include: The acquisition module is used to acquire the audio signal to be recognized; The output module is used to input the audio signal to be identified into the multi-task learning model framework and output the multi-instrument music notation results; The multi-task learning model framework includes a shared feature extraction module, a feature fusion module, a main task multi-instrument music notation branch, and two auxiliary task branches for pitch estimation and instrument classification. The shared feature extraction module is used to extract multi-scale high-level semantic features of the audio signal to be identified. The two auxiliary task branches are used to extract pitch features and timbre features based on the multi-scale high-level semantic features, respectively. The feature fusion module is used to perform deep fusion of the pitch features and the timbre features. The main task branch is used to output multi-instrument music notation results based on the fused features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-task music notation method that integrates pitch and timbre features as described in any one of claims 1 to 7.

10. A non-transitory readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the multi-task music notation method that integrates pitch and timbre features as described in any one of claims 1 to 7.

Images