Piano audio data processing method and device, electronic equipment and storage medium

Abstract

The piano audio data processing method and device, the electronic equipment and the storage medium provided by the embodiment of the present application relate to the technical field of artificial intelligence. The method comprises the following steps: dividing a target piano spectrum according to a preset frame length to obtain an original spectrum slice; performing embedding representation on the original spectrum slice to obtain a spectrum slice embedding vector, and performing position coding on the original spectrum slice to obtain a spectrum slice position vector; performing vector addition on the spectrum slice embedding vector and the spectrum slice position vector to obtain a spectrum slice target representation vector; merging the spectrum slice target representation vector to obtain a spectrum vector sequence; inputting the spectrum vector sequence into a preset attention model to extract sequence features and obtain a note context vector sequence; inputting the note context vector sequence into a preset note prediction model to predict sequence features and obtain piano note data. The embodiment of the present application can improve the note recognition accuracy of the piano audio.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a piano audio data processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] An Auto Piano Transcription (APT) task refers to a process of transcribing an input piano music into a symbolic representation of a piano score. Existing APT techniques generally adopt a template method. The template method compares the similarity between an input sound segment and a corresponding template in a database to determine the note data of the input sound, such as the note pitch. The advantage of the template method is that it requires less computing resources and has high detection efficiency. However, the template method has poor detection effect for music with many chords or high speed, because the template method identifies notes by template matching. For example, as a representative of multi-pitch musical instruments, piano audio often has many chords and high speed, and the template method has poor accuracy in identifying notes of piano audio.

[0003] Therefore, how to provide a piano audio data processing method to improve the note recognition accuracy of piano audio has become a technical problem to be solved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a piano audio data processing method and device, an electronic device, and a storage medium, which can improve the note recognition accuracy of piano audio.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application provides a piano audio data processing method, which comprises:

[0006] obtaining a spectrum of piano audio to obtain a target piano spectrum;

[0007] dividing the target piano spectrum according to a preset frame length to obtain an original spectrum slice;

[0008] performing embedding representation on the original spectrum slice to obtain a spectrum slice embedding vector, and performing position coding on the original spectrum slice to obtain a spectrum slice position vector;

[0009] performing vector addition on the spectrum slice embedding vector and the spectrum slice position vector to obtain a spectrum slice target representation vector;

[0010] merging the spectrum slice target representation vector to obtain a spectrum vector sequence;

[0011] inputting the spectrum vector sequence into a preset attention model to extract sequence features to obtain a note context vector sequence;

[0012] The note context vector sequence is input into a preset note prediction model to predict sequence features, thereby obtaining piano note data.

[0013] In some embodiments, the embedding representation of the original spectral slice to obtain a spectral slice embedding vector includes:

[0014] Obtain the spectrum slice adjacent to the original spectrum slice to obtain the selected spectrum slice;

[0015] The original spectrum slice is sampled to obtain a first sampled spectrum slice, and the selected spectrum slice is sampled to obtain a second sampled spectrum slice;

[0016] The first sampled spectrum slice and the first sampled spectrum slice are merged to obtain the spectrum slice embedding vector.

[0017] In some embodiments, the step of performing spectral sampling on the original spectral slice to obtain a first sampled spectral slice, and performing spectral sampling on a selected spectral slice to obtain a second sampled spectral slice, includes:

[0018] The original spectrum slice is sampled by a preset spectrum filter to obtain the first sampled spectrum slice.

[0019] If the selected spectrum slice is after the original spectrum slice, the center point of the spectrum filter is increased by a preset displacement length to obtain a shifted spectrum filter; or, if the selected spectrum slice is before the original spectrum slice, the center point of the spectrum filter is decreased by the preset displacement length to obtain the shifted spectrum filter.

[0020] The selected spectrum slice is sampled a second time by the shift spectrum filter to obtain the second sampled spectrum slice.

[0021] In some embodiments, merging the first sampled spectrum slice and the first sampled spectrum slice to obtain the spectrum slice embedding vector includes:

[0022] The first sampled spectrum slice is vectorized to obtain the first spectrum slice vector;

[0023] The second sampled spectrum slice is vector-transformed to obtain the second spectrum slice vector;

[0024] The first spectrum slice vector and the first spectrum slice vector are merged to obtain the spectrum slice embedding vector.

[0025] In some embodiments, the step of merging the first spectral slice vector and the first spectral slice vector to obtain the spectral slice embedding vector includes:

[0026] The first spectrum slice vector is input into a preset weight model to calculate the weight, thereby obtaining the first weight;

[0027] The second spectral slice vector is input into the weighting model to calculate the weight, thereby obtaining the second weight; wherein the first weight is greater than the second weight.

[0028] The first spectral slice vector and the second spectral slice vector are weighted and summed according to the first weight and the second weight to obtain the spectral slice embedding vector.

[0029] In some embodiments, obtaining the spectrum of the piano audio to obtain the target piano spectrum includes:

[0030] The initial piano spectrum is obtained by extracting the spectrum from the piano audio data using a preset Mel filter bank.

[0031] The initial piano spectrum is convolved by a preset convolutional layer to obtain a first intermediate spectrum;

[0032] The first intermediate spectrum is normalized by a preset normalization layer to obtain the second intermediate spectrum;

[0033] The target piano spectrum is obtained by performing max pooling on the second intermediate spectrum through a preset pooling layer.

[0034] In some embodiments, the attention model includes a gated position self-attention layer, a local feedforward layer, a multi-head self-attention layer, and a feedforward network layer. The step of inputting the spectral vector sequence into a preset attention model for sequence feature extraction to obtain a note context vector sequence includes:

[0035] Attention calculations are performed on the spectral vector sequence through the gated position self-attention layer to obtain the first attention vector sequence;

[0036] The second attention vector sequence is obtained by extracting features from the first attention vector sequence through the local feedforward layer.

[0037] The third attention vector sequence is obtained by performing attention calculation on the second attention vector sequence through the multi-head self-attention layer;

[0038] The third attention vector sequence is used to extract features from the feedforward network layer to obtain the note context vector sequence.

[0039] To achieve the above objectives, a second aspect of this application provides a piano audio data processing apparatus, the apparatus comprising:

[0040] The spectrum acquisition module is used to acquire the spectrum of the piano audio to obtain the spectrum of the target piano.

[0041] The spectrum segmentation module is used to segment the spectrum of the target piano according to a preset frame length to obtain the original spectrum slice;

[0042] The vector representation module is used to embed the original spectrum slice to obtain a spectrum slice embedding vector, and to perform position encoding on the original spectrum slice to obtain a spectrum slice position vector.

[0043] The vector addition module is used to add the spectrum slice embedding vector and the spectrum slice position vector to obtain the spectrum slice target representation vector;

[0044] The vector merging module is used to merge the target representation vectors of the spectrum slices to obtain a spectrum vector sequence;

[0045] The sequence feature extraction module is used to input the spectrum vector sequence into a preset attention model to extract sequence features and obtain a note context vector sequence.

[0046] The sequence feature prediction module is used to input the note context vector sequence into a preset note prediction model to perform sequence feature prediction and obtain piano note data.

[0047] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0048] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0049] This application proposes a piano audio data processing method, apparatus, electronic device, and storage medium. The method obtains a target representation vector for each original spectral slice by embedding and positionally encoding the slices, thus preserving local spectral information as much as possible. Sequence feature extraction using an attention model analyzes the relationships between long-distance spectral slices, fully considering local information deviations between slices to obtain a note context vector sequence. Then, a note prediction model predicts the sequence features of the note context sequence to obtain piano note data. This application involves two sequences: a spectral vector sequence and a note context sequence. It also involves two models: an attention model and a note prediction model. By employing a sequence-to-sequence structure and a two-stage neural network model, this application can capture long-distance dependencies in piano audio data, thereby improving note recognition accuracy. Attached Figure Description

[0050] Figure 1 This is a flowchart of the piano audio data processing method provided in the embodiments of this application;

[0051] Figure 2 yes Figure 1 The flowchart of step S101 in the text;

[0052] Figure 3 yes Figure 1 The flowchart of step S103 in the process;

[0053] Figure 4 yes Figure 3 The flowchart of step S302 in the document;

[0054] Figure 5 yes Figure 3 The flowchart of step S303 in the process;

[0055] Figure 6 yes Figure 1 The flowchart of step S106 in the process;

[0056] Figure 7 This is a block diagram of the module structure of the piano audio data processing device provided in the embodiments of this application;

[0057] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] First, let's analyze some of the terms used in this application:

[0062] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0063] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0064] Automatic Music Transcription (AMT) is the core task of music information retrieval, which aims to convert music into a symbolic representation of note events including pitch, onset, offset, and tempo.

[0065] Auto Piano Transcription (APT) refers to the process of transcribing an input piano piece into a symbolic representation of the piano score.

[0066] Currently, technologies for automatic music transcription based on deep learning methods have emerged. However, existing deep learning methods generally convert the music audio waveform into a short-time Fourier transform spectrum or Mel spectrum, then treat the input spectrum as an image, extract features based on a convolutional neural network structure, and finally use a fully connected classifier for classification to obtain frame-by-frame pitch classification. This approach is generally suitable for single-note audio and requires a large amount of high-quality datasets for training.

[0067] Another limitation of automated music transcription tasks is that existing open-source music transcription datasets only contain one to several hundred hours of audio. Dataset sizes vary significantly across different domains of music transcription; for example, the piano dataset is tens of times larger than those for other orchestral instruments. Therefore, when existing music transcription models are directly applied to APT, the accuracy of note recognition for piano audio is often low.

[0068] Therefore, there is an urgent need for an automatic piano transcription technology that can improve the accuracy of note recognition. The piano audio data processing method provided in this application can improve the accuracy of piano note recognition.

[0069] The piano audio data processing method provided in this application embodiment can be applied to a server or terminal, or it can be software running on a server. The server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or it can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the piano audio data processing method, etc., but is not limited to the above forms.

[0070] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0071] This application's embodiments can be applied in various scenarios, such as intelligent music teaching, music information retrieval systems, song recognition, and humming / singing retrieval. It is primarily used to detect whether the input piano sound matches the sheet music.

[0072] This application provides a piano audio data processing method, a piano audio data processing device, an electronic device, and a storage medium. The specific embodiments are described below. First, the piano audio data processing method in this application is described.

[0073] Figure 1 This is an optional flowchart of the piano audio data processing method provided in the embodiments of this application, which may include, but is not limited to, steps S101 to S107.

[0074] Step S101: Obtain the spectrum of the piano audio to obtain the spectrum of the target piano;

[0075] Step S102: The spectrum of the target piano is segmented according to the preset frame length to obtain the original spectrum slice;

[0076] Step S103: Embed the original spectrum slice to obtain the spectrum slice embedding vector, and encode the original spectrum slice to obtain the spectrum slice position vector.

[0077] Step S104: Add the spectrum slice embedding vector and the spectrum slice position vector together to obtain the spectrum slice target representation vector;

[0078] Step S105: Merge the target representation vectors of the spectrum slices to obtain a spectrum vector sequence;

[0079] Step S106: Input the spectrum vector sequence into the preset attention model to extract sequence features and obtain the note context vector sequence;

[0080] Step S107: Input the note context vector sequence into the preset note prediction model to predict the sequence features and obtain piano note data.

[0081] Steps S101 to S107 as shown in the embodiments of this application obtain the target representation vector of the spectrum slice by embedding representation and position encoding of each original spectrum slice, which can preserve the local information of the spectrum as much as possible; the sequence feature extraction by the attention model can analyze the relationship between long-distance spectrum slices, fully consider the local information deviation between spectrum slices, and obtain the note context vector sequence; then the note prediction model is used to predict the sequence features of the note context sequence to obtain piano note data.

[0082] It should be noted that this application involves two sequences: a spectral vector sequence and a note context sequence; it also involves two models: an attention model and a note prediction model. This application, through its sequence-to-sequence structure and two-stage neural network model, can capture the long-distance dependencies of piano audio data, thereby improving note recognition accuracy. Especially when the piano audio data in this application is a complete piano piece, the method can analyze the overall melody distribution to obtain more accurate note data.

[0083] In step S101 of some embodiments, the spectrum of the piano audio is obtained to obtain the spectrum of the target piano.

[0084] Piano audio refers to music played on a piano. Piano audio can be downloaded from a server; specifically, a piano audio dataset is downloaded from the server, and the piano audio data required for this application embodiment is obtained from the piano audio dataset. Piano audio can also be obtained through live recording; specifically, piano audio emitted by a piano instrument is recorded using a microphone to obtain the piano audio data required for this application embodiment. The piano audio data in this application embodiment can be a complete piano piece, a segment of a complete piano piece cut to a fixed duration, or a climax segment of a complete piano piece.

[0085] In one embodiment, piano audio data may contain noise, which is detrimental to note recognition. Noise reduction preprocessing can be performed on the noisy piano audio data. A noise reduction model can be used to process the piano audio data to obtain clean piano audio data. Specifically, a deep recurrent neural network (DRNN) model can be selected, with three hidden layers. Increasing the number of hidden layers can improve the approximation speed of noisy audio data to clean audio data in each iteration. During the training of the DRNN, clean audio data and output audio data are required. Then, based on the loss between the output audio data and the clean audio data, the model parameters are adjusted until the loss converges, resulting in the trained DRNN. The loss function can be measured by the mean of the squared vector differences between the clean audio data and the output audio data.

[0086] After obtaining the piano audio data, the piano spectrum needs to be extracted from it. For example, this can be achieved by performing a short-time Fourier transform on the piano audio data. Alternatively, a preset Mel filter bank can be used to extract the spectrum from the piano audio data to obtain the target piano spectrum.

[0087] Piano audio data contains rich harmonic information, and a single spectrum is insufficient to accurately and comprehensively represent the information carried by the piano audio data, leading to the loss of some important information features and inaccurate note recognition. Therefore, in addition to conventional spectrum extraction methods, this application proposes a spectrum extraction method that can improve the accuracy of note recognition.

[0088] Specifically, refer to Figure 2 Step S101 specifically includes, but is not limited to, steps S201 to S204:

[0089] Step S201: Extract the spectrum of the piano audio data using a preset Mel filter bank to obtain the initial piano spectrum;

[0090] Step S202: The initial piano spectrum is convolved by a preset convolutional layer to obtain the first intermediate spectrum;

[0091] Step S203: Normalize the first intermediate spectrum through a preset normalization layer to obtain the second intermediate spectrum;

[0092] Step S204: Max pooling is performed on the second intermediate spectrum through a preset pooling layer to obtain the target piano spectrum.

[0093] Steps S201 to S204, as illustrated in this embodiment, extract a first intermediate spectrum with frame-level features from the initial piano spectrum using a convolutional layer, resulting in richer spectral information. Normalization is performed using a normalization layer to improve processing efficiency. Max pooling is then performed using a pooling layer to preserve significant features in the spectral information, thus obtaining the target piano spectrum. Through these spectrum extraction methods, the obtained target piano spectrum better represents the note information contained in the piano audio data, contributing to improved note recognition accuracy.

[0094] In step S102 of some embodiments, the target piano spectrum is segmented according to a preset frame length to obtain original spectrum slices. Adjacent original spectrum slices do not overlap. The target piano spectrum can be a Mel spectrum with T frames. The preset frame length is used to represent the slice size. For example, each slice consists of 512 frames, so the original spectrum slice has 512 frames. If the last original spectrum slice has less than 512 frames, it is padded with zeros.

[0095] In step S103 of some embodiments, the original spectrum slice is embedded to obtain a spectrum slice embedding vector, and the original spectrum slice is position-encoded to obtain a spectrum slice position vector.

[0096] Specifically, refer to Figure 3 Step S103 specifically includes, but is not limited to, steps S301 to S303:

[0097] Step S301: Obtain the spectrum slice adjacent to the original spectrum slice to obtain the selected spectrum slice;

[0098] Step S302: Perform spectrum sampling on the original spectrum slice to obtain a first sampled spectrum slice, and perform spectrum sampling on the selected spectrum slice to obtain a second sampled spectrum slice;

[0099] Step S303: Merge the first sampled spectrum slice and the first sampled spectrum slice to obtain the spectrum slice embedding vector.

[0100] Steps S301 to S303 shown in this embodiment aim to further utilize the spatial relationship between adjacent original spectrum slices. This is because this embodiment focuses on piano audio, and as a representative of multi-tonal instruments, piano audio often features many chords and a fast tempo. Therefore, by fully considering the spatial relationship between adjacent spectrum slices through steps S301 to S303, it is possible to improve the accuracy of note recognition.

[0101] It should be noted that the target piano spectrum is divided into multiple raw spectrum slices, which are arranged sequentially. Taking the multiple raw spectrum slices arranged from left to right as an example: for the first raw spectrum slice, the adjacent spectrum slices are only the spectrum slices in the right position; for the last raw spectrum slice, the adjacent spectrum slices are only the spectrum slices in the left position; for the middle raw spectrum slice, the adjacent spectrum slices include the spectrum slices in the left position and the spectrum slices in the right position.

[0102] In one embodiment, reference is made to Figure 4 Step S302 specifically includes, but is not limited to, steps S401 to S403:

[0103] Step S401: Perform first spectrum sampling on the original spectrum slice using a preset spectrum filter to obtain a first sampled spectrum slice;

[0104] Step S402: If the selected spectrum slice is after the original spectrum slice, the center point of the spectrum filter is increased by a preset displacement length to obtain a shifted spectrum filter; or, if the selected spectrum slice is before the original spectrum slice, the center point of the spectrum filter is decreased by a preset displacement length to obtain a shifted spectrum filter.

[0105] Step S403: Perform a second spectrum sampling on the selected spectrum slice using a shift spectrum filter to obtain a second sampled spectrum slice.

[0106] Steps S401 to S403 as shown in the embodiments of this application involve shifting the spectrum filter by adding or reducing a preset displacement length to the center point of the spectrum filter, thus obtaining a shifted filter. The first spectrum sampling is performed through the spectrum filter, and the second spectrum sampling is performed through the shifted filter. This achieves spectrum sampling while ensuring the sampling consistency of the spectrum filter between adjacent spectrum slices.

[0107] It should be noted that, taking multiple original spectrum slices arranged from left to right as an example, if the selected spectrum slice is after the original spectrum slices, it means that the selected spectrum slice is located to the right of the original spectrum slices. In this case, the center point of the spectrum filter is increased by a preset displacement length, so that the shifted spectrum filter can sample the selected spectrum slice located to the right. If the selected spectrum slice is before the original spectrum slices, it means that the selected spectrum slice is located to the left of the original spectrum slices. In this case, the center point of the spectrum filter is decreased by a preset displacement length, so that the shifted spectrum filter can sample the selected spectrum slice located to the left.

[0108] In one embodiment, reference is made to Figure 5 Step S303 specifically includes, but is not limited to, steps S501 to S503:

[0109] Step S501: Perform vector transformation on the first sampled spectrum slice to obtain the first spectrum slice vector;

[0110] Step S502: Perform vector transformation on the second sampled spectrum slice to obtain the second spectrum slice vector;

[0111] Step S503: The first spectrum slice vector and the first spectrum slice vector are merged to obtain the spectrum slice embedding vector.

[0112] It's important to note that a vector is an array of values ​​across different dimensions; it represents a point in a multidimensional space. The line segment connecting this point to the origin of the multidimensional coordinate system has magnitude and direction, which are the magnitude and direction of the vector. Each of the aforementioned values ​​represents the point value projected onto the corresponding coordinate axis in the multidimensional coordinate system—that is, the vector. Vectors can be numerical or symbolic, but those used as input to a model are generally numerical.

[0113] Steps S501 to S503, as shown in the embodiments of this application, convert the first sampled spectrum slice into a first spectrum slice vector and the second sampled spectrum slice into a second spectrum slice vector, and then merge the vectors to obtain a spectrum slice embedding vector so that the model can process it.

[0114] In step S503, the merging method can be to directly add the first spectrum slice vector and the second spectrum slice vector to obtain the spectrum slice embedding vector. In another embodiment, considering that the first spectrum slice vector and the second spectrum slice vector have different levels of importance, for example, in this embodiment, the first spectrum slice vector is more important, so a first weight can be applied to the first spectrum slice vector and a second weight can be applied to the second spectrum slice vector, with the first weight being greater than the second weight, and then a weighted sum is performed to obtain the spectrum slice embedding vector.

[0115] The first weight and the second weight can be calculated using a weighting model. In one embodiment, step S503 includes: inputting the first spectral slice vector into a preset weighting model to calculate the weight, thereby obtaining the first weight; inputting the second spectral slice vector into the weighting model to calculate the weight, thereby obtaining the second weight; and performing a weighted summation of the first spectral slice vector and the second spectral slice vector according to the first weight and the second weight to obtain the spectral slice embedding vector.

[0116] A weighted model refers to assigning different weights to different pieces of information needed to solve a problem in a specific scenario. Information that is more helpful to the problem is given higher weights, while information that is less helpful is given lower weights, thus making better use of this information to solve the problem. Specifically, in this embodiment, the first spectral slice vector is obtained by transforming the original spectral slice, while the second spectral slice vector is obtained by transforming a selected spectral slice (which is adjacent to the original spectral slice). The selected spectral slice is used to provide spatial relationship information to the original spectral slice, but its contribution to a problem like note recognition is relatively small. Therefore, a first weight is applied to the first spectral slice vector, and a second weight is applied to the second spectral slice vector, with the first weight being greater than the second weight.

[0117] In step S104 of some embodiments, the spectrum slice embedding vector and the spectrum slice position vector are vector-added to obtain the spectrum slice target representation vector.

[0118] In one example, step S103 flattens each original spectrum slice into a one-dimensional vector representation and performs position encoding on the original spectrum slice to obtain a spectrum slice position vector; then step S104 adds the one-dimensional vector representation and the position vector to obtain the spectrum slice target representation vector.

[0119] In step S105 of some embodiments, the target representation vectors of the spectrum slices are merged to obtain a spectrum vector sequence. Specifically, the target representation vectors of the spectrum slices corresponding to several original spectrum slices are sequentially concatenated to obtain the spectrum vector sequence.

[0120] In step S106 of some embodiments, the spectrum vector sequence is input into a preset attention model to extract sequence features and obtain a note context vector sequence.

[0121] In one embodiment, the attention model includes a gated position self-attention layer, a local feedforward layer, a multi-head self-attention layer, and a feedforward neural network layer, as shown below. Figure 6 Step S106 specifically includes, but is not limited to, steps S601 to S604:

[0122] Step S601: Perform attention calculation on the spectrum vector sequence through the gated position self-attention layer to obtain the first attention vector sequence;

[0123] Step S602: Extract features from the first attention vector sequence through a local feedforward layer to obtain the second attention vector sequence;

[0124] Step S603: Perform attention calculation on the second attention vector sequence through a multi-head self-attention layer to obtain the third attention vector sequence;

[0125] Step S604: Feature extraction is performed on the third attention vector sequence through a feedforward network layer to obtain the note context vector sequence.

[0126] Steps S501 to S503, as shown in the embodiments of this application, reshape the matching vector sequence using an attention model. The resulting note context sequence better represents the note information of the piano audio data, thereby improving the accuracy of subsequent note recognition.

[0127] Specifically, the attention model in this application embodiment is constructed using a local-global architecture, which includes local branches and global branches. The local branches include a gated position self-attention layer (GPSA layer) and a local feed-forward layer (LFF layer). The global branches include a multi-head self-attention layer (MSA layer) and a feed-forward network layer (FFN layer).

[0128] GPSA is a form of positional self-attention that can be equipped with "soft" convolutional inductive bias. GPSA layers are initialized to mimic the locality of convolutional layers, and then the balance between standard content-based self-attention and convolutionally initialized positional self-attention is controlled by adjusting the gating parameters that regulate attention to positional and content information, allowing each attention head to freely jump out of local features. The GPSA and LFF layers help the attention model provide local regions to exchange local features, enhancing the locality of prior information and inductive bias.

[0129] In the local branch, the GPSA layer simulates the locality of convolution, allowing the acquisition of local information from the spectral vector sequence. The LFF layer provides local regions for information exchange, further enhancing the ability to exchange local information. In the global branch, the multi-head self-attention layer ignores positional information, while the feedforward network layer, also known as the position feedforward network, is used to perceive global positional information, improving information extraction capabilities.

[0130] In step S107 of some embodiments, the note context vector sequence is input into a preset note prediction model to predict sequence features and obtain piano note data.

[0131] Specifically, the note prediction model can be a Convolutional Neural Network (CNN) or a Long Short-Term Memory (LSTM) neural network. Taking a CNN as an example, a CNN typically consists of an input layer, multiple hidden layers, and an output layer. The input layer has multiple processing nodes, each with a weight matrix (convolutional kernel). Each hidden layer also has multiple processing nodes, each with a weight matrix (convolutional kernel). The output layer also has multiple processing nodes, each with a weight matrix (convolutional kernel). Each processing node in the input layer convolves the CNN's input vector with its own convolutional kernel to obtain its output, which is then used as the input to the processing nodes in the next hidden layer. Each processing node in the next hidden layer convolves the outputs of the previous layer's processing nodes with its own convolutional layer to obtain its output, which is then used as the input to the processing nodes in the next lower hidden layer. This process continues until each processing node in the output layer convolves the outputs of the last hidden layer's processing nodes with its own convolutional layer to obtain the piano note data.

[0132] In addition to using CNNs, this application proposes a note prediction model that can further improve the accuracy of note recognition. In this embodiment, the note prediction model includes multiple sequentially connected processing layers and a fully connected layer. The first processing layer receives a sequence of note context vectors. Each processing layer outputs its processed result to the next processing layer as input. The last processing layer outputs its processed result to the fully connected layer, which generates piano note data. Each processing layer includes a multi-head attention sublayer, a residual connection sublayer, a normalization sublayer, and a feedforward network sublayer. Therefore, step S107 specifically includes: performing attention calculation on the note context vector sequence through the multi-head attention sublayer to obtain a first attention note vector sequence; connecting the note context vector sequence and the first attention note vector sequence through the residual connection sublayer to obtain a second attention note vector sequence; normalizing the second attention note vector sequence through the normalization sublayer to obtain a third attention note vector sequence; extracting features from the third attention note vector sequence through the feedforward network sublayer to obtain a fourth attention note vector sequence; and classifying the fourth attention note vector sequence through the fully connected layer to obtain piano note data.

[0133] It should be noted that the piano note data obtained in step S107 includes at least one of the following: note start time data, note offset time data, note pitch data, and note dynamics data.

[0134] Taking piano note data, including note start time data, note offset time data, note pitch data, and note velocity data, as an example, before step S107, the piano audio data processing method of this application embodiment further includes:

[0135] Training the note prediction model specifically includes:

[0136] Obtain the spectrum of the sample piano audio;

[0137] The sample piano spectrum is segmented according to the preset frame length to obtain sample spectrum slices;

[0138] The sample spectrum slices are embedded to obtain the sample spectrum slice embedding vector, and the sample spectrum slices are position encoded to obtain the sample spectrum slice position vector.

[0139] The sample spectrum slice embedding vector and the sample spectrum slice position vector are added together to obtain the sample spectrum slice target representation vector;

[0140] The target representation vectors of the sample spectrum slices are merged to obtain the sample spectrum vector sequence;

[0141] The sample spectrum vector sequence is input into the attention model for sequence feature extraction to obtain the sample note context vector sequence.

[0142] The sample note context vector sequence is input into the initial prediction model to predict sequence features and obtain the predicted note data; wherein, the predicted note data includes the predicted note start time data, the predicted note offset time data, the predicted note pitch data, and the predicted note dynamics data;

[0143] The first loss data is obtained by calculating the loss based on the label note start time data and the predicted note start time data; the second loss data is obtained by calculating the loss based on the label note offset time data and the predicted note offset time data; the third loss data is obtained by calculating the loss based on the label note pitch data and the predicted note pitch data; and the fourth loss data is obtained by calculating the loss based on the label note dynamics data and the predicted note dynamics data.

[0144] The first loss data, the second loss data, the third loss data, and the fourth loss data are merged to obtain the target loss data;

[0145] The initial prediction model is adjusted based on the target loss data to obtain the note prediction model.

[0146] Specifically, the initial prediction model and the note prediction model have the same structure but different parameters. After adjusting the parameters of the initial prediction model, the note prediction model used for note recognition in this embodiment of the application is obtained. The formula for calculating the loss can be the cross-entropy loss function or other loss functions, and this embodiment of the application does not specifically limit it.

[0147] It should be noted that this application embodiment identifies notes from piano audio data. However, the representation of notes in piano audio is extremely complex, often requiring a large dataset for training the note prediction model, thus making model training difficult. Therefore, this application embodiment proposes a training method that reduces the amount of data required for model training.

[0148] Specifically, this application involves two models: an attention model for extracting sequence features to obtain a note context vector sequence, and a note prediction model for predicting sequence features from the note context sequence to obtain piano audio data. To reduce the amount of data required for model training, this application improves the attention model, thereby improving the obtained note context vector sequence and thus increasing the note prediction accuracy of the note prediction model. This achieves the performance of a large dataset with only a small dataset. The improvement to the attention model mainly involves adding a pre-training process, specifically including: acquiring the spectrum of sample piano audio data to obtain the sample piano spectrum; randomly masking the sample piano spectrum to obtain a fragmented spectrum and a masked spectrum; inputting the fragmented spectrum into the initial feature extraction model for spectrum prediction to obtain the predicted spectrum; calculating the loss based on the masked spectrum and the predicted spectrum to obtain spectrum loss data; and adjusting the parameters of the initial feature extraction model based on the spectrum loss data to obtain the attention model. After pre-training, the attention model only requires a small amount of labeled data for fine-tuning the note prediction model during training, significantly reducing the amount of training data and improving the model training speed.

[0149] Please see Figure 7 This application also provides a piano audio data processing apparatus, which can implement the above-described piano audio data processing method. Figure 7The block diagram of the module structure of the piano audio data processing device provided in the embodiments of this application is shown. The device includes: a spectrum acquisition module 701, a spectrum segmentation module 702, a vector representation module 703, a vector addition module 704, a vector merging module 705, a sequence feature extraction module 706, and a sequence feature prediction model 707. The system comprises the following modules: a spectrum acquisition module 701, used to acquire the spectrum of piano audio to obtain the target piano spectrum; a spectrum segmentation module 702, used to segment the target piano spectrum according to a preset frame length to obtain original spectrum slices; a vector representation module 703, used to embed the original spectrum slices to obtain spectrum slice embedding vectors, and to encode the positions of the original spectrum slices to obtain spectrum slice position vectors; a vector addition module 704, used to add the spectrum slice embedding vectors and spectrum slice position vectors to obtain the target representation vector of the spectrum slices; a vector merging module 705, used to merge the target representation vectors of the spectrum slices to obtain a spectrum vector sequence; a sequence feature extraction module 706, used to input the spectrum vector sequence into a preset attention model for sequence feature extraction to obtain a note context vector sequence; and a sequence feature prediction module 707, used to input the note context vector sequence into a preset note prediction model for sequence feature prediction to obtain piano note data.

[0150] It should be noted that the specific implementation of the piano audio data processing device is basically the same as the specific embodiment of the piano audio data processing method described above, and will not be repeated here.

[0151] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned piano audio data processing method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0152] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0153] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0154] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the piano audio data processing method of the embodiments of this application.

[0155] The 803 input / output interface is used to implement information input and output.

[0156] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0157] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0158] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0159] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described piano audio data processing method.

[0160] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0161] The piano audio data processing method, apparatus, electronic device, and storage medium provided in this application obtain target representation vectors for each original spectral slice by embedding and positionally encoding the slices, thus preserving local spectral information as much as possible. Sequence feature extraction using an attention model analyzes the relationships between long-distance spectral slices, fully considering local information deviations between slices to obtain a note context vector sequence. Then, a note prediction model predicts the sequence features of the note context sequence to obtain piano note data. This application, through a sequence-to-sequence structure and a two-stage neural network model, captures long-distance dependencies in piano audio data, improving note recognition accuracy.

[0162] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0163] It will be understood by those skilled in the art that Figures 1-6 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0166] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0167] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for processing piano audio data, characterized in that, The method includes: Obtain the spectrum of the piano audio to get the spectrum of the target piano; The spectrum of the target piano is segmented according to a preset frame length to obtain the original spectrum slice; The original spectrum slice is embedded to obtain a spectrum slice embedding vector, and the original spectrum slice is position-encoded to obtain a spectrum slice position vector. The spectrum slice embedding vector and the spectrum slice position vector are added together to obtain the spectrum slice target representation vector; The target representation vectors of the spectrum slices are merged to obtain a spectrum vector sequence; The spectral vector sequence is input into a preset attention model for sequence feature extraction to obtain a note context vector sequence; The note context vector sequence is input into a preset note prediction model to predict sequence features, thereby obtaining piano note data.

2. The method according to claim 1, characterized in that, The process of embedding the original spectral slice to obtain a spectral slice embedding vector includes: Obtain the spectrum slice adjacent to the original spectrum slice to obtain the selected spectrum slice; The original spectrum slice is sampled to obtain a first sampled spectrum slice, and the selected spectrum slice is sampled to obtain a second sampled spectrum slice; The first sampled spectrum slice and the first sampled spectrum slice are merged to obtain the spectrum slice embedding vector.

3. The method according to claim 2, characterized in that, The step of performing spectral sampling on the original spectral slice to obtain a first sampled spectral slice, and performing spectral sampling on a selected spectral slice to obtain a second sampled spectral slice, includes: The original spectrum slice is sampled by a preset spectrum filter to obtain the first sampled spectrum slice. If the selected spectrum slice is after the original spectrum slice, the center point of the spectrum filter is increased by a preset displacement length to obtain a shifted spectrum filter; or, if the selected spectrum slice is before the original spectrum slice, the center point of the spectrum filter is decreased by the preset displacement length to obtain the shifted spectrum filter. The selected spectrum slice is sampled a second time by the shift spectrum filter to obtain the second sampled spectrum slice.

4. The method according to claim 3, characterized in that, The step of merging the first sampled spectrum slice and the first sampled spectrum slice to obtain the spectrum slice embedding vector includes: The first sampled spectrum slice is vectorized to obtain the first spectrum slice vector; The second sampled spectrum slice is vector-transformed to obtain the second spectrum slice vector; The first spectrum slice vector and the first spectrum slice vector are merged to obtain the spectrum slice embedding vector.

5. The method according to claim 4, characterized in that, The step of merging the first spectral slice vector and the first spectral slice vector to obtain the spectral slice embedding vector includes: The first spectrum slice vector is input into a preset weight model to calculate the weight, thereby obtaining the first weight; The second spectral slice vector is input into the weighting model to calculate the weight, thereby obtaining the second weight; wherein the first weight is greater than the second weight. The first spectral slice vector and the second spectral slice vector are weighted and summed according to the first weight and the second weight to obtain the spectral slice embedding vector.

6. The method according to any one of claims 1 to 5, characterized in that, The process of obtaining the spectrum of the piano audio to obtain the spectrum of the target piano includes: The initial piano spectrum is obtained by extracting the spectrum from the piano audio data using a preset Mel filter bank. The initial piano spectrum is convolved by a preset convolutional layer to obtain a first intermediate spectrum; The first intermediate spectrum is normalized by a preset normalization layer to obtain the second intermediate spectrum; The target piano spectrum is obtained by performing max pooling on the second intermediate spectrum through a preset pooling layer.

7. The method according to any one of claims 1 to 5, characterized in that, The attention model includes a gated position self-attention layer, a local feedforward layer, a multi-head self-attention layer, and a feedforward network layer. The step of inputting the spectral vector sequence into the preset attention model for sequence feature extraction to obtain a note context vector sequence includes: Attention calculations are performed on the spectral vector sequence through the gated position self-attention layer to obtain the first attention vector sequence; The second attention vector sequence is obtained by extracting features from the first attention vector sequence through the local feedforward layer. The third attention vector sequence is obtained by performing attention calculation on the second attention vector sequence through the multi-head self-attention layer; The third attention vector sequence is used to extract features from the feedforward network layer to obtain the note context vector sequence.

8. A piano audio data processing device, characterized in that, The device includes: The spectrum acquisition module is used to acquire the spectrum of the piano audio to obtain the spectrum of the target piano. The spectrum segmentation module is used to segment the spectrum of the target piano according to a preset frame length to obtain the original spectrum slice; The vector representation module is used to embed the original spectrum slice to obtain a spectrum slice embedding vector, and to perform position encoding on the original spectrum slice to obtain a spectrum slice position vector. The vector addition module is used to add the spectrum slice embedding vector and the spectrum slice position vector to obtain the spectrum slice target representation vector; The vector merging module is used to merge the target representation vectors of the spectrum slices to obtain a spectrum vector sequence; The sequence feature extraction module is used to input the spectrum vector sequence into a preset attention model to extract sequence features and obtain a note context vector sequence. The sequence feature prediction module is used to input the note context vector sequence into a preset note prediction model to perform sequence feature prediction and obtain piano note data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images