Training method, device, medium and equipment for pitch recognition model of polyphonic audio
By improving the pitch recognition model, the pitch of human voices can be directly identified from polyphonic audio, solving the problem that the quality of human voices affects the accuracy of recognition and achieving more efficient and accurate pitch recognition.
Patent Information
- Application Number
- CN202410530115.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-04-29
AI Technical Summary
In existing technologies, when performing pitch recognition after separating human voices from polyphonic audio, the quality of the human voice affects the accuracy of the recognition.
The UNet and BiGRU modules in the RMVPE model are replaced with the Mamba-UNet module and the Transformer-encoder module. The Mamba-UNet module is used for feature extraction, the Transformer-encoder module is used for correlation analysis, and the fully connected layer is used for pitch recognition.
It improves the accuracy of pitch recognition, reduces computational costs and training time, avoids gradient problems, and improves the efficiency of processing large-scale data.
Smart Images

Figure CN118298845B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a training method, device, medium and equipment for a pitch recognition model of polyphonic audio. Background Art
[0002] Sound is composed of a series of vibrations with different frequencies and amplitudes emitted by the sound-producing body. Among these vibrations, there is a vibration with the lowest frequency. The sound emitted by it is the fundamental tone. The frequency of the fundamental tone is called the fundamental frequency, which is usually represented by f0. Pitch is the subjective perception of the sound frequency by the human auditory system, which determines the height of the sound we hear. In music and speech, pitch is usually determined by the fundamental frequency (f0) of the sound wave. In audio signal processing, fundamental frequency and pitch are often regarded as the same concept. Many audio-based applications require pitch recognition (also called fundamental frequency estimation), such as speech analysis and music information retrieval.
[0003] According to the complexity of the sound source, audio pitch recognition algorithms can be divided into pitch recognition algorithms for human voices with single-source sounds (also called monotonous audio) (for example, a speech without background sound) and pitch recognition algorithms for human voices with multi-source sounds (also called polyphonic audio) (for example, the voice of a singer with background sounds such as accompaniment).
[0004] When performing pitch recognition on polyphonic audio, the sound source separation model can be used to separate the human voice from background sounds such as accompaniment, and then the pitch recognition method of monotone audio can be used to perform pitch recognition on the separated human voice. In other words, the pitch recognition of polyphonic audio is divided into two steps: separating the human voice and recognizing the pitch of the human voice.
[0005] The effect of the above pitch recognition is not only closely related to the pitch recognition algorithm itself, but also depends on the performance of the sound source separation model. If the quality of the human voice separated by the sound source separation model is not high, it will affect the accuracy of pitch recognition. Summary of the invention
[0006] The present application provides a training method, device, medium and equipment for a pitch recognition model of polyphonic audio, which is used to solve the problem that when human voice is separated from polyphonic audio and then pitch recognition is performed on the human voice, the quality of the separated human voice will affect the accuracy of pitch recognition. The technical solution is as follows:
[0007] According to a first aspect of the present application, a method for training a pitch recognition model of polyphonic audio is provided, the method comprising:
[0008] Acquire a training set, wherein the training samples in the training set include polyphonic audio and real pitch information;
[0009] Convert each frame of audio data in the polyphonic audio into a Mel-spectrogram;
[0010] Building a pitch recognition model, the pitch recognition model at least comprising a Mamba-UNet module, a Transformer-encoder module and a fully connected layer;
[0011] Using the Mamba-UNet module to extract features from each frame of the Mel-spectrogram to obtain a first intermediate feature;
[0012] Using the Transformer-encoder module, performing correlation analysis on the first intermediate features corresponding to the consecutive frames to obtain the second intermediate features;
[0013] Using the fully connected layer to perform fully connected processing on the second intermediate feature, so as to obtain a pitch recognition result of a human voice in the polyphonic audio;
[0014] The pitch recognition model is trained according to the real pitch information and the pitch recognition result.
[0015] In a possible implementation, the Mamba-UNet module includes a first encoder and a first decoder, and the step of extracting features from each frame of the Mel-spectrogram using the Mamba-UNet module to obtain a first intermediate feature includes:
[0016] Each frame of the Mel spectrum is divided into blocks and converted into a one-dimensional sequence;
[0017] Using the first encoder to perform visual state space module processing and fusion layer processing on the one-dimensional sequence and create hierarchical features, outputting the obtained first encoding result to the first decoder, the first decoder is composed of visual state space modules and expansion layers matching the dimension of the first encoder; and using skip connections to output feature data extracted by each visual state space module in the first encoder to the visual state space module of the corresponding layer in the first decoder, so as to enhance the spatial details lost in the downsampling process;
[0018] The first decoder is used to expand and process the first encoding result and the feature data using a visual state space module to obtain a first intermediate feature.
[0019] In a possible implementation, the Transformer-encoder module includes a second encoder, and the using the Transformer-encoder module to perform association analysis on first intermediate features corresponding to consecutive frames to obtain second intermediate features includes:
[0020] The second encoder is used to perform multi-head attention processing and feedforward network processing on the first intermediate features corresponding to consecutive frames to obtain second intermediate features.
[0021] In a possible implementation, converting each frame of audio data in the polyphonic audio into a Mel-spectrogram includes:
[0022] Pre-emphasize, frame and window the polyphonic audio to obtain a plurality of short-time analysis windows;
[0023] Perform Fourier transform on each short-time analysis window to obtain the corresponding spectrum;
[0024] Filtering the spectrum using a Mel filter to obtain a filtering result;
[0025] A logarithmic operation is performed on the filtering result to obtain a Mel-spectrogram.
[0026] In a possible implementation, the method further includes:
[0027] Acquire polyphonic audio to be recognized, wherein the polyphonic audio includes human voice and background sound;
[0028] Convert each frame of audio data in the polyphonic audio into a Mel-spectrogram;
[0029] Obtaining the pre-trained pitch recognition model;
[0030] Using the Mamba-UNet module to extract features from each frame of the Mel-spectrogram to obtain a third intermediate feature;
[0031] Using the Transformer-encoder module, performing correlation analysis on the third intermediate features corresponding to the consecutive frames to obtain a fourth intermediate feature;
[0032] The fourth intermediate feature is fully connected by using the fully connected layer to obtain a pitch recognition result of the human voice in the polyphonic audio.
[0033] According to a second aspect of the present application, a training device for a pitch recognition model of polyphonic audio is provided, the device comprising:
[0034] A training set acquisition module, used to acquire a training set, wherein the training samples in the training set include polyphonic audio and real pitch information;
[0035] A spectrogram conversion module, used for converting each frame of audio data in the polyphonic audio into a Mel-spectrogram;
[0036] A model building module, used to build a pitch recognition model, wherein the pitch recognition model includes at least a Mamba-UNet module, a Transformer-encoder module and a fully connected layer;
[0037] A feature extraction module, used to extract features from each frame of the Mel-spectrogram using the Mamba-UNet module to obtain a first intermediate feature;
[0038] A feature analysis module, used to perform correlation analysis on the first intermediate features corresponding to the continuous frames using the Transformer-encoder module to obtain a second intermediate feature;
[0039] A pitch recognition module, configured to perform full-connection processing on the second intermediate feature using the fully-connected layer to obtain a pitch recognition result of a human voice in the polyphonic audio;
[0040] A model training module is used to train the pitch recognition model according to the real pitch information and the pitch recognition result.
[0041] In a possible implementation, the Mamba-UNet module includes a first encoder and a decoder, and the feature extraction module is further used to:
[0042] Each frame of the Mel spectrum is divided into blocks and converted into a one-dimensional sequence;
[0043] Using the first encoder to perform visual state space module processing and fusion layer processing on the one-dimensional sequence and create hierarchical features, outputting the obtained first encoding result to the first decoder, the first decoder is composed of visual state space modules and expansion layers matching the dimension of the first encoder; and using skip connections to output feature data extracted by each visual state space module in the first encoder to the visual state space module of the corresponding layer in the first decoder, so as to enhance the spatial details lost in the downsampling process;
[0044] The first decoder is used to expand and process the first encoding result and the feature data using a visual state space module to obtain a first intermediate feature.
[0045] In a possible implementation, the Transformer-encoder module includes a second encoder, and the feature analysis module is further configured to:
[0046] The second encoder is used to perform multi-head attention processing and feedforward network processing on the first intermediate features corresponding to consecutive frames to obtain second intermediate features.
[0047] According to a third aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the training method of the pitch recognition model of polyphonic audio as described above.
[0048] According to a fourth aspect of the present application, a computer device is provided, comprising a training device for the pitch recognition model of the polyphonic audio.
[0049] The beneficial effects of the technical solution provided by this application include at least:
[0050] The Mamba-UNet module can achieve linear complexity calculation without sacrificing the global receptive field, greatly reducing the computational cost of the model.
[0051] The Transformer-encoder module can capture long-distance dependencies in the sequence through the self-attention mechanism, effectively avoiding the gradient problem. At the same time, due to its parallel computing characteristics, it shows higher efficiency and better performance when processing large-scale data.
[0052] The pitch recognition model can directly identify the pitch of human voice in polyphonic audio without first separating the human voice from the polyphonic audio and then performing pitch recognition on the human voice. This ensures that the recognition result is not affected by the quality of the separated human voice, thereby improving the accuracy of pitch recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0054] Figure 1 It is a structural diagram of a Mamba-UNet module provided by an embodiment of the present application;
[0055] Figure 2 It is a structural diagram of a Transformer-encoder module provided by an embodiment of the present application;
[0056] Figure 3 is a training flow chart of a pitch recognition model provided by an embodiment of the present application;
[0057] Figure 4 is a flow chart of a method for pitch recognition of polyphonic audio provided by an embodiment of the present application;
[0058] Figure 5It is a structural block diagram of a training device for a pitch recognition model of polyphonic audio provided by an embodiment of the present application. DETAILED DESCRIPTION
[0059] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0060] In order to combine the two steps of separating the human voice and recognizing the pitch of the human voice into one, researchers in this field have tried to develop a model that can directly recognize the pitch of the human voice from polyphonic audio. According to our research, there are currently six models, CRN-Raw, DEEPF0, JDC, GIO, TAPE, and RMVPE, that can directly recognize the pitch of the human voice from polyphonic audio. Among them, the three models of CRN-Raw, DEEPF0, and JDC are obviously insufficient for the pitch recognition of polyphonic audio with complex musical accompaniment. The GIO model improves its performance in the pitch recognition task by introducing additional timbre features and using a dual-branch architecture to simultaneously process the Mel spectrogram and timbre features, but the additional introduction of timbre features makes the training and calculation of the GIO model more difficult. The TAPE model only focuses on the timbre features and does not effectively utilize the Mel spectrogram, so that the TAPE model only performs well in pitch recognition in the polyphonic audio of the violin-piano duet, and the model generalization ability is insufficient.
[0061] The RMVPE model has the best overall performance in the pitch recognition task. First, the polyphonic audio is converted into a mel-spectrogram as the input of the RMVPE model; in the feature extraction part, the RMVPE model uses a deep UNet module to extract implicit features of each frame in the polyphonic audio; in order to associate the extracted features, the RMVPE model uses a Bi-GRU module to further process the extracted features; in the loss function setting part, the RMVPE model refers to the loss function setting scheme of CREPE, and introduces a classification correction coefficient to slightly improve the loss of the CREPE model.
[0062] Since the Mel-spectrogram obtained by converting each frame of audio data can be regarded as image data, feature extraction can be performed according to the idea of image processing. The difference is that the processing of audio data requires correlation analysis of consecutive frames after the features extracted from each frame, so as to obtain the feature representation of the complete audio data.
[0063] The UNet module introduces the idea of convolution, so the decoder part needs to introduce deconvolution operation. This deconvolution operation will cause the first few layers of the decoder to have a checkerboard artifact effect and a large increase in the amount of calculation when processing image data, thereby affecting the accuracy of feature extraction of the subsequent network and increasing the difficulty of extraction. Therefore, we try to replace the UNet module with deconvolution operation. At the same time, considering the high computational complexity of the convolutional neural network (CNN) architecture of the UNet module, the visual state space (Visual State Space) module based on VMamba can realize linear complexity calculation without sacrificing the global receptive field, greatly reducing the computational cost of the model. Therefore, in this application, the Mamba-UNet module without deconvolution operation and the introduction of state space to reduce the complexity of the model is used to replace the UNet module in RMVPE.
[0064] As a variant of the recurrent neural network, the BiGRU module takes into account the contextual information on the time series when processing sequence data, but for long sequences (such as the audio in this application is long, the sampling rate is high, and the frame shift is small), there may be gradient vanishing or explosion problems, resulting in low training efficiency or difficulty in convergence. The Transformer-encoder module can capture long-distance dependencies in the sequence through the self-attention mechanism, effectively avoiding the gradient problem, and due to its parallel computing characteristics, it shows higher efficiency and better performance when processing large-scale data. Therefore, the Transformer-encoder module is used in this application to replace the BiGRU module in the RMVPE model.
[0065] The loss function refers to the design of the RMVPE model without any modification.
[0066] In summary, the present application improves the RMVPE model and designs a pitch recognition model, which at least includes a Mamba-UNet module, a Transformer-encoder module and a fully connected layer. The input of the pitch recognition model is the Mel-spectrogram obtained after the polyphonic audio conversion. The encoder-decoder structure of the Mamba-UNet module is used to extract features from the Mel-spectrogram, and then the Transformer-encoder module is used to perform correlation analysis on the extracted features. Finally, the fully connected layer is used to fully connect the analyzed features to obtain the pitch recognition result of the human voice. The structures of the Mamba-UNet module and the Transformer-encoder module are explained below.
[0067] (1) Mamba-UNet module
[0068] The Mamba-UNet module includes a first encoder and a decoder, Figure 1 The left half of represents the first encoder, Figure 1 The right half of represents the decoder. When the Mamba-UNet module is used to extract the features of the Mel-spectrogram, the Mel-spectrogram is first divided into blocks and converted into a one-dimensional sequence, and then the one-dimensional sequence is input into the first encoder, that is, the input of the first encoder is a one-dimensional sequence of the Mel-spectrogram (Image). Then, the first encoder is used to process the one-dimensional sequence with the visual state space module (VSS Block) and the fusion layer and create hierarchical features, and the obtained first encoding result is output to the decoder. In addition, each visual state space module in the first encoder is connected to the visual state space module of the corresponding layer in the decoder (the decoder is composed of a visual state space module and an expansion layer that matches the dimension of the first encoder) through a skip connection, so as to output the feature data extracted by each visual state space module in the first encoder to the visual state space module of the corresponding layer in the decoder through a skip connection, so as to enhance the spatial details lost in the downsampling process. The input of the decoder is the first encoding result and the feature data extracted by each visual state space module. After the first encoding result and the feature data are expanded and processed by the visual state space module, the first intermediate features (Level1Features) are output.
[0069] (2) Transformer-encoder module
[0070] The Transformer-encoder module includes a second encoder, Figure 2 Represents the second encoder. The second encoder consists of N identical layers. Each layer has two sublayers. The first sublayer includes multi-head self-attention (Multi-HeadAttention), and the input and output of the multi-head attention processing are added (Add) and then layer normalized (Norm) as the output of the first sublayer. The second sublayer includes a simple, fully connected feedforward network (FeedForward), and the input and output of the feedforward network are added (Add) and then layer normalized (Norm) as the output of the second sublayer. The input of the second encoder is the first intermediate feature. By performing multi-head attention processing and feedforward network processing on the first intermediate features corresponding to consecutive frames, the second intermediate features (Level2 Features) are output.
[0071] like Figure 3As shown, it shows a method flow chart of a method for training a pitch recognition model of polyphonic audio provided by an embodiment of the present application, and the training method of the pitch recognition model of polyphonic audio can be applied to a computer device. The training method of the pitch recognition model of polyphonic audio may include:
[0072] Step 301: Obtain a training set, where training samples in the training set include polyphonic audio and real pitch information.
[0073] In this embodiment, the polyphonic audio of the public data set MIR_ST500 and Cmedia can be selected to generate a training set. The real pitch information is the information obtained by marking the real pitch of the polyphonic audio.
[0074] Step 302: Convert each frame of audio data in the polyphonic audio into a Mel-spectrogram.
[0075] For each polyphonic audio, the libsora tool can be used to perform pre-emphasis, frame division and windowing on the polyphonic audio to obtain multiple short-time analysis windows; Fourier transform is performed on each short-time analysis window to obtain the corresponding spectrum; the spectrum is filtered using a Mel filter to obtain a filtering result; and a logarithmic operation is performed on the filtering result to obtain a Mel spectrum diagram.
[0076] For example, the polyphonic audio can be first framed at a sampling frequency of 16kHz, and then each frame can be converted into a mel spectrogram of logarithmic amplitude with 256 mel bins, where the hop length of the mel spectrogram is 320 (20ms), the Hann window size is 2048, and a mel filter is used to select data from 30Hz to 8000Hz to finally obtain a mel spectrogram.
[0077] Step 303, building a pitch recognition model.
[0078] Among them, the pitch recognition model is the model described above, which includes a Mamba-UNet module, a Transformer-encoder module and a fully connected layer.
[0079] In this embodiment, the Adam optimizer may also be used during training, and the initialization learning rate is set to 0.0005 and the batch size is set to 8.
[0080] Step 304: Use the Mamba-UNet module to extract features from each frame of the Mel-spectrogram to obtain a first intermediate feature.
[0081] Wherein, the Mamba-UNet module includes a first encoder and a decoder, and the feature extraction process may include:
[0082] (1) Each frame of the Mel-spectrogram is divided into blocks and converted into a one-dimensional sequence;
[0083] (2) using the first encoder to process the one-dimensional sequence with a visual state space module and a fusion layer and create hierarchical features, and outputting the obtained first encoding result to a decoder, wherein the decoder is composed of a visual state space module and an expansion layer that match the dimension of the first encoder; and using a skip connection to output feature data extracted by each visual state space module in the first encoder to the visual state space module of the corresponding layer in the decoder, so as to enhance the spatial details lost in the downsampling process;
[0084] (3) Using a decoder to expand the first encoding result and feature data and perform visual state space module processing to obtain a first intermediate feature.
[0085] Step 305: Use the Transformer-encoder module to perform correlation analysis on the first intermediate features corresponding to the consecutive frames to obtain the second intermediate features.
[0086] Among them, the Transformer-encoder module includes a second encoder, and the association analysis process may include: using the second encoder to perform multi-head attention processing and feedforward network processing on the first intermediate features corresponding to the continuous frames to obtain the second intermediate features.
[0087] Step 306: Use a fully connected layer to perform fully connected processing on the second intermediate feature to obtain a pitch recognition result of a human voice in the polyphonic audio.
[0088] Step 307: training the pitch recognition model according to the real pitch information and the pitch recognition result.
[0089] The computer device can use the loss function to calculate the loss of the pitch recognition result and the real pitch information, and update the parameters of the pitch recognition model according to the calculation results, and stop training until the recognition accuracy of the pitch recognition model meets the conditions.
[0090] The accuracy of the pitch recognition model includes pitch accuracy, timbre accuracy and overall accuracy:
[0091] (1) Raw Pitch Accuracy (RPA): The pitch of the vocal frame in the label sequence is compared with the corresponding frame in the extracted sequence frame by frame. If the pitch difference is less than 50 cents, it is correct.
[0092] (2) Raw Chroma Accuracy (RCA): The calculation method is roughly the same as RPA, and the octave error is ignored when comparing.
[0093] (3) Overall Accuracy (OA): The extracted sequence is compared with all frames of the labeled sequence frame by frame. If the pitch difference is less than 50 cents, it is correct.
[0094] In terms of model training time, the pitch recognition model and RMVPE model were trained from scratch using the public datasets MIR_ST500 and Cmedia. When the above results were achieved through training on the above two training sets, the training time of the pitch recognition model was reduced by 54% and 47% respectively compared with the RMVPE model.
[0095] In summary, the pitch recognition method for polyphonic audio provided by the embodiment of the present application, the Mamba-UNet module can achieve linear complexity calculation without sacrificing the global receptive field, greatly reducing the computational cost of the model. The Transformer-encoder module can capture long-distance dependencies in the sequence through the self-attention mechanism, effectively avoiding the gradient problem, and at the same time, due to its parallel computing characteristics, it shows higher efficiency and better performance when processing large-scale data.
[0096] After the pitch recognition model is trained, it can be used to perform pitch recognition on polyphonic audio. Figure 4 As shown, it shows a method flow chart of a polyphonic audio pitch recognition method provided by an embodiment of the present application. The polyphonic audio pitch recognition method may include:
[0097] Step 401: obtain polyphonic audio to be recognized, where the polyphonic audio includes human voice and background sound.
[0098] Step 402: Convert each frame of audio data in the polyphonic audio into a Mel-spectrogram.
[0099] For each polyphonic audio, the libsora tool can be used to perform pre-emphasis, frame division and windowing on the polyphonic audio to obtain multiple short-time analysis windows; Fourier transform is performed on each short-time analysis window to obtain the corresponding spectrum; the spectrum is filtered using a Mel filter to obtain a filtering result; and a logarithmic operation is performed on the filtering result to obtain a Mel spectrum diagram.
[0100] For example, the polyphonic audio can be first framed at a sampling frequency of 16kHz, and then each frame can be converted into a mel spectrogram of logarithmic amplitude with 256 mel bins, where the hop length of the mel spectrogram is 320 (20ms), the Hann window size is 2048, and a mel filter is used to select data from 30Hz to 8000Hz to finally obtain a mel spectrogram.
[0101] Step 403: Obtain a pre-trained pitch recognition model.
[0102] The pitch recognition model is the model described above, which includes a Mamba-UNet module, a Transformer-encoder module, and a fully connected layer.
[0103] Step 404: Use the Mamba-UNet module to perform feature extraction on the mel-spectrogram sequence to obtain a third intermediate feature.
[0104] Wherein, the Mamba-UNet module includes a first encoder and a decoder, and the feature extraction process may include:
[0105] (1) Each frame of the Mel-spectrogram is divided into blocks and converted into a one-dimensional sequence;
[0106] (2) using the first encoder to process the one-dimensional sequence with a visual state space module and a fusion layer and create hierarchical features, and outputting the obtained first encoding result to a decoder, wherein the decoder is composed of a visual state space module and an expansion layer that match the dimension of the first encoder; and using a skip connection to output feature data extracted by each visual state space module in the first encoder to the visual state space module of the corresponding layer in the decoder, so as to enhance the spatial details lost in the downsampling process;
[0107] (3) Using a decoder to expand the first encoding result and feature data and perform visual state space module processing to obtain a third intermediate feature.
[0108] Step 405: Use the Transformer-encoder module to perform correlation analysis on the third intermediate features corresponding to the consecutive frames to obtain the fourth intermediate features.
[0109] Among them, the Transformer-encoder module includes a second encoder, and the association analysis process may include: using the second encoder to perform multi-head attention processing and feedforward network processing on the third intermediate features corresponding to the continuous frames to obtain the fourth intermediate features.
[0110] Step 406: Use a fully connected layer to perform fully connected processing on the fourth intermediate feature to obtain a pitch recognition result of a human voice in the polyphonic audio.
[0111] In summary, the pitch recognition method for polyphonic audio provided in the embodiment of the present application can directly identify the pitch of the human voice in the polyphonic audio through the pitch recognition model, without the need to first separate the human voice from the polyphonic audio and then perform pitch recognition on the human voice. This ensures that the recognition result is not affected by the quality of the separated human voice, thereby improving the accuracy of pitch recognition.
[0112] The Mamba-UNet module can achieve linear complexity calculation without sacrificing the global receptive field, greatly reducing the computational cost of the model.
[0113] The Transformer-encoder module can capture long-distance dependencies in the sequence through the self-attention mechanism, effectively avoiding the gradient problem. At the same time, due to its parallel computing characteristics, it shows higher efficiency and better performance when processing large-scale data.
[0114] The song "First Time" (16kHz in MP3 format) was marked with pitch and timbre using praat software as a reference answer. The pitch recognition model in this application was used to perform an inference test on "First Time". By comparing the results of each frame of the entire song, the RPA was (89.41±5.53)%, the RCA was (89.63±5.35)%, and the OA was (82.41±5.24)%. The test results of using the RMVPC model for comparison were RPA (89.37±5.44)%, RCA (89.52±5.21)%, and OA (82.29±5.53)%. Therefore, the performance of the pitch recognition model is better than that of the RMVPC.
[0115] like Figure 5 As shown, it shows a structural block diagram of a training device for a pitch recognition model of polyphonic audio provided by an embodiment of the present application, and the training device for the pitch recognition model of polyphonic audio can be applied to a computer device. The training device for the pitch recognition model of polyphonic audio may include:
[0116] A training set acquisition module 510 is used to acquire a training set, wherein the training samples in the training set include polyphonic audio and real pitch information;
[0117] The spectrogram conversion module 520 is used to convert each frame of audio data in the polyphonic audio into a Mel-spectrogram;
[0118] A model building module 530 is used to build a pitch recognition model, and the pitch recognition model at least includes a Mamba-UNet module, a Transformer-encoder module and a fully connected layer;
[0119] A feature extraction module 540 is used to extract features from each frame of the Mel-spectrogram using a Mamba-UNet module to obtain a first intermediate feature;
[0120] A feature analysis module 550 is used to perform correlation analysis on the first intermediate features corresponding to the consecutive frames using a Transformer-encoder module to obtain a second intermediate feature;
[0121] A pitch recognition module 560 is used to perform full connection processing on the second intermediate feature using a fully connected layer to obtain a pitch recognition result of a human voice in the polyphonic audio;
[0122] The model training module 570 is used to train the pitch recognition model according to the real pitch information and the pitch recognition result.
[0123] In an optional embodiment, the Mamba-UNet module includes a first encoder and a decoder, a feature extraction module 540, and is further configured to:
[0124] Each frame of the Mel spectrum is divided into blocks and converted into a one-dimensional sequence;
[0125] The first encoder is used to perform visual state space module processing and fusion layer processing on the one-dimensional sequence and create hierarchical features, and the first encoding result is output to the first decoder, where the first decoder is composed of visual state space modules and expansion layers that match the dimension of the first encoder; and the feature data extracted by each visual state space module in the first encoder is output to the visual state space module of the corresponding layer in the first decoder by using a skip connection, so as to enhance the spatial details lost in the downsampling process;
[0126] The first encoding result and feature data are expanded and processed by a visual state space module using a first decoder to obtain a first intermediate feature.
[0127] In an optional embodiment, the Transformer-encoder module includes a second encoder, a feature analysis module 550, and is further configured to:
[0128] The second encoder is used to perform multi-head attention processing and feedforward network processing on the first intermediate features corresponding to consecutive frames to obtain the second intermediate features.
[0129] In an optional embodiment, the training set acquisition module 510 is further used to:
[0130] Pre-emphasize, frame and window the polyphonic audio to obtain multiple short-time analysis windows;
[0131] Perform Fourier transform on each short-time analysis window to obtain the corresponding spectrum;
[0132] Filter the spectrum using a Mel filter to obtain a filtering result;
[0133] Perform logarithmic operation on the filtering result to obtain the Mel spectrum.
[0134] In an optional embodiment, the device further includes an audio acquisition module and a model acquisition module;
[0135] An audio acquisition module is used to acquire polyphonic audio to be recognized, where the polyphonic audio includes human voice and background sound;
[0136] The spectrogram conversion module 520 is further used to convert each frame of audio data in the polyphonic audio into a Mel-spectrogram;
[0137] A model acquisition module is used to obtain a pre-trained pitch recognition model;
[0138] The feature extraction module 540 is further used to extract features from each frame of the Mel-spectrogram using the Mamba-UNet module to obtain a third intermediate feature;
[0139] The feature analysis module 550 is further used to perform correlation analysis on the third intermediate features corresponding to the continuous frames using the Transformer-encoder module to obtain a fourth intermediate feature;
[0140] The pitch recognition module 560 is further used to perform full-connection processing on the fourth intermediate feature using the fully-connected layer to obtain a pitch recognition result of the human voice in the polyphonic audio.
[0141] In summary, the training device for the pitch recognition model of polyphonic audio provided in the embodiment of the present application can directly identify the pitch of the human voice in the polyphonic audio through the pitch recognition model, without the need to first separate the human voice from the polyphonic audio and then perform pitch recognition on the human voice. This ensures that the recognition result is not affected by the quality of the separated human voice, thereby improving the accuracy of pitch recognition.
[0142] The Mamba-UNet module can achieve linear complexity calculation without sacrificing the global receptive field, greatly reducing the computational cost of the model.
[0143] The Transformer-encoder module can capture long-distance dependencies in the sequence through the self-attention mechanism, effectively avoiding the gradient problem. At the same time, due to its parallel computing characteristics, it shows higher efficiency and better performance when processing large-scale data.
[0144] An embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the training method of the pitch recognition model of polyphonic audio as described above.
[0145] An embodiment of the present application provides a computer device, which includes a training device for a pitch recognition model of any polyphonic audio.
[0146] It should be noted that: when the training device for the pitch recognition model of polyphonic audio provided in the above embodiment is training the pitch recognition model of polyphonic audio, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the training device for the pitch recognition model of polyphonic audio is divided into different functional modules to complete all or part of the functions described above. In addition, the training device for the pitch recognition model of polyphonic audio provided in the above embodiment and the training method embodiment of the pitch recognition model of polyphonic audio belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0147] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0148] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.
Claims
1. A method for training a pitch recognition model for polyphonic audio, characterized in that: The method comprises: Acquire a training set, wherein the training samples in the training set include polyphonic audio and real pitch information; Convert each frame of audio data in the polyphonic audio into a Mel-spectrogram; Building a pitch recognition model, the pitch recognition model at least comprising a Mamba-UNet module, a Transformer-encoder module and a fully connected layer; Using the Mamba-UNet module to extract features from each frame of the Mel-spectrogram to obtain a first intermediate feature; Using the Transformer-encoder module, performing correlation analysis on the first intermediate features corresponding to the consecutive frames to obtain the second intermediate features; Using the fully connected layer to perform fully connected processing on the second intermediate feature, so as to obtain a pitch recognition result of a human voice in the polyphonic audio; Training the pitch recognition model according to the real pitch information and the pitch recognition result; The Mamba-UNet module includes a first encoder and a decoder. The Mamba-UNet module is used to extract features from each frame of the Mel spectrum map to obtain a first intermediate feature, including: dividing each frame of the Mel spectrum map into blocks and converting it into a one-dimensional sequence; using the first encoder to perform visual state space module processing and fusion layer processing on the one-dimensional sequence and create hierarchical features, and outputting the obtained first encoding result to the decoder, wherein the decoder is composed of a visual state space module and an expansion layer that match the dimension of the first encoder; and using a jump connection to output the feature data extracted by each visual state space module in the first encoder to the visual state space module of the corresponding layer in the decoder to enhance the spatial details lost during the downsampling process; using the decoder to expand the first encoding result and the feature data and perform visual state space module processing to obtain the first intermediate feature.
2. The training method of the pitch recognition model of polyphonic audio according to claim 1, wherein The Transformer-encoder module includes a second encoder, and the Transformer-encoder module is used to perform association analysis on the first intermediate features corresponding to the continuous frames to obtain the second intermediate features, including: The second encoder is used to perform multi-head attention processing and feedforward network processing on the first intermediate features corresponding to consecutive frames to obtain second intermediate features.
3. The training method of the pitch recognition model of polyphonic audio according to claim 1, wherein The step of converting each frame of audio data in the polyphonic audio into a Mel-spectrogram comprises: Pre-emphasize, frame and window the polyphonic audio to obtain a plurality of short-time analysis windows; Perform Fourier transform on each short-time analysis window to obtain the corresponding spectrum; Filtering the spectrum using a Mel filter to obtain a filtering result; A logarithmic operation is performed on the filtering result to obtain a Mel-spectrogram.
4. The training method of the pitch recognition model of the arbitrary described polyphonic audio of claim 1 to 3 is characterized in that, The method further comprises: Acquire polyphonic audio to be recognized, wherein the polyphonic audio includes human voice and background sound; Convert each frame of audio data in the polyphonic audio into a Mel-spectrogram; Obtaining the pre-trained pitch recognition model; Using the Mamba-UNet module to extract features from each frame of the Mel-spectrogram to obtain a third intermediate feature; Using the Transformer-encoder module, performing correlation analysis on the third intermediate features corresponding to the consecutive frames to obtain a fourth intermediate feature; The fourth intermediate feature is fully connected by using the fully connected layer to obtain a pitch recognition result of the human voice in the polyphonic audio.
5. A training device for a pitch recognition model of polyphonic audio, characterized in that: The device comprises: A training set acquisition module, used to acquire a training set, wherein the training samples in the training set include polyphonic audio and real pitch information; A spectrogram conversion module, used for converting each frame of audio data in the polyphonic audio into a Mel-spectrogram; A model building module, used to build a pitch recognition model, wherein the pitch recognition model includes at least a Mamba-UNet module, a Transformer-encoder module and a fully connected layer; A feature extraction module, used to extract features from each frame of the Mel-spectrogram using the Mamba-UNet module to obtain a first intermediate feature; A feature analysis module, used to perform correlation analysis on the first intermediate features corresponding to the continuous frames using the Transformer-encoder module to obtain a second intermediate feature; A pitch recognition module, configured to perform full-connection processing on the second intermediate feature using the fully-connected layer to obtain a pitch recognition result of a human voice in the polyphonic audio; A model training module, used for training the pitch recognition model according to the real pitch information and the pitch recognition result; The Mamba-UNet module includes a first encoder and a decoder, and the feature extraction module is also used to: divide each frame of the Mel spectrum map into blocks and convert it into a one-dimensional sequence; use the first encoder to perform visual state space module processing and fusion layer processing on the one-dimensional sequence and create hierarchical features, and output the obtained first encoding result to the first decoder, the first decoder is composed of a visual state space module and an expansion layer that match the dimension of the first encoder; and use a jump connection to output the feature data extracted by each visual state space module in the first encoder to the visual state space module of the corresponding layer in the first decoder to enhance the spatial details lost during the downsampling process; use the first decoder to expand and perform visual state space module processing on the first encoding result and the feature data to obtain a first intermediate feature.
6. The training device of the pitch recognition model of polyphonic audio according to claim 5, characterized in that, The Transformer-encoder module includes a second encoder, and the feature analysis module is further used for: The second encoder is used to perform multi-head attention processing and feedforward network processing on the first intermediate features corresponding to consecutive frames to obtain second intermediate features.
7. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the training method of the pitch recognition model of polyphonic audio as described in any one of claims 1 to 4.
8. A computer device, characterized in that: The computer device comprises: a training device for the pitch recognition model of polyphonic audio according to claim 5 or 6.
