A humming recognition method, device and medium

Abstract

The application discloses a humming recognition method and device and a medium. The method comprises the following steps: extracting a first audio feature of a to-be-recognized humming audio by using a feature extraction model; the feature extraction model is a model obtained by training a humming audio sample and a label; the label is a pitch sequence corresponding to a MIDI sample file matched with the humming audio sample; the first audio feature is a feature output by a previous hidden layer of an output layer in the feature extraction model; matching the first audio feature with a second audio feature corresponding to a MIDI file in a music score library; the second audio feature is an audio feature obtained by inputting a pitch sequence corresponding to the MIDI file into a probability diffusion model; the second audio feature is a feature output by a previous hidden layer of an output layer in the probability diffusion model; and if the second audio feature is matched, determining a humming recognition result of the to-be-recognized humming audio based on the second audio feature. The accuracy of humming recognition can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of humming recognition, and particularly relates to a humming recognition method, device and medium. BACKGROUND

[0002] With the digital development of the music multimedia industry, the increasing online streaming music service resources make users spend a lot of time and effort to search for their favorite songs. Sometimes, users may forget the specific information of the songs and also have no archives of the music, but still want to search according to the fuzzy melody in memory. Humming recognition can be used to solve this problem. At present, in humming recognition, the pitch sequence is usually extracted from the humming audio of the user, the pitch sequence is matched with the existing score in the score library, and the score consistent with the pitch sequence mode is searched from the score library. However, this method only uses the pitch information of the humming audio for matching, and loses a lot of audio information, resulting in low humming recognition accuracy. In summary, in the process of implementing the present application, the inventors have found at least the problem of low humming recognition accuracy in the prior art. SUMMARY

[0003] Therefore, the present application aims to provide a humming recognition method, device and medium, which can improve the accuracy of humming recognition. The specific solutions are as follows:

[0004] In a first aspect, the present application discloses a humming recognition method, comprising:

[0005] extracting a first audio feature of a to-be-recognized humming audio by using a feature extraction model; wherein the feature extraction model is a model trained by using a humming audio sample and a label, the label is a pitch sequence corresponding to a MIDI sample file matched with the humming audio sample, and the first audio feature is a feature output by a previous hidden layer of an output layer in the feature extraction model;

[0006] matching the first audio feature with a second audio feature corresponding to a MIDI file in a score library; wherein the second audio feature is an audio feature obtained by inputting a pitch sequence corresponding to the MIDI file into a probability diffusion model, and the second audio feature is a feature output by a previous hidden layer of an output layer in the probability diffusion model;

[0007] if the second audio feature is matched, determining a humming recognition result of the to-be-recognized humming audio based on the second audio feature.

[0008] Optionally, the training process of the feature extraction model is as follows:

[0009] obtaining a first training sample set; wherein the first training sample set comprises humming audio samples and labels corresponding to the humming audio samples;

[0010] inputting the humming audio samples into a first initial model to obtain an output result of the first initial model;

[0011] determining a training loss based on the output result and the labels corresponding to the humming audio samples;

[0012] adjusting parameters of the first initial model based on the training loss;

[0013] when the training loss is less than a first preset loss threshold, determining the first initial model with the adjusted parameters as a feature extraction model.

[0014] Optionally, the obtaining of the first training sample set comprises:

[0015] obtaining a plurality of MIDI sample files and humming audio samples matched with each of the MIDI sample files;

[0016] determining a pitch sequence corresponding to each of the MIDI sample files;

[0017] determining the pitch sequence corresponding to each of the MIDI sample files as a label of the humming audio sample matched with the MIDI sample file;

[0018] determining a set composed of all the humming audio samples and the labels corresponding to the humming audio samples as the first training sample set.

[0019] Optionally, the determining of the pitch sequence corresponding to each of the MIDI sample files comprises:

[0020] generating a pitch sequence based on each MIDI sample file, directly determining the pitch sequence as the pitch sequence corresponding to the MIDI sample file, or adding noise to the pitch sequence to obtain the pitch sequence corresponding to the MIDI sample file.

[0021] Optionally, the generating of the pitch sequence based on each MIDI sample file comprises:

[0022] generating a pitch duration sequence based on each MIDI sample file; each tuple in the pitch duration sequence contains an element of a pitch and a duration of the pitch;

[0023] expanding the pitch duration sequence into a pitch sequence based on the duration, and / or adding a disturbance signal to the pitch duration sequence to obtain a disturbance-containing pitch duration sequence, and expanding the disturbance-containing pitch duration sequence into a pitch sequence.

[0024] Optionally, the training process of the probability diffusion model comprises:

[0025] obtaining a second training sample set; the second training sample set comprises pitch sequences corresponding to MIDI sample files;

[0026] training a second initial model by using the second training sample set to obtain a probability diffusion model; wherein, in the training process, the label information of the pitch sequence corresponding to the MIDI sample file is the pitch sequence itself.

[0027] Optionally, the matching of the first audio feature with the second audio feature corresponding to the MIDI file in the music score library comprises:

[0028] inputting the first audio feature and the second audio feature corresponding to the MIDI file in the music score library into a matching model to obtain a matching degree between the first audio feature and the second audio feature;

[0029] judging whether the first audio feature and the second audio feature match based on the matching degree.

[0030] Optionally, the training process of the matching model comprises:

[0031] obtaining a third training sample set; the third training sample set comprises a plurality of three-element feature groups, each three-element feature group comprising a third audio feature of a MIDI sample file, a fourth audio feature of a humming audio sample matched with the MIDI sample file, and a fifth audio feature of a humming audio sample not matched with the MIDI sample file; the third audio feature is a feature extracted by using the probability diffusion model, and the fourth audio feature and the fifth audio feature are features extracted by using the feature extraction model;

[0032] training a third initial model by using the third training sample set, and calculating a three-element group loss in the training process, and obtaining a matching model when the three-element group loss is less than a second preset loss threshold.

[0033] In a second aspect, the present application discloses an electronic device, comprising a memory and a processor, wherein:

[0034] the memory is used to save a computer program;

[0035] the processor is used to execute the computer program to realize the humming recognition method.

[0036] In a third aspect, the present application discloses a computer readable storage medium for saving a computer program, wherein the computer program is executed by a processor to realize the humming recognition method.

[0037] It can be seen that the first audio feature of the to-be-identified humming audio is extracted by using the feature extraction model; the feature extraction model is a model trained by using humming audio samples and labels, the label is a pitch sequence corresponding to a MIDI sample file matched with the humming audio sample, and the first audio feature is a feature output by a previous hidden layer of an output layer in the feature extraction model; the first audio feature is matched with a second audio feature corresponding to a MIDI file in a music score library; the second audio feature is an audio feature obtained by inputting a pitch sequence corresponding to the MIDI file into a probability diffusion model, and the second audio feature is a feature output by a previous hidden layer of an output layer in the probability diffusion model; if the second audio feature is matched, a humming recognition result of the to-be-identified humming audio is determined based on the second audio feature. That is, in the present application, the second audio feature used for matching the to-be-identified humming audio is a feature output by a previous hidden layer of an output layer in a probability diffusion model by inputting a pitch sequence corresponding to the MIDI file into the probability diffusion model, the feature is a higher-dimensional feature, for the to-be-identified humming audio, the feature is a feature output by a previous hidden layer of an output layer in the feature extraction model, and the label used in the training process of the feature extraction model is a pitch sequence corresponding to the MIDI file. In this way, the first audio feature of the to-be-identified humming audio and the second audio feature of the MIDI file are higher-dimensional features in the same domain, the information in the to-be-identified humming audio except the pitch is retained in the matching process, and the accuracy of humming recognition can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of the provided drawings.

[0039] Figure 1 A system framework schematic diagram applicable to the humming recognition scheme provided by the present application;

[0040] Figure 2 A humming recognition method flowchart disclosed by the present application;

[0041] Figure 3 A specific humming recognition method flowchart disclosed by the present application;

[0042] Figure 4 An electronic device structure diagram provided by the present application. DETAILED DESCRIPTION

[0043] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0044] Currently, in humming recognition, a pitch sequence is usually extracted from the humming audio of a user, the pitch sequence is matched with the existing music scores in a music score library, and the music score consistent with the pitch sequence mode is retrieved from the music score library. However, this way only utilizes the pitch information of the humming audio for matching, resulting in low humming recognition accuracy. In summary, in the process of implementing the present application, the inventors have at least found that the prior art has the problem of low humming recognition accuracy. Therefore, the present application provides a humming recognition scheme, which can improve the accuracy of humming recognition.

[0045] In the humming recognition scheme of the present application, the system framework used can be seen from Figure 1 as shown, which can specifically include a background server and a plurality of user terminals in communication connection with the background server. The user terminals include but are not limited to tablet computers, notebook computers, smart phones, personal computers (PC), which are not limited here.

[0046] In the present application, the user terminal is used to send the humming audio to be recognized to the background server, and the background server executes the steps of the humming recognition method, including extracting the first audio feature of the humming audio to be recognized by using a feature extraction model; wherein the feature extraction model is a model trained by using humming audio samples and labels, the label is a pitch sequence corresponding to a MIDI (Musical Instrument Digital Interface) sample file matched with the humming audio sample, and the first audio feature is a feature output by a previous hidden layer of an output layer in the feature extraction model; matching the first audio feature with a second audio feature corresponding to a MIDI file in a music score library; wherein the second audio feature is an audio feature obtained by inputting a pitch sequence corresponding to the MIDI file into a probability diffusion model, and the second audio feature is a feature output by a previous hidden layer of an output layer in the probability diffusion model; if the second audio feature is matched, determining the humming recognition result of the humming audio to be recognized based on the second audio feature. The background server pushes the humming recognition result to the user terminal for display, and the humming recognition result can include music name, performer or performer information.

[0047] Referring to Figure 2As shown, the embodiment of the present application discloses a humming recognition method, comprising:

[0048] Step S11: extracting first audio features of the humming audio to be recognized by using a feature extraction model; wherein the feature extraction model is a model trained by using humming audio samples and labels, the label is a pitch sequence corresponding to a MIDI sample file matched with the humming audio sample, and the first audio feature is a feature output by a previous hidden layer of an output layer in the feature extraction model.

[0049] In an implementation, the embodiment of the present application can first convert the humming audio to be recognized into a frequency spectrum by short-time Fourier transform, then denoises the frequency spectrum to obtain a denoised frequency spectrum, and then inputs the denoised frequency spectrum into the feature extraction model to obtain the feature output by the previous hidden layer of the output layer in the feature extraction model as the first audio feature.

[0050] The training process of the feature extraction model specifically comprises the following steps:

[0051] Step 00: obtaining a first training sample set; wherein the first training sample set comprises humming audio samples and labels corresponding to the humming audio samples.

[0052] In an implementation, the specific process of the above step 00 can comprise the following steps:

[0053] Step 001: obtaining a plurality of MIDI sample files and humming audio samples matched with each of the MIDI sample files.

[0054] In an implementation, a plurality of humming audio samples matched with each of the MIDI sample files can be obtained. Specifically, a plurality of MIDI sample files [m1, m2, …, m k ] of music are prepared, and for each MIDI sample file, a corresponding humming audio sample [q 11 , q 12 , …, q kn ] is prepared, each of which is a recorded audio. Wherein, m i represents the i-th MIDI sample file, and q ij represents the j-th humming audio sample corresponding to the MIDI sample file m i .

[0055] Step 002: determining a pitch sequence corresponding to each of the MIDI sample files.

[0056] In an implementation, a pitch sequence can be generated based on each MIDI sample file, and the pitch sequence is directly determined as the pitch sequence corresponding to the MIDI sample file.

[0057] Further, the specific process of generating the pitch sequence based on each MIDI sample file can be: generating a pitch duration sequence based on each MIDI sample file; each tuple in the pitch duration sequence contains a pitch and a duration of the pitch; expanding the pitch duration sequence into a pitch sequence based on the duration, and / or adding a perturbation signal to the pitch duration sequence to obtain a pitch duration sequence containing a perturbation, and expanding the pitch duration sequence containing the perturbation into a pitch sequence.

[0058] The unit of the duration can be a frame, and each tuple can be expanded frame by frame to obtain the pitch sequence. In addition, a tuple in the pitch duration sequence to which the perturbation signal is added can be determined to obtain a target tuple, and the perturbation signal is added to the elements in the target tuple. The target tuple can be one or more tuples in the pitch duration sequence, for example, the perturbation signal is added to each element in each tuple in the pitch duration sequence.

[0059] Further, the embodiment of the present application can use a low-pass filter to low-pass filter the pitch sequence to obtain a filtered pitch sequence. In this way, the problem of numerical jump between tuples can be solved, which is more consistent with the characteristics of human fundamental frequency variation, and through such processing, matching can be more easily completed.

[0060] For example, each MIDI sample file can be simplified as a pitch duration sequence m i = [(f1, d1), (f2, d2), …, (f t , d t )], where m i represents the i-th MIDI sample file, f t represents the t-th pitch, and d t represents the duration of the t-th pitch. In actual matching, since the humming of the user is generally not very accurate, the pitch and duration offset often occurs. The embodiment of the present application can augment the original MIDI file, which can effectively alleviate the problem. Specifically, for each tuple in the pitch duration sequence, a small perturbation e 11 , e 12 may be added to each element to form a new pitch duration sequence m ia = [(f1+e 11 , d1+e 12 ), (f2+e 11 , d2+e 12 ), …, (f t +e 11 , d t +e 12These newly generated pitch duration sequences are added to the training, which can improve the robustness of the model. Further, the pitch duration sequences are preprocessed to make them more similar to the humming audio samples. First, after obtaining the pitch duration sequence, it is first expanded into a pitch sequence according to the duration, for example, [(68, 2), (41, 1), …, (52, 3)] is expanded into [68, 68, 41, …, 52, 52, 52], 2 represents 2 frames, and then a low-pass filter is applied to the pitch sequence.

[0061] In another embodiment, a pitch sequence can be generated based on each MIDI sample file, and noise is added to the pitch sequence to obtain a pitch sequence corresponding to the MIDI sample file. It can be understood that adding the noise-added pitch sequence as a label can improve the robustness of the model.

[0062] Step 003: Determine the pitch sequence corresponding to each MIDI sample file as the label of the humming audio sample matched with the MIDI sample file.

[0063] That is, in one embodiment, the label is a noise-added pitch sequence, and in another embodiment, the label is a noise-unadded pitch sequence.

[0064] Step 004: Determine the set composed of all the humming audio samples and the labels corresponding to the humming audio samples as the first training sample set.

[0065] Step 01: Input the humming audio sample into the first initial model to obtain the output result of the first initial model.

[0066] That is, the humming audio sample is selected from the first training sample set and input into the first initial model.

[0067] In one embodiment, the humming audio sample can be converted into a spectrum by short-time Fourier transform, and then the spectrum is denoised to obtain a denoised spectrum, and then the denoised spectrum is input into the first initial model.

[0068] Step 02: Determine the training loss based on the output result and the label corresponding to the humming audio sample.

[0069] In one embodiment, the root mean square error between the output result and the label corresponding to the humming audio sample can be calculated to obtain the training loss.

[0070] Step 03: Parameter adjustment is performed on the first initial model based on the training loss.

[0071] Step 04: When the training loss is less than a first preset loss threshold, the first initial model adjusted in the parameter is determined as the feature extraction model.

[0072] That is, the embodiment of the present application trains the input model humming audio sample with the label as the target, constrains the minimum root mean square error of the output result and the label, and obtains the feature extraction model.

[0073] It should be pointed out that, in order to obtain higher dimensional features, the features extracted by the feature extraction model of the embodiment of the present application are the features output by the previous hidden layer of the output layer of the feature extraction model.

[0074] Step S12: Matching the first audio feature with a second audio feature corresponding to a MIDI file in a score library; wherein the second audio feature is an audio feature obtained by inputting a pitch sequence corresponding to the MIDI file into a probability diffusion model, and the second audio feature is a feature output by a previous hidden layer of an output layer of the probability diffusion model.

[0075] That is, the embodiment of the present application regards the MIDI file as a kind of random signal, restores it to a dimension matching the user's humming melody through the probability diffusion model, and matches the humming audio and the MIDI file from a higher dimension. In the matching process, part of the signal except the pitch sung by the user is retained, so that a higher matching accuracy is obtained.

[0076] In one embodiment, the training process of the probability diffusion model can specifically include:

[0077] Step 11: Obtain a second training sample set; the second training sample set includes a pitch sequence corresponding to a MIDI sample file.

[0078] Wherein, the process of obtaining the pitch sequence corresponding to the MIDI sample file can refer to the foregoing content, which will not be repeated here.

[0079] Step 12: Training a second initial model using the second training sample set to obtain a probability diffusion model; wherein, in the training process, the label information of the pitch sequence corresponding to the MIDI sample file is the pitch sequence itself.

[0080] It should be pointed out that the probability diffusion model is a process of restoring a piece of noise to a specified content by using a Gaussian noise process. In order to train the probability diffusion model, for any MIDI sample file m, it is changed into noise x by continuously adding Gaussian noise. The corresponding noise reduction process is introduced in the training to ensure that the obtained noise can be restored to the MIDI sample file m.

[0081] Furthermore, embodiments of this application can pre-extract features from MIDI files in the sheet music library using a probability diffusion model in order to improve matching speed.

[0082] Furthermore, in one embodiment, this application embodiment can input the first audio feature and the second audio feature corresponding to the MIDI file in the music score library into a matching model to obtain the matching degree between the first audio feature and the second audio feature; based on the matching degree, it is determined whether the first audio feature and the second audio feature match. Specifically, when the matching degree is greater than a threshold matching degree threshold, it is determined that the first audio feature and the second audio feature match. The training process of the matching model may specifically include the following steps:

[0083] Step 20: Obtain the third training sample set; the third training sample set includes multiple ternary feature groups, each of which includes a third audio feature of a MIDI sample file, a fourth audio feature of a humming audio sample that matches the MIDI sample file, and a fifth audio feature of a humming audio sample that does not match the MIDI sample file; the third audio feature is a feature extracted using the probability diffusion model, and the fourth and fifth audio features are features extracted using the feature extraction model.

[0084] In a specific implementation, multiple triples can be constructed. Any triple includes a MIDI sample file, a humming audio sample that matches the MIDI sample file, and a humming audio sample that does not match the MIDI sample file. For any triple, the probability diffusion model is used to extract the third audio feature of the MIDI sample file, and the feature extraction model is used to extract the fourth audio feature of the humming audio sample that matches the MIDI sample file, and the fifth audio feature of the humming audio sample that does not match the MIDI sample file, thus obtaining a triple feature set.

[0085] Step 21: Train the third initial model using the third training sample set, and calculate the triplet loss during the training process. When the triplet loss is less than the second preset loss threshold, the matching model is obtained.

[0086] In one implementation, a first cosine distance between the training results corresponding to the third and fourth audio features, and a second cosine distance between the training results corresponding to the third and fifth audio features can be calculated. The triplet loss is calculated using the first and second cosine distances. During training, the triplet loss decreases as the first cosine distance decreases and the second cosine distance increases.

[0087] It is understood that the matching degree output by the model in this embodiment can specifically be a matching score determined based on the cosine distance between the first audio feature and the second audio feature.

[0088] That is, during the training phase, a triplet (m, q1, q2) needs to be constructed, where m represents a MIDI sample file, q1 represents a humming audio sample that matches the MIDI sample file, and q2 represents a humming audio sample that does not match the MIDI sample file. Using a probability diffusion model and a feature extraction model, a triplet feature set (x, h1, h2) can be obtained, where x represents the third audio feature, h1 represents the fourth audio feature, and h2 represents the fifth audio feature. The third initial model can be a CRNN (Convolutional Recurrent Neural Network), which completes a non-linear function mapping. Assuming that the abstract function is represented by f, the goal of the training phase is to maximize cos(f(x), f(h1)) and minimize cos(f(x), f(h2)), which is achieved using Triplet Loss. During the inference phase, it can be determined whether the MIDI sample file and the humming audio to be identified belong to the same content by calculating cos(f(n),f(h)), where n represents the first audio feature and h represents the second audio feature.

[0089] Step S13: If a second audio feature is matched, the humming recognition result of the humming audio to be recognized is determined based on the second audio feature.

[0090] In a specific implementation, the music information corresponding to the humming audio to be identified can be determined based on the second audio feature, such as the music name, performer or singer.

[0091] For example, see Figure 3 As shown, Figure 3 This is a flowchart of a specific humming recognition method disclosed in an embodiment of this application. For MIDI files in a music library, they are first preprocessed into pitch sequences, and then input into a probability diffusion model to obtain second audio features. For the humming audio to be recognized in a user's humming recognition request, it is first preprocessed, i.e., converted by a short-time Fourier transform and denoised, and then input into a feature extraction neural network to obtain first audio features. The first and second audio features are input into a matching neural network to obtain a matching score that measures whether the two match. Based on the matching score, a match is determined. If a match is found, the recognition result is determined. In this way, the dimensionality of the MIDI file is increased through the probability diffusion model to obtain an intermediate representation. The feature extraction model is used to transform the original humming sound signal into an intermediate representation in the same domain as the converted MIDI. Finally, a recurrent neural network model is used to calculate the degree of matching between the two, thereby avoiding the loss of too much information by directly extracting pitch from the sound signal.

[0092] It can be seen that, in the embodiment of the application, the first audio feature of the to-be-recognized humming audio is extracted by using a feature extraction model; the feature extraction model is a model trained by using a humming audio sample and a label, the label is a pitch sequence corresponding to a MIDI sample file matched with the humming audio sample, and the first audio feature is a feature output by a previous hidden layer of an output layer in the feature extraction model; the first audio feature is matched with a second audio feature corresponding to a MIDI file in a music score library; the second audio feature is an audio feature obtained by inputting a pitch sequence corresponding to the MIDI file into a probability diffusion model, and the second audio feature is a feature output by a previous hidden layer of an output layer in the probability diffusion model; if the second audio feature is matched, a humming recognition result of the to-be-recognized humming audio is determined based on the second audio feature. That is, in the embodiment of the application, the second audio feature used for matching the to-be-recognized humming audio is a feature output by a previous hidden layer of an output layer in a probability diffusion model by inputting a pitch sequence corresponding to the MIDI file into the probability diffusion model, the feature is a higher-dimensional feature, for the to-be-recognized humming audio, it is a feature output by a previous hidden layer of an output layer in a feature extraction model, and the label used by the feature extraction model in the training process is a pitch sequence corresponding to the MIDI file. In this way, the first audio feature of the to-be-recognized humming audio and the second audio feature of the MIDI file are higher-dimensional features in the same domain, the information in the to-be-recognized humming audio except the pitch is retained in the matching process, and the accuracy of humming recognition can be improved.

[0093] In addition, the existing humming recognition method is usually a multi-step algorithm, that is, the pitch is first extracted, then filtered, and then matched by using a DTW (Dynamic Time Warping), which has high complexity. The embodiment of the application is an end-to-end method, which reduces the complexity and has more stable performance.

[0094] Next, the technical solution of the application will be described by taking a certain music APP as an example.

[0095] The background server of the music APP obtains MIDI files of a plurality of musics as MIDI sample files based on a music score library of the music APP, and obtains a humming audio sample matched with each MIDI sample file. For each MIDI sample file, a pitch duration sequence is first generated, and a disturbance signal is added to the pitch duration sequence. The pitch duration sequence with the added disturbance and the pitch duration sequence without the added disturbance are both expanded into a pitch sequence, and a low-pass filter is used for processing to obtain a filtered pitch sequence. The probability diffusion model is trained using the filtered pitch sequences. The humming audio sample is converted into a frequency spectrum through short-time Fourier transform, and then denoised to obtain a denoised frequency spectrum. The filtered pitch sequence is added with noise, and the frequency spectrum is used to train the feature extraction model with the loaded pitch sequence corresponding to the matched MIDI sample file as a label. Then, a triple group including the MIDI sample file, the humming audio sample matched with the MIDI sample file, and the humming audio sample not matched with the MIDI sample file is constructed, and the triple feature group is extracted using the trained probability diffusion model and feature extraction model, and the matching model is trained using the triple feature groups. Moreover, the probability diffusion model is used to extract the second audio feature of the MIDI file in the music score library.

[0096] The user terminal enters the humming recognition interface of the music APP based on a user operation, and records the humming of the user to obtain a to-be-recognized humming audio. The user terminal sends a humming recognition request to the background server, carrying the to-be-recognized humming audio. The background server extracts the first audio feature of the to-be-recognized humming audio using the feature extraction model, and the first audio feature is the feature output by the previous hidden layer of the output layer of the feature extraction model. The first audio feature is matched with the second audio feature corresponding to the MIDI file in the music score library, and the second audio feature is the feature output by the previous hidden layer of the output layer of the probability diffusion model. If the second audio feature is matched, the music information corresponding to the to-be-recognized humming audio is determined based on the second audio feature, such as the music name, the performer or the singer, and the music information is pushed to the user terminal for display. If the second audio feature is not matched, the user terminal is notified that the result is not matched.

[0097] Further, the embodiment of the present application also provides an electronic device. Figure 4 The electronic device 20 shown in the figure is a structural diagram according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0098] Figure 4A structural schematic diagram of an electronic device 20 is provided in the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the humming recognition method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the embodiments of the present application can be specifically a server.

[0099] In the embodiments of the present application, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0100] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, and the resources stored thereon can include an operating system 221, a computer program 222 and data 223, etc., and the storage mode can be temporary storage or permanent storage.

[0101] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the processor 21 on the data 223 in the memory 22, and the operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the humming recognition method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0102] Further, the embodiments of the present application further disclose a storage medium, and the storage medium stores a computer program. When the computer program is loaded and executed by a processor, the steps of the humming recognition method disclosed in any of the foregoing embodiments are implemented.

[0103] In the present specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other.

[0104] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be intended to distinguish one entity or operation from another entity or operation without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by an "including a" statement does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the one so defined.

[0105] The above provides a detailed introduction to the humming recognition method, device and medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this document. The above example description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation mode and application range; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A humming recognition method, characterized in that, include: The first audio feature of the humming audio to be identified is extracted using a feature extraction model; wherein, the feature extraction model is a model trained using humming audio samples and labels, the labels are pitch sequences corresponding to MIDI sample files that match the humming audio samples, during the training process, the training loss is determined based on the labels corresponding to the humming audio samples and the output results of the model, and the first audio feature is the feature output by the previous hidden layer of the output layer in the feature extraction model. The first audio feature is matched with the second audio feature corresponding to the MIDI file in the music score library; wherein, the second audio feature is the audio feature obtained by inputting the pitch sequence corresponding to the MIDI file into the probability diffusion model, and the second audio feature is the feature output by the previous hidden layer of the output layer in the probability diffusion model; If a second audio feature is matched, the humming recognition result of the humming audio to be recognized is determined based on the second audio feature.

2. The humming recognition method according to claim 1, characterized in that, The training process of the feature extraction model is as follows: Obtain a first training sample set; wherein, the first training sample set includes humming audio samples and the labels corresponding to the humming audio samples; Input the humming audio sample into the first initial model to obtain the output of the first initial model; The training loss is determined based on the output results and the labels corresponding to the humming audio samples. The parameters of the first initial model are adjusted based on the training loss; If the training loss is less than the first preset loss threshold, then the first initial model after parameter adjustment is determined as the feature extraction model.

3. The humming recognition method according to claim 2, characterized in that, The process of obtaining the first training sample set includes: Acquire multiple MIDI sample files and humming audio samples that match each of the MIDI sample files; Determine the pitch sequence corresponding to each of the MIDI sample files; The pitch sequence corresponding to each MIDI sample file is determined as the tag of the humming audio sample that matches the MIDI sample file; The set consisting of all the humming audio samples and the corresponding labels of the humming audio samples is determined as the first training sample set.

4. The humming recognition method according to claim 3, characterized in that, Determining the pitch sequence corresponding to each MIDI sample file includes: A pitch sequence is generated based on each MIDI sample file. The pitch sequence is directly determined as the pitch sequence corresponding to the MIDI sample file, or noise is added to the pitch sequence to obtain the pitch sequence corresponding to the MIDI sample file.

5. The humming recognition method according to claim 4, characterized in that, The generation of pitch sequences based on each MIDI sample file includes: A pitch duration sequence is generated based on each MIDI sample file; each tuple in the pitch duration sequence contains the pitch and the duration of that pitch. Based on the duration, the pitch duration sequence is expanded into a pitch sequence, and / or, a perturbation signal is added to the pitch duration sequence to obtain a perturbed pitch duration sequence, and the perturbed pitch duration sequence is expanded into a pitch sequence.

6. The humming recognition method according to claim 1, characterized in that, The training process of the probability diffusion model includes: Obtain a second training sample set; the second training sample set includes pitch sequences corresponding to MIDI sample files; The second initial model is trained using the second training sample set to obtain a probability diffusion model; wherein, during the training process, the label information of the pitch sequence corresponding to the MIDI sample file is the pitch sequence itself.

7. The humming recognition method according to any one of claims 1 to 6, characterized in that, The step of matching the first audio feature with the second audio feature corresponding to the MIDI file in the music score library includes: The first audio feature and the second audio feature corresponding to the MIDI file in the music library are input into the matching model to obtain the matching degree between the first audio feature and the second audio feature. Based on the matching degree, it is determined whether the first audio feature and the second audio feature match.

8. The humming recognition method according to claim 7, characterized in that, The training process of the matching model includes: Obtain a third training sample set; the third training sample set includes multiple ternary feature groups, each of which includes a third audio feature of a MIDI sample file, a fourth audio feature of a humming audio sample that matches the MIDI sample file, and a fifth audio feature of a humming audio sample that does not match the MIDI sample file; the third audio feature is a feature extracted using the probability diffusion model, and the fourth and fifth audio features are features extracted using the feature extraction model. The third initial model is trained using the third training sample set, and the triplet loss is calculated during the training process. When the triplet loss is less than the second preset loss threshold, a matching model is obtained.

9. An electronic device, characterized in that, Includes memory and processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the humming recognition method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the humming recognition method as described in any one of claims 1 to 8.

Images