An audio recognition method, apparatus, device, and program product

Abstract

This application discloses an audio recognition method, apparatus, device, and program product. The method includes: obtaining a first audio to be recognized; obtaining real song information corresponding to the first audio to be recognized; the real song information represents the song to which the first audio to be recognized belongs; recognizing the first audio to be recognized according to a pre-trained first processing model to obtain first predicted song information of the first audio to be recognized; if it is determined that the first predicted song information is inconsistent with the real song information, updating the first processing model according to the first audio to be recognized and the real song information until a convergence condition is reached to obtain a trained first processing model; the trained first processing model is used to perform song recognition on the audio.

Description

Technical Field

[0001] This application relates to the field of audio recognition technology, and in particular to an audio recognition method, apparatus, device, and program product. Background Technology

[0002] Currently, some applications should have a song recognition function. Based on this function, the application can identify which song a recorded audio segment belongs to. For example, while a user is singing a song A, the application can record the audio of the user singing and identify that the audio segment belongs to song A.

[0003] Song recognition functionality can be implemented using various technologies. One such technology involves training a deep convolutional neural network model using a pre-collected music library. The trained model can then extract audio features from the audio to be recognized, and the song to be identified can be determined based on these extracted features.

[0004] This approach suffers from low accuracy in some scenarios, and may result in situations where, for example, a user sings a segment of an audio clip of song A, but the audio is identified as belonging to song B. Summary of the Invention

[0005] Therefore, this application discloses the following technical solution:

[0006] The first aspect of this application provides an audio recognition method, including:

[0007] Obtain a first audio to be identified, and acquire the real song information corresponding to the first audio to be identified; the real song information represents the song to which the first audio to be identified belongs.

[0008] The first audio to be identified is identified based on the pre-trained first processing model to obtain the first predicted song information of the first audio to be identified.

[0009] If it is determined that the first predicted song information is inconsistent with the real song information, the first processing model is updated according to the first audio to be identified and the real song information until the convergence condition is met, and a trained first processing model is obtained; the trained first processing model is used to perform song recognition on the audio.

[0010] Optionally, obtaining the real song information corresponding to the first audio to be identified includes:

[0011] The first audio to be identified is processed according to the second processing model to obtain the second predicted song information and confidence level corresponding to the first audio to be identified.

[0012] If the confidence level is greater than a preset confidence threshold, the second predicted song information is used as the real song information corresponding to the first audio to be identified.

[0013] Optionally, the step of identifying the first audio to be identified based on a pre-trained first processing model to obtain the first predicted song information of the first audio to be identified includes:

[0014] The first audio to be identified is processed according to the pre-trained first processing model to obtain the first audio feature;

[0015] The first audio feature is compared with multiple reference audio features in a pre-built audio feature library to determine a first reference audio feature that matches the first audio feature. The reference audio feature is obtained by the first processing model processing the reference audio segment.

[0016] The song information corresponding to the first reference audio feature is determined as the first predicted song information of the first audio to be identified, and the song information corresponding to the first reference audio feature represents the song to which the corresponding reference audio segment belongs.

[0017] Optionally, updating the first processing model based on the first audio to be identified and the real song information includes:

[0018] The second audio feature is obtained based on the real song information. The second audio feature is obtained by processing the second audio by the first processing model. The second audio is the audio contained in the song represented by the real song information.

[0019] The model loss is determined based on the first audio feature, the second audio feature, and the first reference audio feature;

[0020] The first processing model is updated based on the model loss.

[0021] Optionally, determining the model loss based on the first audio feature, the second audio feature, and the first reference audio feature includes:

[0022] The model loss is determined based on the differences between the first audio feature and the first reference audio feature, and the differences between the first audio feature and the second audio feature.

[0023] Optionally, determining the model loss based on the first audio feature, the second audio feature, and the first reference audio feature includes:

[0024] The confidence level of the real song information is obtained, and both the confidence level and the real song information are obtained by processing the first audio to be identified by a pre-trained second processing model;

[0025] The model loss is determined based on the difference between the first audio feature and the first reference audio feature, the difference between the first audio feature and the second audio feature, and the confidence level.

[0026] Optionally, after updating the first processing model based on the first audio to be identified and the real song information, the method further includes:

[0027] Each of the reference audio segments is processed according to the updated first processing model to update the reference audio features of the audio feature library.

[0028] Optionally, processing the first audio to be identified according to the pre-trained first processing model to obtain the first audio features includes:

[0029] The duration of the first audio to be identified is adjusted to the target duration to obtain the adjusted first audio to be identified;

[0030] The adjusted first audio to be identified is processed according to the pre-trained first processing model to obtain the first audio feature of the first audio to be identified.

[0031] A second aspect of this application provides an audio recognition device, comprising:

[0032] The obtaining unit is used to obtain a first audio to be identified and to acquire the real song information corresponding to the first audio to be identified; the real song information represents the song to which the first audio to be identified belongs.

[0033] The recognition unit is used to recognize the first audio to be recognized according to the pre-trained first processing model, and obtain the first predicted song information of the first audio to be recognized.

[0034] An update unit is used to update the first processing model based on the first audio to be identified and the real song information when it is determined that the first predicted song information is inconsistent with the real song information, until the convergence condition is met, and a trained first processing model is obtained; the trained first processing model is used to perform song recognition on the audio.

[0035] A third aspect of this application provides a computer device, including a memory and a processor;

[0036] The memory is used to store computer programs;

[0037] The processor is used to execute the computer program to implement the audio recognition method provided in any of the first aspects of this application.

[0038] A fourth aspect of this application provides a computer program product including executable computer instructions;

[0039] When the computer instructions are executed by the processor, they are used to implement the audio recognition method provided by any one of the first aspects of this application.

[0040] The beneficial effects of this plan are as follows:

[0041] By repeatedly updating the first processing model with the obtained audio to be identified and real song information, the first processing model can learn the features of the audio to be identified recorded in various recording scenarios (including but not limited to street, shopping mall and other scenarios), thereby improving the performance of the first processing model. This allows the first processing model to obtain accurate audio features that represent the audio to be identified when processing audio to be identified recorded in various recording scenarios, thereby improving the accuracy of song information obtained based on audio feature recognition. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a flowchart of an audio recognition method provided in an embodiment of this application;

[0044] Figure 2 This is a flowchart of another audio recognition method provided in the embodiments of this application;

[0045] Figure 3 This is a schematic diagram illustrating the principle of a model loss provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of an audio recognition device provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0049] This application provides an audio recognition method. Please refer to [link to relevant documentation]. Figure 1Here is a flowchart of the method, which may include the following steps.

[0050] S101, obtain the first audio to be identified, obtain the real song information corresponding to the first audio to be identified, and the real song information represents the song to which the first audio to be identified belongs.

[0051] The method provided in this embodiment can be executed by an electronic device used to run the first processing model, such as by a server where the first processing model resides.

[0052] The first audio to be identified can be any audio segment collected by a user using relevant devices. For example, when user A hears music on the street, user A can record it using their personal terminal device, and then operate the personal terminal device to upload the recorded audio segment to the server that provides the music recognition service. The uploaded audio segment can be regarded as the first audio to be identified obtained in step S101.

[0053] Understandably, the first audio to be identified may include a segment of music heard by the user, or various environmental noises from the user's environment, such as the noise of cars driving and honking on the street, or the voices of people around in the supermarket.

[0054] The actual song information can identify which song the first audio to be identified belongs to. For example, if a user records a segment of song A using a device and obtains the first audio to be identified, then the actual song information of the first audio to be identified can indicate that the first audio to be identified belongs to song A.

[0055] Authentic song information can be obtained in various ways. For example, a pre-trained second processing model can be used to process the first audio to be identified to obtain authentic song information, or the first audio to be identified can be manually identified and labeled to obtain authentic song information.

[0056] The identification and annotation can be performed manually by the user of the terminal device that uploaded the first audio to be identified, or manually by relevant personnel on the server side that received the first audio to be identified.

[0057] S102, the first audio to be identified is identified according to the pre-trained first processing model to obtain the first predicted song information of the first audio to be identified.

[0058] The first predicted song information represents the song to which the first audio to be identified belongs, as predicted by the first processing model. This first predicted song information may or may not be consistent with the actual song information. Referring to the previous example, the first predicted song information can represent that the first audio to be identified belongs to song A, or it can represent that the first audio to be identified belongs to song B.

[0059] The method for obtaining the first predicted song information based on the first processing model can be as follows: process the first audio to be identified using the first processing model to obtain a label representing a specific song output by the first processing model, and use the song information corresponding to the label as the first predicted song information; or it can be:

[0060] S1, process the first audio to be identified according to the pre-trained first processing model to obtain the first audio features;

[0061] S2, compare the first audio feature with multiple reference audio features in the pre-built audio feature library to determine the first reference audio feature that matches the first audio feature. The reference audio feature is obtained by the first processing model processing the reference audio segment.

[0062] S3, the song information corresponding to the first reference audio feature is determined as the first predicted song information of the first audio to be identified, and the song information corresponding to the first reference audio feature represents the song to which the corresponding reference audio segment belongs.

[0063] In step S1, the audio data corresponding to the first audio to be identified can be directly input into the first processing model to obtain the first audio feature output by the first processing model; or the Mel frequency cepstral coefficient (MFCC) feature of the first audio to be identified can be extracted, and the MFCC can be input into the first processing model to obtain the first audio feature output by the first processing model; or other spectral features of the first audio to be identified can be extracted, and the extracted spectral features can be input into the first processing model to obtain the first audio feature output by the first processing model.

[0064] The audio feature library can be constructed in the following ways:

[0065] Each song in the music library used for training is divided into several reference audio segments. For example, the original version of song A is divided into M 15-second reference audio segments, the cover version of song A sung by singer B is divided into M 15-second reference audio segments, the original version of song B is divided into N 15-second reference audio segments, and the cover version of song B sung by singer D is divided into N 15-second reference audio segments.

[0066] Each reference audio segment is processed using the first processing model to obtain the reference audio features corresponding to each reference audio segment. Each reference audio segment corresponds to a reference audio feature. For example, the original song A is divided into M reference audio segments using the first processing model to obtain M reference audio features.

[0067] The method for processing the obtained reference audio features is described in step S1.

[0068] Based on the audio feature library, the method for comparing the first audio feature and the reference audio feature in S2 can be as follows: calculate the similarity between the first audio feature and each reference audio feature, determine the reference audio feature with the highest similarity as the first reference audio feature that matches the first audio feature, and then determine the song information corresponding to the first reference audio feature as the first predicted song information of the first audio to be identified.

[0069] For example, assuming the audio feature library contains 100 reference audio features, the similarity between the first audio feature and the 100 reference audio features can be calculated in S2. The similarity of a certain reference audio feature is the maximum value among these 100 similarities. Therefore, in S3, this reference audio feature is determined to be the first reference audio feature that matches the first audio feature. This first reference audio feature is obtained by the first processing model processing the reference audio segment X. The reference audio segment X is segmented from the original song A. Therefore, the song information corresponding to the first reference audio feature can be song A. Thus, the first predicted song information of the first audio to be identified can be determined to be song A, that is, the first audio to be identified is determined to be a segment of song A.

[0070] The first processing model can be any pre-trained model with a neural network structure. In some embodiments, the first processing model can be a pre-trained cover song recognition model. When the first processing model is a cover song recognition model, the first predicted song information can be obtained based on the methods described in steps S1 to S3 above.

[0071] Please see Figure 2 Taking the first processing model as an example, which is a cover song recognition model, the first processing model can be trained in the following way.

[0072] First, a music library containing multiple song groups can be built, with each song group corresponding to a song, and each song group can contain audio data of different versions of its corresponding song.

[0073] For example, song A corresponds to song group 1, and song B corresponds to song group 2. Song A was originally sung by singer A. The audio data recorded when singer A sang song A is the original song data corresponding to song A. At the same time, after song A was released, it was covered by singers B and C. The audio data recorded when singer B sang song A is the cover song data corresponding to singer B's version of song A. The audio data recorded when singer C sang song A is the cover song data corresponding to singer C's version of song A. Song group 1 can include the original song data of song A, the cover song data of singer B's version of song A, and the cover song data of singer C's version of song A, that is, it includes the original song and the two versions of the cover song.

[0074] For any song group in the music library, you can select the original song and any version of the cover song contained in the song group, and at the same time select another song from another song group (which can be the original song or a cover song from another song group), and extract a segment of the original song, a segment of the cover song and a segment of the other song respectively. In this way, you can obtain a triplet sample consisting of three segments.

[0075] Optionally, segments can be extracted based on a specific target duration, meaning each extracted segment is a segment of the corresponding song with a target duration. The target duration can be set as needed; for example, the target duration could be 15 seconds.

[0076] For example, a triplet sample may include: a 15-second clip of the original version of song A, a 15-second clip of a cover version of song A by singer B, and a 15-second clip of a cover version of song B.

[0077] After obtaining several triplet samples in the above manner, these triplet samples can be used to train an initial neural network model with the goal of making the features of the original song fragment and the cover song fragment as similar as possible, and making the features of the original song fragment as dissimilar as possible to the features of other song fragments. When the model loss of the neural network model converges, the neural network model can be used as a trained cover song recognition model.

[0078] For training a cover song recognition model using triplet samples, please refer to relevant existing technologies, which will not be elaborated here.

[0079] S103, if it is determined that the first predicted song information is inconsistent with the real song information, the first processing model is updated according to the first audio to be identified and the real song information until the convergence condition is met, and the trained first processing model is obtained; the trained first processing model is used to identify songs from the audio.

[0080] In step S103, the first predicted song information obtained in S102 and the real song information obtained in S101 can be compared. If they are inconsistent, the first processing model can be updated based on the first audio to be identified and the real song information. Specifically, all or part of the model parameters of the first processing model can be updated.

[0081] After each update of the first processing model in step S103, step S102 can be executed again to process the first audio to be identified based on the updated first processing model to obtain new first predicted song information. If the newly obtained first predicted song information is inconsistent with the real song information, the process of updating the first processing model and obtaining new first predicted song information based on the updated model is repeated until the obtained first predicted song information meets the convergence condition.

[0082] Optionally, when updating the first processing model in step S103, the adjustment range of the model parameters can be limited to a small range, that is, each update only makes small adjustments to the model parameters of the first processing model. Here, "small" means that the adjustment range of the model parameters during the update is smaller than the adjustment range of the model parameters during training.

[0083] In this embodiment, the inconsistency between the first predicted song information and the actual song information indicates an error in the song recognition result obtained based on the first processing model. For example, the first audio to be identified is an audio recording of song A sung by any singer. In this case, the actual song information indicates that the first audio to be identified belongs to song A, while the first predicted song information obtained in S102 is song B. Therefore, the inconsistency between the first predicted song information and the actual song information indicates an error in the song recognition result obtained based on the first processing model.

[0084] Correspondingly, reaching the convergence condition indicates that the song recognition result obtained based on the first processing model is correct. In this embodiment, after each update of the first processing model, the first audio to be recognized can be identified based on the updated first processing model to obtain new first predicted song information. Then, the new first predicted song information is compared with the real song information. If the new first predicted song information and the real song information are consistent at this time, it can be determined that the first processing model obtained after this update has reached the convergence condition.

[0085] Based on the previous example, if the real song information indicates that the first audio to be identified belongs to song A, and the first predicted song information obtained based on the updated first processing model also indicates that the first audio to be identified belongs to song A, then it is determined that the updated first processing model has reached the convergence condition.

[0086] The reason why existing solutions are inaccurate in their recognition results is that there is a significant difference between the songs used to train the model in the music library and the audio that actually needs to be recognized. For example, the songs used to train the model are usually recorded in low-noise environments such as recording studios, while the audio that actually needs to be recognized is usually recorded in noisy environments such as streets and shopping malls. This difference causes the audio features extracted by the model to fail to accurately represent the audio that actually needs to be recognized, thus leading to inaccurate recognition results.

[0087] The beneficial effect of this embodiment is that, in order to address the above-mentioned problems, this embodiment uses the obtained audio to be identified to repeatedly update the first processing model, which enables the first processing model to learn the features of the audio to be identified recorded in various recording scenarios (including but not limited to street, shopping mall and other scenarios), thereby improving the performance of the first processing model and enabling the first processing model to obtain more accurate first predicted song information when processing audio to be identified recorded in various recording scenarios.

[0088] Optionally, to obtain more accurate first audio features, in S1, the duration of the first audio to be identified can be adjusted first, and then the corresponding first audio features can be obtained:

[0089] Adjust the duration of the first audio file to be identified to the target duration to obtain the adjusted first audio file to be identified;

[0090] The first audio to be identified is processed and adjusted according to the pre-trained first processing model to obtain the first audio features of the first audio to be identified.

[0091] In the process of adjusting the duration, the actual duration and target duration of the first audio to be identified can be compared first. If the actual duration and target duration are consistent, the first audio to be identified can be identified as the adjusted first audio to be identified.

[0092] If the actual duration is less than the target duration, the first audio to be identified can be padded with zeros at the beginning and end to obtain an adjusted first audio to be identified with the target duration, for example, an adjusted first audio to be identified with a duration of 15 seconds.

[0093] If the actual duration is longer than the target duration, the first audio to be identified can be trimmed to obtain an adjusted first audio to be identified with the target duration.

[0094] After adjusting in the above manner, it can be ensured that the duration of the audio to be identified processed by the first processing model and the duration of the segment used for training are both the target duration, thus avoiding inaccurate audio features output by the first processing model due to inconsistent durations.

[0095] In some optional embodiments, the method for obtaining the real song information corresponding to the first audio to be identified may be:

[0096] The first audio to be identified is processed according to the second processing model to obtain the second predicted song information and confidence level corresponding to the first audio to be identified.

[0097] If the confidence level is greater than the preset confidence level threshold, the second predicted song information is used as the real song information corresponding to the first audio to be identified.

[0098] The second processing model may have a different model structure and model parameters than the first processing model, and the second processing model may have higher accuracy than the first processing model, or the second processing model may have higher accuracy than the first processing model at least when recognizing a portion of the audio to be recognized in a specific scenario.

[0099] The second processing model can be any model that differs from the first processing model and is capable of determining song information for the audio to be identified. For an example, please refer to [link to example]. Figure 2 The second processing model can be an audio fingerprint algorithm model.

[0100] The training methods and working principles of the audio fingerprint algorithm model can be found in relevant technical documents, and will not be elaborated further here.

[0101] Generally, audio fingerprinting algorithm models have high accuracy when used to identify part of the audio, but lower accuracy when used to identify another part of the audio. To distinguish between these two cases, the processing results output by the audio fingerprinting algorithm model can include confidence level and song information corresponding to the audio to be identified. If the confidence level is high enough, it means that the identified song information has high accuracy; if the confidence level is low, it means that the identified song information has low accuracy.

[0102] like Figure 2 As shown, after processing the first audio to be identified using the audio fingerprint algorithm model and obtaining the song information and confidence level of the first audio to be identified, it can be determined whether the song information is a high-confidence result. If the confidence level is greater than the confidence threshold, the song information can be determined to be a high-confidence result, and the song information output by the audio fingerprint algorithm model is taken as the real song information of the first audio to be identified (i.e., Figure 2 As shown in the song information 2), if the confidence level is less than the confidence level threshold, it can be determined that the song information output by the audio fingerprint algorithm model is not the real song information of the first audio to be identified.

[0103] The confidence threshold can be set as needed without limitation. For example, if the confidence range is 0% to 100%, the confidence threshold can be set to 95%.

[0104] Optionally, if the confidence level output by the audio fingerprint algorithm model is less than or equal to the confidence level threshold, the real song information of the first predicted song information can be obtained by other methods (e.g., by obtaining it through manual annotation), or the method of this embodiment can be terminated directly.

[0105] like Figure 2 As shown, after obtaining the song information 2 (i.e., real song information) and song information 1 (i.e., first predicted song information) of the first audio to be identified, if the two are inconsistent, the first processing model (i.e. cover song recognition model) can be updated according to the first audio to be identified and the real song information.

[0106] In some optional embodiments, updating the first processing model based on the first audio to be identified and the real song information may include:

[0107] A1. Obtain the second audio feature based on the real song information. The second audio feature is obtained by processing the second audio by the first processing model. The second audio is the audio contained in the song represented by the real song information.

[0108] A2, determine the model loss based on the first audio feature, the second audio feature, and the first reference audio feature;

[0109] A3, update the first processing model based on the model loss.

[0110] The first reference audio feature is obtained by processing the first reference audio segmented from the music library using the first processing model.

[0111] In step A1, the song corresponding to the first reference audio feature that matches the first audio feature in the audio feature library can be determined. Then, another audio segment known to belong to the song indicated by the real song information can be determined as the second audio. The second audio is processed by the first processing model to obtain the second audio feature.

[0112] For example, suppose that the first reference audio feature that matches the first audio feature belongs to song A, causing the first audio to be identified to be misidentified as belonging to song A based on the first processing model. At the same time, the real song information of the first audio to be identified indicates that the first audio to be identified belongs to song B. Then an audio segment that also belongs to song B can be determined as the second audio. The second audio is processed by the first processing model to obtain the second audio feature.

[0113] The second audio clip can be a pre-collected audio clip stored in an audio library, or it can be an audio clip uploaded by a user.

[0114] The second audio and the first audio to be identified can be two different audio segments belonging to the song represented by the real song information. For example, the first audio to be identified can be a segment of version A of song B, and the second audio can be a segment of version B of song B, or the first audio to be identified can be a segment from second 31 to second 60 of version C of song B, and the second audio can be a segment from second 91 to second 120 of version C of song B.

[0115] In step A2, the first audio feature can be used as the baseline sample, the second audio feature as the positive sample, and the first reference audio feature as the negative sample. The model loss is determined based on the baseline sample, the negative sample, and the positive sample in the manner described above.

[0116] Among them, the smaller the difference between the benchmark sample and the negative sample, the greater the model loss; the greater the difference between the benchmark sample and the negative sample, the smaller the model loss; the smaller the difference between the benchmark sample and the positive sample, the smaller the model loss; and the greater the difference between the benchmark sample and the positive sample, the greater the model loss.

[0117] In step A3, the update magnitude of the model parameters that need to be updated in the first processing model can be determined based on the model loss. The model parameters of the first processing model are updated one by one based on the update magnitude. After updating each model parameter that needs to be updated, the update of the first processing model ends.

[0118] The model parameters that need to be updated in the first processing model may include all or part of the model parameters of the first processing model.

[0119] Methods for determining the update magnitude of model parameters based on model loss can be found in relevant technical documents and will not be elaborated here.

[0120] Optionally, after each execution of step A3, the first processing model at this time can be used to process the second audio, the first audio to be identified, and the first reference audio respectively to obtain new second audio features, new first audio features, and new first reference audio features. Based on the new second audio features, new first audio features, and new first reference audio features, step A2 is executed again to obtain a new model loss. If the new model loss does not meet the loss convergence condition, step A3 can be executed again. If the new model loss meets the loss convergence condition, A3 is no longer executed.

[0121] After determining that the model loss meets the loss convergence condition, step S102 can be executed again based on the updated first processing model to obtain new first predicted song information. The above update process can be repeated several times until the first predicted song information that meets the convergence condition is obtained based on the updated first processing model.

[0122] An alternative way to determine the model loss based on the first audio feature, the second audio feature, and the first reference audio feature is:

[0123] The model loss is determined based on the differences between the first audio feature and the first reference audio feature, and the differences between the first audio feature and the second audio feature.

[0124] The method for determining the model loss described above can be expressed by the following formula (1).

[0125]

[0126] Please see Figure 3 L is the final determined model loss, a represents the baseline sample, P represents the positive sample, n represents the negative sample, and d(a, p) represents the cosine distance between the baseline sample and the positive sample (equivalent to...). Figure 3The length of the line connecting the two samples), d(a,n) represents the cosine distance between the reference sample and the negative sample, and margin is a pre-set adjustable degree coefficient. The larger the cosine distance, the greater the difference between the two samples; the smaller the cosine distance, the smaller the difference between the two samples. For the calculation method of cosine distance, please refer to relevant technical documents.

[0127] Before updating the first processing model based on the above model loss, there may be a situation where the cosine distance between the reference sample and the negative sample is small, and the cosine distance between the reference sample and the positive sample is large. However, after updating the first processing model based on the above model loss, the cosine distance between the reference sample and the positive sample can be significantly reduced, and the cosine distance between the reference sample and the negative sample can be significantly increased.

[0128] Based on the definitions of baseline samples, positive samples, and negative samples above, it can be seen that baseline samples and positive samples are two audio features obtained after processing two audio segments belonging to the same song (i.e., the song represented by the real song information) by the first processing model. Baseline samples and negative samples are two audio features obtained after processing two audio segments belonging to two different songs by the first processing model. Therefore, updating the first processing model according to the above model loss can make the two audio features output by the first processing model corresponding to the same song more similar, and at the same time make the two audio features output by the first processing model corresponding to different songs less similar, thereby improving the accuracy of the audio features output by the first processing model, and thus improving the accuracy of the song information identified based on the audio features.

[0129] When obtaining information about the second song based on an audio fingerprint algorithm model, one possible way to determine the model loss based on the first audio feature, the second audio feature, and the first reference audio feature is:

[0130] The confidence level of the real song information is obtained. Both the confidence level and the real song information are obtained by processing the first audio to be identified by a pre-trained second processing model.

[0131] The model loss is determined based on the differences between the first audio feature and the first reference audio feature, the differences between the first audio feature and the second audio feature, and the confidence level.

[0132] In this embodiment, the loss L can be determined first according to the aforementioned formula (1), and then the loss L can be calculated according to the following formula (2). The calculated result L1 is used as the model loss for updating the first processing model.

[0133]

[0134] Where conf is the confidence score output by the audio fingerprinting algorithm model after processing the first audio to be identified. e is the base of the natural logarithm.

[0135] In some alternative embodiments, the second audio feature can be used as a baseline sample, the first audio feature as a positive sample, and the first reference audio feature as a negative sample, and the model loss can be determined according to the above method.

[0136] The model loss determined by any of the above methods is the ternary loss corresponding to the first audio to be identified. In some optional embodiments, other methods in the field of deep learning can also be used to process the first audio to be identified and the real song information to determine other types of model losses. For example, the first audio to be identified and the real song information can be processed based on other loss functions such as prototype loss in metric learning, and the obtained prototype loss can be used as the model loss to update the first processing model. The method for determining the prototype loss can be found in relevant technical documents and will not be elaborated here.

[0137] Optionally, in the method of this embodiment, multiple first audio segments to be identified can be obtained, and then each first audio segment to be identified can be processed by a pre-trained first processing model to obtain the first predicted song information of each first audio segment to be identified. Each first audio segment to be identified corresponds to a real song information. The real song information of different first audio segments to be identified can be the same or different, and the first predicted song information of different first audio segments to be identified can be the same or different.

[0138] In this case, if the first predicted song information of multiple first audios to be identified is inconsistent with the corresponding real song information, the model loss corresponding to each first audio to be identified can be determined according to the above method, and finally the first processing model can be updated according to the average of multiple model losses.

[0139] For example, suppose there are three first predicted song information for audios to be identified that are inconsistent with the corresponding real song information, and are referred to as audio 1, audio 2 and audio 3 respectively. Then, the model loss 1 corresponding to audio 1, the model loss 2 corresponding to audio 2 and the model loss 3 corresponding to audio 3 can be determined by any of the above methods for determining model loss. Then, the first processing model is updated according to the average value of model loss 1 to 3.

[0140] Optionally, after updating the first processing model based on the first audio features, the following steps may also be performed:

[0141] Each reference audio segment is processed according to the updated first processing model to update the reference audio features in the audio feature library.

[0142] Specifically, when the first processing model is updated and the first predicted song information obtained based on the updated first processing model is consistent with the real song information, each reference audio segment previously used to process and obtain reference audio features can be input into the updated first processing model. The updated first processing model is then used to process and obtain the updated reference audio features. Finally, the updated reference audio features replace the original reference audio features in the audio feature library.

[0143] For example, when constructing the audio feature library, the original song A in the music library is divided into M 15-second reference audio segments. The first processing model before the update is used to process these M 15-second reference audio segments to obtain M reference audio features in the audio feature library corresponding to the original song A.

[0144] After obtaining the updated first processing model and the output first predicted song information reaching the convergence condition, the updated first processing model can be used to process the M reference audio segments into which the original song A is divided, to obtain the M updated reference audio features corresponding to the original song A. Then, the original M reference audio features corresponding to the original song A are deleted from the audio feature library, and the M updated reference audio features corresponding to the original song A are added to the audio feature library.

[0145] This application provides an audio recognition device; please refer to [link / reference]. Figure 4 The device may include the following units.

[0146] The obtaining unit 401 is used to obtain the first audio to be identified and to obtain the real song information corresponding to the first audio to be identified; the real song information represents the song to which the first audio to be identified belongs.

[0147] The recognition unit 402 is used to recognize the first audio to be recognized according to the pre-trained first processing model, and obtain the first predicted song information of the first audio to be recognized.

[0148] The update unit 403 is used to update the first processing model according to the first audio to be identified and the real song information when it is determined that the first predicted song information is inconsistent with the real song information, until the convergence condition is reached and the trained first processing model is obtained; the trained first processing model is used to perform song recognition on the audio.

[0149] Optionally, when obtaining the real song information corresponding to the first audio to be identified, the obtaining unit 401 can be used for:

[0150] The first audio to be identified is processed according to the second processing model to obtain the second predicted song information and confidence level corresponding to the first audio to be identified.

[0151] If the confidence level is greater than the preset confidence level threshold, the second predicted song information is used as the real song information corresponding to the first audio to be identified.

[0152] Optionally, when the recognition unit 402 identifies the first audio to be recognized based on the pre-trained first processing model and obtains the first predicted song information of the first audio to be recognized, it can be used for:

[0153] The first audio to be identified is processed according to the pre-trained first processing model to obtain the first audio features;

[0154] The first audio feature is compared with multiple reference audio features in a pre-built audio feature library to determine the first reference audio feature that matches the first audio feature. The reference audio feature is obtained by processing the reference audio segment by the first processing model.

[0155] The song information corresponding to the first reference audio feature is determined as the first predicted song information of the first audio to be identified, and the song information corresponding to the first reference audio feature represents the song to which the corresponding reference audio segment belongs.

[0156] Optionally, when updating the first processing model based on the first audio to be identified and the real song information, the update unit 403 can be used for:

[0157] The second audio feature is obtained based on the real song information. The second audio feature is obtained by processing the second audio by the first processing model. The second audio is the audio contained in the song represented by the real song information.

[0158] The model loss is determined based on the first audio feature, the second audio feature, and the first reference audio feature;

[0159] The first processing model is updated based on the model loss.

[0160] Optionally, when the update unit 403 determines the model loss based on the first audio feature, the second audio feature, and the first reference audio feature, it can be used for:

[0161] The model loss is determined based on the differences between the first audio feature and the first reference audio feature, and the differences between the first audio feature and the second audio feature.

[0162] Optionally, when the update unit 403 determines the model loss based on the first audio feature, the second audio feature, and the first reference audio feature, it can be used for:

[0163] The confidence level of the real song information is obtained. Both the confidence level and the real song information are obtained by processing the first audio to be identified by a pre-trained second processing model.

[0164] The model loss is determined based on the differences between the first audio feature and the first reference audio feature, the differences between the first audio feature and the second audio feature, and the confidence level.

[0165] Optionally, after updating the first processing model based on the first audio to be identified and the real song information, the update unit 403 can also be used for:

[0166] Each reference audio segment is processed according to the updated first processing model to update the reference audio features in the audio feature library.

[0167] Optionally, when the recognition unit 402 processes the first audio to be recognized according to the pre-trained first processing model to obtain the first audio features, it can be used for:

[0168] Adjust the duration of the first audio file to be identified to the target duration to obtain the adjusted first audio file to be identified;

[0169] The first audio to be identified is processed and adjusted according to the pre-trained first processing model to obtain the first audio features of the first audio to be identified.

[0170] The working principle of the audio recognition device provided in this embodiment can be found in the relevant steps of the audio recognition method provided in any embodiment of this application, and will not be repeated here.

[0171] This application also provides a computer device; please refer to [link to relevant documentation]. Figure 5 The electronic device may include a memory 501 and a processor 502.

[0172] Memory 501 is used to store computer programs;

[0173] The processor 502 is used to execute the above-described computer program to implement the audio recognition method provided in any embodiment of this application.

[0174] This application also provides a computer program product that may include executable computer instructions; when the computer instructions are executed by a processor (e.g., the processor 502 described above), they are used to implement the audio recognition method provided in any embodiment of this application.

[0175] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0176] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0177] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0178] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0179] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An audio recognition method, characterized in that, include: Obtain the first audio to be identified, and then obtain the real song information corresponding to the first audio to be identified. The real song information represents the song to which the first audio to be identified belongs; The first audio to be identified is identified based on the pre-trained first processing model to obtain the first predicted song information of the first audio to be identified. If it is determined that the first predicted song information is inconsistent with the real song information, a second audio feature is obtained based on the real song information. The second audio feature is obtained by the first processing model processing the second audio, and the second audio is the audio contained in the song represented by the real song information. The model loss is determined based on the first audio feature, the second audio feature, and the first reference audio feature. The first audio feature is obtained by processing the first audio to be identified by a pre-trained first processing model. The first reference audio feature is the first reference audio feature that matches the first audio feature among a plurality of preset reference audio features. The first processing model is updated according to the model loss until the convergence condition is met, resulting in a trained first processing model; the trained first processing model is used for song recognition in audio.

2. The method according to claim 1, characterized in that, The step of obtaining the real song information corresponding to the first audio to be identified includes: The first audio to be identified is processed according to the second processing model to obtain the second predicted song information and confidence level corresponding to the first audio to be identified. If the confidence level is greater than a preset confidence threshold, the second predicted song information is used as the real song information corresponding to the first audio to be identified.

3. The method according to claim 1, characterized in that, The step of identifying the first audio to be identified based on a pre-trained first processing model to obtain the first predicted song information of the first audio to be identified includes: The first audio to be identified is processed according to the pre-trained first processing model to obtain the first audio feature; The first audio feature is compared with multiple reference audio features in a pre-built audio feature library to determine a first reference audio feature that matches the first audio feature. The reference audio feature is obtained by the first processing model processing the reference audio segment. The song information corresponding to the first reference audio feature is determined as the first predicted song information of the first audio to be identified, and the song information corresponding to the first reference audio feature represents the song to which the corresponding reference audio segment belongs.

4. The method according to claim 1, characterized in that, The step of determining the model loss based on the first audio feature, the second audio feature, and the first reference audio feature includes: The model loss is determined based on the differences between the first audio feature and the first reference audio feature, and the differences between the first audio feature and the second audio feature.

5. The method according to claim 1, characterized in that, The step of determining the model loss based on the first audio feature, the second audio feature, and the first reference audio feature includes: The confidence level of the real song information is obtained, and both the confidence level and the real song information are obtained by processing the first audio to be identified by a pre-trained second processing model. The model loss is determined based on the difference between the first audio feature and the first reference audio feature, the difference between the first audio feature and the second audio feature, and the confidence level.

6. The method according to claim 3, characterized in that, After updating the first processing model based on the first audio to be identified and the real song information, the method further includes: Each of the reference audio segments is processed according to the updated first processing model to update the reference audio features of the audio feature library.

7. An audio recognition device, characterized in that, include: The obtaining unit is used to obtain the first audio to be identified and to acquire the real song information corresponding to the first audio to be identified. The real song information represents the song to which the first audio to be identified belongs; The recognition unit is used to recognize the first audio to be recognized according to the pre-trained first processing model, and obtain the first predicted song information of the first audio to be recognized. The updating unit is used to obtain a second audio feature based on the real song information when it is determined that the first predicted song information is inconsistent with the real song information. The second audio feature is obtained by the first processing model processing the second audio, and the second audio is the audio contained in the song represented by the real song information. The model loss is determined based on the first audio feature, the second audio feature, and the first reference audio feature. The first audio feature is obtained by processing the first audio to be identified by a pre-trained first processing model. The first reference audio feature is the first reference audio feature that matches the first audio feature among a plurality of preset reference audio features. The first processing model is updated according to the model loss until the convergence condition is met, resulting in a trained first processing model; the trained first processing model is used for song recognition in audio.

8. A computer device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the audio recognition method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, Includes executable computer instructions; When the computer instructions are executed by the processor, they are used to implement the audio recognition method as described in any one of claims 1 to 6.

Images