Audio recognition method, device, equipment and program product

Abstract

The invention discloses an audio recognition method and device, equipment and a program product, and the method comprises the steps: obtaining a first to-be-recognized audio, and obtaining the real song information corresponding to the first to-be-recognized audio; the real song information represents a song to which the first to-be-identified audio belongs; identifying the first to-be-identified audio according to a pre-trained first processing model to obtain first predicted song information of the first to-be-identified audio; when it is determined that the first predicted song information is inconsistent with the real song information, a first processing model is updated according to the first audio to be recognized and the real song information until a convergence condition is reached, and a trained first processing model is obtained; the trained first processing model is used for performing song recognition on the audio.

Description

Technical Field

[0001] The present application relates to the technical field of audio recognition, and in particular to an audio recognition method, device, equipment and program product. Background Art

[0002] Currently, some applications have the function of identifying songs by listening. Based on this function, the application can identify which song a recorded audio belongs to. For example, while a user is singing song A, the application can record the audio of the user singing and identify that the audio belongs to song A.

[0003] The function of identifying songs by listening can be realized through a variety of technologies. One technical solution for realizing song identification by listening in the related art is to train a deep convolutional neural network model with a pre-collected music library, and then use the trained model to extract the audio features of the audio to be identified, and determine the song based on the extracted audio features.

[0004] This solution has the defect of low recognition accuracy in some scenarios. For example, it is easy for an audio clip of a user singing song A to be recognized as belonging to song B. Summary of the invention

[0005] To this end, this application discloses the following technical solutions:

[0006] The first aspect of the present application provides an audio recognition method, comprising:

[0007] Obtaining a first audio to be identified, and obtaining real song information corresponding to the first audio to be identified; the real song information represents the song to which the first audio to be identified belongs;

[0008] Recognize the first audio to be recognized according to the pre-trained first processing model to obtain first predicted song information of the first audio to be recognized;

[0009] When it is determined that the first predicted song information is inconsistent with the real song information, the first processing model is updated according to the first audio to be identified and the real song information until the convergence condition is reached to obtain a trained first processing model; the trained first processing model is used to perform song recognition on the audio.

[0010] Optionally, the obtaining real song information corresponding to the first audio to be identified includes:

[0011] Processing the first audio to be recognized according to the second processing model to obtain second predicted song information and confidence corresponding to the first audio to be recognized;

[0012] When the confidence level is greater than a preset confidence level threshold, the second predicted song information is used as the real song information corresponding to the first audio to be identified.

[0013] Optionally, the step of identifying the first audio to be identified according to the pre-trained first processing model to obtain first predicted song information of the first audio to be identified includes:

[0014] Processing the first audio to be recognized according to a pre-trained first processing model to obtain a first audio feature;

[0015] Comparing the first audio feature with a plurality of reference audio features in a pre-built audio feature library to determine a first reference audio feature that matches the first audio feature, the reference audio feature being obtained by processing a reference audio segment with the first processing model;

[0016] The song information corresponding to the first reference audio feature is determined as the first predicted song information of the first audio to be identified, and the song information corresponding to the first reference audio feature represents the song to which the corresponding reference audio clip belongs.

[0017] Optionally, updating the first processing model according to the first audio to be recognized and the real song information includes:

[0018] Obtain a second audio feature according to the real song information, where the second audio feature is obtained by processing a second audio by the first processing model, and the second audio is the audio contained in the song represented by the real song information;

[0019] determining a model loss based on the first audio feature, the second audio feature, and the first reference audio feature;

[0020] The first processing model is updated according to the model loss.

[0021] Optionally, determining the model loss according to the first audio feature, the second audio feature, and the first reference audio feature includes:

[0022] A model loss is determined based on a difference between the first audio feature and the first reference audio feature, and a difference between the first audio feature and the second audio feature.

[0023] Optionally, determining the model loss according to the first audio feature, the second audio feature, and the first reference audio feature includes:

[0024] Obtaining the confidence of the real song information, where both the confidence and the real song information are obtained by processing the first audio to be recognized by a pre-trained second processing model;

[0025] A model loss is determined based on a difference between the first audio feature and the first reference audio feature, a difference between the first audio feature and the second audio feature, and the confidence.

[0026] Optionally, after updating the first processing model according to the first audio to be recognized and the real song information, the method further includes:

[0027] Each of the reference audio segments is processed according to the updated first processing model to update each of the reference audio features in the audio feature library.

[0028] Optionally, the processing the first audio to be recognized according to the pre-trained first processing model to obtain the first audio feature includes:

[0029] Adjusting the duration of the first audio to be recognized to a target duration to obtain an adjusted first audio to be recognized;

[0030] The adjusted first audio to be recognized is processed according to a pre-trained first processing model to obtain a first audio feature of the first audio to be recognized.

[0031] A second aspect of the present application provides an audio recognition device, comprising:

[0032] An obtaining unit, configured to obtain a first audio to be identified, and acquire real song information corresponding to the first audio to be identified; the real song information represents the song to which the first audio to be identified belongs;

[0033] A recognition unit, configured to recognize the first audio to be recognized according to a pre-trained first processing model, and obtain first predicted song information of the first audio to be recognized;

[0034] An updating unit is used to update the first processing model according to the first audio to be identified and the real song information when it is determined that the first predicted song information is inconsistent with the real song information, until a convergence condition is reached to obtain a trained first processing model; the trained first processing model is used to perform song recognition on the audio.

[0035] A third aspect of the present application provides a computer device, including a memory and a processor;

[0036] The memory is used to store computer programs;

[0037] The processor is used to execute the computer program to implement the audio recognition method provided in any one of the first aspects of the present application.

[0038] A fourth aspect of the present application provides a computer program product, comprising executable computer instructions;

[0039] When the computer instructions are executed by the processor, they are used to implement the audio recognition method provided in any one of the first aspects of the present application.

[0040] The beneficial effects of this solution are:

[0041] By repeatedly updating the first processing model using the obtained audio to be identified and real song information, the first processing model can learn the features of the audio to be identified recorded in various recording scenarios (including but not limited to streets, shopping malls, etc.), thereby improving the performance of the first processing model, so that the first processing model can obtain audio features that accurately represent the audio to be identified when processing the audio to be identified recorded in various recording scenarios, thereby improving the accuracy of the song information obtained based on audio feature recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0043] Figure 1 is a flow chart of an audio recognition method provided by an embodiment of the present application;

[0044] Figure 2 is a flow chart of another audio recognition method provided by an embodiment of the present application;

[0045] Figure 3 It is a schematic diagram of the principle of a model loss provided in an embodiment of the present application;

[0046] Figure 4 is a structural schematic diagram of an audio recognition device provided in an embodiment of the present application;

[0047] Figure 5 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] This application embodiment provides an audio recognition method, see Figure 1, is a flow chart of the method, and the method may include the following steps.

[0050] S101, obtaining a first audio to be identified, and obtaining real song information corresponding to the first audio to be identified, where the real song information represents the song to which the first audio to be identified belongs.

[0051] The method provided in this embodiment may be executed by an electronic device used to run the first processing model, for example, by a server where the first processing model is located.

[0052] The first audio to be identified can be an audio segment collected by any user using relevant equipment. For example, when user A hears a piece of music on the street, user A can record it with his personal terminal device, and then operate the personal terminal device to upload the recorded audio segment to the server providing the song recognition service. The uploaded audio segment can be regarded as the first audio to be identified obtained in step S101.

[0053] It is understandable that the first audio to be recognized may include a piece of music heard by the user, or may include a variety of environmental noises in the user's environment, such as the noise of cars driving and honking on the street, the voices of people around in the supermarket, etc.

[0054] The real song information can indicate which song the first audio to be identified belongs to. For example, if a user uses a device to record a segment of song A and obtains the first audio to be identified, the real song information of the first audio to be identified can indicate that the first audio to be identified belongs to song A.

[0055] The real song information can be obtained in a variety of ways. For example, the first audio to be identified can be processed by a pre-trained second processing model to obtain the real song information, or the first audio to be identified can be manually identified and labeled to obtain the real song information.

[0056] Among them, manual identification and labeling can be performed by a user of a terminal device that uploads the first audio to be identified, or manual identification and labeling can be performed by relevant personnel on the server side that receives the first audio to be identified.

[0057] S102, identifying the first audio to be identified according to the pre-trained first processing model to obtain first predicted song information of the first audio to be identified.

[0058] The first predicted song information represents the song to which the first processing model predicts the first audio to be identified belongs. The first predicted song information may be consistent with or inconsistent with the actual song information. In combination with the above examples, the first predicted song information may represent that the first audio to be identified belongs to song A, or the first predicted song information may represent that the first audio to be identified belongs to song B.

[0059] The method of obtaining the first predicted song information based on the first processing model may be to process the first audio to be recognized using the first processing model, obtain a label representing a specific song output by the first processing model, and use the song information corresponding to the label as the first predicted song information; or:

[0060] S1, processing a first audio to be recognized according to a pre-trained first processing model to obtain a first audio feature;

[0061] S2, comparing the first audio feature with a plurality of reference audio features in a pre-built audio feature library to determine a first reference audio feature that matches the first audio feature, where the reference audio feature is obtained by processing a reference audio segment with a first processing model;

[0062] S3, determining the song information corresponding to the first reference audio feature as the first predicted song information of the first audio to be identified, the song information corresponding to the first reference audio feature represents the song to which the corresponding reference audio clip belongs.

[0063] In step S1, the audio data corresponding to the first audio to be identified can be directly input into the first processing model to obtain the first audio feature output by the first processing model; the Mel-frequency cepstral coefficient feature (MFCC) of the first audio to be identified can be extracted, and the MFCC can be input into the first processing model to obtain the first audio feature output by the first processing model; other spectral features of the first audio to be identified can also be extracted, and the extracted spectral features can be input into the first processing model to obtain the first audio feature output by the first processing model.

[0064] The audio feature library can be constructed in the following way:

[0065] Each song contained in the music library used for training is divided into several reference audio clips. For example, the original song A is divided into M reference audio clips of 15 seconds, the cover version of song A by singer B is divided into M reference audio clips of 15 seconds, the original song B is divided into N reference audio clips of 15 seconds, and the cover version of song B by singer D is divided into N reference audio clips of 15 seconds.

[0066] Use the first processing model to process each reference audio segment to obtain the reference audio features corresponding to each reference audio segment. Each reference audio segment corresponds to a reference audio feature. For example, use the first processing model to process the M reference audio segments into which the original song A is divided to obtain M reference audio features.

[0067] The method of processing and obtaining the reference audio features is shown in step S1.

[0068] Based on the audio feature library, the method for comparing the first audio feature and the reference audio feature in S2 can be to calculate the similarity between the first audio feature and each reference audio feature, and determine the reference audio feature with the largest similarity as the first reference audio feature that matches the first audio feature, and then the song information corresponding to the first reference audio feature can be determined as the first predicted song information of the first audio to be identified.

[0069] Exemplarily, assuming that the audio feature library contains 100 reference audio features, the similarity between the first audio feature and the 100 reference audio features can be calculated in S2, where the similarity of a certain reference audio feature is the maximum value among the 100 similarities, and then in S3, the reference audio feature is determined to be the first reference audio feature that matches the first audio feature, and the first reference audio feature is obtained by processing the reference audio segment X by the first processing model, and the reference audio segment X is segmented from the original song of song A. Therefore, the song information corresponding to the first reference audio feature can be song A, and then the first predicted song information of the first audio to be identified can be determined to be song A, that is, it is determined that the first audio to be identified is a segment of song A.

[0070] The first processing model can be any pre-trained model with a neural network structure. In some embodiments, the first processing model can be a pre-trained cover song recognition model. When the first processing model is a cover song recognition model, the first predicted song information can be obtained based on the above steps S1 to S3.

[0071] See also Figure 2 , taking the first processing model being a cover song recognition model as an example, the first processing model can be trained in the following manner.

[0072] First, a music library containing multiple same song groups may be constructed, each of which may correspond to a song, and each of which may contain different versions of audio data of the corresponding song.

[0073] Exemplarily, song A corresponds to same song group 1, song B corresponds to same song group 2, song A was originally sung by singer A, and the audio data obtained by recording when singer A sang song A is the original song data corresponding to song A. At the same time, song A was covered by singer B and singer C after being released. The audio data obtained by recording when singer B sang song A is the cover song data of singer B's version of song A corresponding to song A, and the audio data obtained by recording when singer C sang song A is the cover song data of singer C's version of song A corresponding to song A. Same song group 1 may include the original song data of song A, the cover song data of singer B's version of song A, and the cover song data of singer C's version of song A, that is, it includes the original song and two versions of the cover song.

[0074] For any same song group in the music library, you can select the original song and any version of the cover song contained in the same song group, and select another song from another same song group (it can be the original song or cover song of another same song group), and respectively capture a segment of the original song, a segment of the cover song, and a segment of the other song, thereby obtaining a triplet sample consisting of three segments.

[0075] Optionally, the clips can be cut according to a specific target duration, that is, each clip is a clip of the corresponding song that lasts for the target duration. The target duration can be set as needed, for example, the target duration can be 15 seconds.

[0076] For example, a triplet sample may include: a 15-second segment of the original song A, a 15-second segment of the cover version of song A by singer B, and a 15-second segment of the cover version of song B.

[0077] After obtaining a number of triplet samples in the above manner, these triplet samples can be used to train an initial neural network model with the goal of making the features of the original song segment and the cover song segment in the processed triplet samples as similar as possible, and the features of the original song segment and the features of other song segments as dissimilar as possible. When the model loss of the neural network model converges, the neural network model can be used as a trained cover recognition model.

[0078] For training a cover song recognition model using triplet samples, please refer to the relevant prior art and will not be described in detail.

[0079] S103, when it is determined that the first predicted song information is inconsistent with the actual song information, the first processing model is updated according to the first audio to be identified and the actual song information until the convergence condition is reached, thereby obtaining a trained first processing model; the trained first processing model is used for song identification of the audio.

[0080] In step S103, the first predicted song information obtained in S102 and the real song information obtained in S101 may be compared. If the two are inconsistent, the first processing model may be updated based on the first audio to be identified and the real song information. Specifically, all or part of the model parameters of the first processing model may be updated.

[0081] Each time the first processing model is updated according to S103, step S102 can be executed again to process the first audio to be identified based on the updated first processing model to obtain new first predicted song information. If the newly obtained first predicted song information is inconsistent with the actual song information, the above process of updating the first processing model and obtaining new first predicted song information based on the updated model is repeated until the obtained first predicted song information meets the convergence conditions.

[0082] Optionally, when the first processing model is updated in step S103, the adjustment range of the model parameters can be limited to a smaller range, that is, each time the model parameters of the first processing model are updated, only a small adjustment is made to the model parameters. The small adjustment here can be that the adjustment range of the model parameters during update is smaller than the adjustment range of the model parameters during training.

[0083] In this embodiment, the first predicted song information and the real song information are inconsistent, which can indicate that the song recognition result obtained by listening to songs based on the first processing model is wrong. Exemplarily, the first audio to be recognized is an audio recording of song A sung by any singer. In this case, the real song information indicates that the first audio to be recognized belongs to song A, while the first predicted song information obtained in S102 is song B. It can be considered that the first predicted song information and the real song information are inconsistent, and the song recognition result obtained by listening to songs based on the first processing model is wrong.

[0084] Correspondingly, reaching the convergence condition can indicate that the song recognition result obtained based on the first processing model is correct. In this embodiment, after each update of the first processing model, the first audio to be identified can be identified based on the updated first processing model to obtain new first predicted song information, and then the new first predicted song information is compared with the real song information. If the new first predicted song information is consistent with the real song information at this time, it can be determined that the first processing model obtained after this update has reached the convergence condition.

[0085] In combination with the above example, if the real song information indicates that the first audio to be identified belongs to song A, and the first predicted song information obtained based on the updated first processing model also indicates that the first audio to be identified belongs to song A, then it is determined that the updated first processing model meets the convergence conditions.

[0086] The reason why the recognition results of existing solutions are inaccurate is that there is a large gap between the songs in the music library used to train the model and the audio that actually needs to be recognized. For example, the songs used to train the model are usually recorded in a less noisy environment such as a recording studio, while the audio that actually needs to be recognized is usually recorded in a noisy environment such as a street or shopping mall. This gap causes the audio features extracted by the model to be unable to accurately represent the audio that actually needs to be recognized, which in turn causes the final recognition results to be inaccurate.

[0087] The beneficial effect of this embodiment is that, in response to the above-mentioned problem, this embodiment uses the obtained audio to be identified to repeatedly update the first processing model, so that the first processing model can learn the characteristics of the audio to be identified recorded in various recording scenarios (including but not limited to streets, shopping malls and other scenarios), thereby improving the performance of the first processing model, so that the first processing model can obtain more accurate first predicted song information when processing the audio to be identified recorded in various recording scenarios.

[0088] Optionally, in order to obtain a more accurate first audio feature, in S1, the duration of the first audio to be recognized may be adjusted as follows, and then the corresponding first audio feature may be obtained:

[0089] Adjusting the duration of the first audio to be recognized to the target duration to obtain an adjusted first audio to be recognized;

[0090] The adjusted first audio to be recognized is processed according to the pre-trained first processing model to obtain a first audio feature of the first audio to be recognized.

[0091] In the step of adjusting the duration, the actual duration of the first audio to be recognized may be compared with the target duration. If the actual duration is consistent with the target duration, the first audio to be recognized may be determined as the adjusted first audio to be recognized.

[0092] If the actual duration is less than the target duration, the first audio to be recognized may be padded with zeros at the front and back to obtain an adjusted first audio to be recognized having the target duration, for example, obtaining an adjusted first audio to be recognized having a duration of 15 seconds;

[0093] If the actual duration is greater than the target duration, the first audio to be recognized may be cropped to obtain an adjusted first audio to be recognized having the target duration.

[0094] After adjustment in the above manner, it can be ensured that the duration of the audio to be recognized processed by the first processing model and the duration of the segment used for training are both the target duration, thereby avoiding inaccurate audio features output by the first processing model due to inconsistent duration.

[0095] In some optional embodiments, the method of obtaining the real song information corresponding to the first audio to be identified may be:

[0096] Processing the first audio to be recognized according to the second processing model to obtain second predicted song information and confidence corresponding to the first audio to be recognized;

[0097] When the confidence level is greater than a preset confidence level threshold, the second predicted song information is used as the real song information corresponding to the first audio to be identified.

[0098] The second processing model may have a different model structure and model parameters from the first processing model, and the second processing model may have a higher accuracy than the first processing model, or the second processing model may have a higher accuracy than the first processing model at least in identifying part of the audio to be identified in a specific scenario.

[0099] The second processing model can be any model that is different from the first processing model and can be used to determine the song information of the audio to be recognized. For example, see Figure 2 , the second processing model can be an audio fingerprint algorithm model.

[0100] The training method and working principle of the audio fingerprint algorithm model can be found in technical materials in related fields and will not be elaborated here.

[0101] Generally, an audio fingerprint algorithm model has a high accuracy when used to identify part of the audio, but has a lower accuracy when used to identify another part of the audio. In order to distinguish these two situations, the processing results output by the audio fingerprint algorithm model may include a confidence level and song information corresponding to the audio to be identified. If the confidence level is high enough, it means that the identified song information has a high accuracy. If the confidence level is low, it means that the identified song information has a low accuracy.

[0102] like Figure 2 As shown, after the first audio to be identified is processed with the audio fingerprint algorithm model and the song information and confidence of the first audio to be identified are obtained, it can be determined whether the song information is a high-confidence result. If the confidence is greater than the confidence threshold, it can be determined that the song information is a high-confidence result, and the song information output by the audio fingerprint algorithm model is used as the real song information of the first audio to be identified (i.e. Figure 2 As shown in the song information 2), if the confidence is less than the confidence threshold, it can be determined that the song information output by the audio fingerprint algorithm model is not the true song information of the first audio to be identified.

[0103] The confidence threshold can be set as needed without limitation. For example, if the confidence value range is 0% to 100%, the confidence threshold can be set to 95%.

[0104] Optionally, if the confidence output by the audio fingerprint algorithm model is less than or equal to the confidence threshold, the real song information of the first predicted song information can be obtained by other methods (for example, obtained by manual labeling), or the method of this embodiment can be directly terminated.

[0105] like Figure 2 As shown, after obtaining song information 2 (i.e., real song information) and song information 1 (i.e., first predicted song information) of the first audio to be identified, if the two are inconsistent, the first processing model (i.e., cover recognition model) can be updated according to the first audio to be identified and the real song information.

[0106] In some optional embodiments, updating the first processing model according to the first audio to be recognized and the real song information may include:

[0107] A1, obtaining a second audio feature according to the real song information, where the second audio feature is obtained by processing the second audio with the first processing model, and the second audio is the audio contained in the song represented by the real song information;

[0108] A2, determining a model loss according to the first audio feature, the second audio feature and the first reference audio feature;

[0109] A3, updating the first processing model according to the model loss.

[0110] The first reference audio feature is obtained by processing the first reference audio segmented from the music library by the first processing model.

[0111] In step A1, a song corresponding to a first reference audio feature that matches the first audio feature can be determined in the audio feature library, and then another audio segment known to belong to the song indicated by the real song information can be determined as the second audio, and the second audio can be processed using the first processing model to obtain a second audio feature.

[0112] Exemplarily, assuming that the first reference audio feature that matches the first audio feature belongs to song A, the first audio to be identified is misidentified as belonging to song A based on the first processing model, and at the same time the actual song information of the first audio to be identified indicates that the first audio to be identified belongs to song B, then an audio segment that also belongs to song B can be determined as the second audio, and the second audio is processed using the first processing model to obtain the second audio feature.

[0113] The second audio may be a piece of audio pre-collected and stored in an audio library, or may be a piece of audio uploaded by a user.

[0114] The second audio and the first audio to be identified may be two different audio segments belonging to the song represented by the real song information. For example, the first audio to be identified may be a segment of the A version of song B, and the second audio may be a segment of the B version of song B, or the first audio to be identified may be a segment from the 31st to the 60th second of the C version of song B, and the second audio may be a segment from the 91st to the 120th second of the C version of song B.

[0115] In step A2, the first audio feature can be used as a benchmark sample, the second audio feature can be used as a positive sample, and the first reference audio feature can be used as a negative sample. The model loss can be determined in the above manner based on the benchmark sample, the negative sample, and the positive sample.

[0116] Among them, the smaller the difference between the benchmark sample and the negative sample, the greater the model loss, the greater the difference between the benchmark sample and the negative sample, the smaller the model loss, the smaller the difference between the benchmark sample and the positive sample, the smaller the model loss, and the greater the difference between the benchmark sample and the positive sample, the greater the model loss.

[0117] In step A3, the update amplitude of the model parameters that need to be updated in the first processing model can be determined according to the model loss, and the model parameters of the first processing model are updated one by one based on the update amplitude. After each model parameter that needs to be updated is updated, the update of the first processing model is completed.

[0118] The model parameters that need to be updated in the first processing model may include all or part of the model parameters of the first processing model.

[0119] The method for determining the update amplitude of model parameters based on model loss can be found in relevant technical materials and will not be described in detail here.

[0120] Optionally, after each execution of step A3, the second audio, the first audio to be identified and the first reference audio can be processed respectively using the first processing model at this time to obtain new second audio features, new first audio features and new first reference audio features, and step A2 is executed again based on the new second audio features, new first audio features and new first reference audio features to obtain a new model loss. If the new model loss does not meet the loss convergence condition, step A3 can be executed again. If the new model loss meets the loss convergence condition, A3 will no longer be executed.

[0121] After determining that the model loss meets the loss convergence condition, step S102 can be executed again based on the updated first processing model to obtain new first predicted song information. The above updating process can be repeated several times until the first predicted song information that meets the convergence condition is obtained based on the updated first processing model.

[0122] An optional way of determining the model loss according to the first audio feature, the second audio feature and the first reference audio feature is:

[0123] The model loss is determined according to a difference between the first audio feature and the first reference audio feature, and a difference between the first audio feature and the second audio feature.

[0124] The above method for determining the model loss can be expressed by the following formula (1).

[0125]

[0126] See also Figure 3 , L is the final model loss, a represents the benchmark sample, P represents the positive sample, n represents the negative sample, d(a, p) represents the cosine distance between the benchmark sample and the positive sample (equivalent to Figure 3The length of the line connecting the two in the figure), d(a, n) represents the cosine distance between the reference sample and the negative sample, and margin is a preset adjustable degree coefficient. The larger the cosine distance, the greater the difference between the two corresponding samples, and the smaller the cosine distance, the smaller the difference between the two corresponding samples. The calculation method of the cosine distance can be found in the relevant technical information.

[0127] Before updating the first processing model based on the above model loss, there may be a situation where the cosine distance between the benchmark sample and the negative sample is small, and the cosine distance between the benchmark sample and the positive sample is large. After updating the first processing model based on the above model loss, the cosine distance between the benchmark sample and the positive sample can be significantly reduced, and the cosine distance between the benchmark sample and the negative sample can be significantly increased.

[0128] Combined with the definitions of benchmark samples, positive samples and negative samples in the previous article, it can be seen that the benchmark sample and the positive sample are two audio features obtained after two audio segments belonging to the same song (i.e., the song represented by the real song information) are processed by the first processing model, and the benchmark sample and the negative sample are two audio features obtained after two audio segments belonging to two different songs are processed by the first processing model. Therefore, updating the first processing model according to the above model loss can make the two audio features corresponding to the same song output by the first processing model more similar, and the two audio features corresponding to different songs output less similar, thereby improving the accuracy of the audio features output by the first processing model, and then improving the accuracy of the song information identified based on the audio features.

[0129] In the case of obtaining the second song information based on the audio fingerprint algorithm model, an optional way of determining the model loss according to the first audio feature, the second audio feature and the first reference audio feature is:

[0130] Obtaining confidence of real song information, where both the confidence and the real song information are obtained by processing the first audio to be recognized by the pre-trained second processing model;

[0131] The model loss is determined according to a difference between the first audio feature and the first reference audio feature, a difference between the first audio feature and the second audio feature, and a confidence level.

[0132] In this embodiment, the loss L can first be determined according to the aforementioned formula (1), and then the loss L can be calculated according to the following formula (2), and the calculated result L1 is used as the model loss for updating the first processing model.

[0133]

[0134] Wherein, conf is the confidence level output by the audio fingerprint algorithm model after processing the first audio to be identified, and e is the base of the natural logarithm.

[0135] In some optional embodiments, the second audio feature may be used as a reference sample, the first audio feature may be used as a positive sample, and the first reference audio feature may be used as a negative sample to determine the model loss according to the above method.

[0136] The model loss determined by any of the above methods is the ternary loss corresponding to the first audio to be identified. In some optional embodiments, other methods in the field of deep learning can also be used to process the first audio to be identified and the real song information to determine other types of model losses. For example, the first audio to be identified and the real song information can be processed based on other loss functions such as prototype loss in metric learning, and the obtained prototype loss is used as the model loss to update the first processing model. The method for determining the prototype loss can be found in relevant technical materials and will not be repeated here.

[0137] Optionally, the method of this embodiment can obtain multiple segments of first audio to be identified, and then process each first audio to be identified using a pre-trained first processing model to obtain first predicted song information for each first audio to be identified. Each first audio to be identified corresponds to a real song information, and the real song information of different first audio to be identified may be the same or different, and the first predicted song information of different first audio to be identified may be the same or different.

[0138] In this case, if the first predicted song information of multiple first audios to be identified are inconsistent with the corresponding real song information, the model loss corresponding to each first audio to be identified can be determined according to the above method, and finally the first processing model can be updated according to the average value of the multiple model losses.

[0139] Exemplarily, assuming that the first predicted song information of three first audios to be identified are inconsistent with the corresponding real song information, and are respectively recorded as audio 1, audio 2 and audio 3, then any of the above-mentioned methods for determining the model loss can be used to determine the model loss 1 corresponding to audio 1, the model loss 2 corresponding to audio 2, and the model loss 3 corresponding to audio 3, and then the first processing model can be updated according to the average value of model losses 1 to 3.

[0140] Optionally, after updating the first processing model according to the first audio feature, the following steps may be further performed:

[0141] Each reference audio segment is processed according to the updated first processing model to update each reference audio feature of the audio feature library.

[0142] Specifically, when the first processing model is updated and the first predicted song information obtained based on the updated first processing model is consistent with the actual song information, each reference audio clip previously used to process to obtain the reference audio features can be input into the updated first processing model, and processed with the updated first processing model to obtain updated reference audio features, and then the updated reference audio features are used to replace the original reference audio features in the audio feature library.

[0143] Exemplarily, when constructing the audio feature library, the original song A in the music library is divided into M 15-second reference audio segments, and the M 15-second reference audio segments are processed by the first processing model before updating to obtain M reference audio features corresponding to the original song A in the audio feature library;

[0144] After obtaining the updated first processing model whose output first predicted song information meets the convergence condition, the updated first processing model can be used to process the M reference audio segments into which the original song of song A is divided, and M updated reference audio features corresponding to the original song of song A are obtained. Then, the original M reference audio features corresponding to the original song of song A are deleted from the audio feature library, and the M updated reference audio features corresponding to the original song of song A are added to the audio feature library.

[0145] This application embodiment provides an audio recognition device, see Figure 4 , the device may include the following units.

[0146] The obtaining unit 401 is used to obtain a first audio to be identified, and obtain real song information corresponding to the first audio to be identified; the real song information represents the song to which the first audio to be identified belongs;

[0147] The recognition unit 402 is used to recognize the first audio to be recognized according to the pre-trained first processing model to obtain the first predicted song information of the first audio to be recognized;

[0148] The updating unit 403 is used to update the first processing model according to the first audio to be identified and the real song information when it is determined that the first predicted song information is inconsistent with the real song information, until the convergence condition is reached to obtain the trained first processing model; the trained first processing model is used to perform song recognition on the audio.

[0149] Optionally, when the obtaining unit 401 obtains the real song information corresponding to the first audio to be identified, it can be used to:

[0150] Processing the first audio to be recognized according to the second processing model to obtain second predicted song information and confidence corresponding to the first audio to be recognized;

[0151] When the confidence level is greater than a preset confidence level threshold, the second predicted song information is used as the real song information corresponding to the first audio to be identified.

[0152] Optionally, when the recognition unit 402 recognizes the first audio to be recognized according to the pre-trained first processing model and obtains the first predicted song information of the first audio to be recognized, it can be used to:

[0153] Processing the first audio to be recognized according to the pre-trained first processing model to obtain a first audio feature;

[0154] Comparing the first audio feature with a plurality of reference audio features in a pre-built audio feature library to determine a first reference audio feature that matches the first audio feature, where the reference audio feature is obtained by processing a reference audio segment with a first processing model;

[0155] The song information corresponding to the first reference audio feature is determined as the first predicted song information of the first audio to be identified, and the song information corresponding to the first reference audio feature represents the song to which the corresponding reference audio clip belongs.

[0156] Optionally, when the updating unit 403 updates the first processing model according to the first audio to be recognized and the real song information, it can be used to:

[0157] Obtain a second audio feature according to the real song information, where the second audio feature is obtained by processing the second audio with the first processing model, and the second audio is the audio contained in the song represented by the real song information;

[0158] determining a model loss based on the first audio feature, the second audio feature, and the first reference audio feature;

[0159] The first processing model is updated according to the model loss.

[0160] Optionally, when the updating unit 403 determines the model loss according to the first audio feature, the second audio feature and the first reference audio feature, it can be used to:

[0161] The model loss is determined according to a difference between the first audio feature and the first reference audio feature, and a difference between the first audio feature and the second audio feature.

[0162] Optionally, when the updating unit 403 determines the model loss according to the first audio feature, the second audio feature and the first reference audio feature, it can be used to:

[0163] Obtaining confidence of real song information, where both the confidence and the real song information are obtained by processing the first audio to be recognized by the pre-trained second processing model;

[0164] The model loss is determined according to a difference between the first audio feature and the first reference audio feature, a difference between the first audio feature and the second audio feature, and a confidence level.

[0165] Optionally, after updating the first processing model according to the first audio to be recognized and the real song information, the updating unit 403 may also be used to:

[0166] Each reference audio segment is processed according to the updated first processing model to update each reference audio feature of the audio feature library.

[0167] Optionally, when the recognition unit 402 processes the first audio to be recognized according to the pre-trained first processing model to obtain the first audio feature, it can be used to:

[0168] Adjusting the duration of the first audio to be recognized to the target duration to obtain an adjusted first audio to be recognized;

[0169] The adjusted first audio to be recognized is processed according to the pre-trained first processing model to obtain a first audio feature of the first audio to be recognized.

[0170] The working principle of the audio recognition device provided in this embodiment can refer to the relevant steps of the audio recognition method provided in any embodiment of the present application, and will not be repeated here.

[0171] The present application also provides a computer device. Figure 5 , the electronic device may include a memory 501 and a processor 502.

[0172] The memory 501 is used to store computer programs;

[0173] The processor 502 is used to execute the above-mentioned computer program to implement the audio recognition method provided in any embodiment of the present application.

[0174] The embodiments of the present application also provide a computer program product, which may include executable computer instructions; when the computer instructions are executed by a processor (for example, by the above-mentioned processor 502), they are used to implement the audio recognition method provided by any embodiment of the present application.

[0175] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0176] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0177] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0178] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0179] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An audio recognition method, characterized in that: include: Obtaining a first audio to be recognized, and obtaining real song information corresponding to the first audio to be recognized; The real song information represents the song to which the first audio to be identified belongs; Recognize the first audio to be recognized according to the pre-trained first processing model to obtain first predicted song information of the first audio to be recognized; When it is determined that the first predicted song information is inconsistent with the real song information, the first processing model is updated according to the first audio to be identified and the real song information until the convergence condition is reached to obtain a trained first processing model; the trained first processing model is used to perform song recognition on the audio.

2. The method according to claim 1, characterized in that The obtaining of real song information corresponding to the first audio to be identified includes: Processing the first audio to be recognized according to the second processing model to obtain second predicted song information and confidence corresponding to the first audio to be recognized; When the confidence level is greater than a preset confidence level threshold, the second predicted song information is used as the real song information corresponding to the first audio to be identified.

3. The method according to claim 1, characterized in that The step of recognizing the first audio to be recognized according to the pre-trained first processing model to obtain first predicted song information of the first audio to be recognized includes: Processing the first audio to be recognized according to a pre-trained first processing model to obtain a first audio feature; Comparing the first audio feature with a plurality of reference audio features in a pre-built audio feature library to determine a first reference audio feature that matches the first audio feature, the reference audio feature being obtained by processing a reference audio segment with the first processing model; The song information corresponding to the first reference audio feature is determined as the first predicted song information of the first audio to be identified, and the song information corresponding to the first reference audio feature represents the song to which the corresponding reference audio clip belongs.

4. The method according to claim 3, characterized in that The updating of the first processing model according to the first audio to be recognized and the real song information until a convergence condition is reached to obtain a trained first processing model includes: Obtain a second audio feature according to the real song information, where the second audio feature is obtained by processing a second audio by the first processing model, and the second audio is the audio contained in the song represented by the real song information; determining a model loss based on the first audio feature, the second audio feature, and the first reference audio feature; The first processing model is updated according to the model loss until a convergence condition is reached to obtain a trained first processing model.

5. The method according to claim 4, characterized in that The determining the model loss according to the first audio feature, the second audio feature and the first reference audio feature comprises: A model loss is determined based on a difference between the first audio feature and the first reference audio feature, and a difference between the first audio feature and the second audio feature.

6. The method according to claim 4, characterized in that The determining the model loss according to the first audio feature, the second audio feature and the first reference audio feature comprises: Obtaining the confidence of the real song information, where both the confidence and the real song information are obtained by processing the first audio to be recognized by a pre-trained second processing model; A model loss is determined based on a difference between the first audio feature and the first reference audio feature, a difference between the first audio feature and the second audio feature, and the confidence.

7. The method according to claim 3, characterized in that After the first processing model is updated according to the first audio to be recognized and the real song information, the method further includes: Each of the reference audio segments is processed according to the updated first processing model to update each of the reference audio features in the audio feature library.

8. An audio recognition device, characterized in that: include: An obtaining unit, configured to obtain a first audio to be identified, and acquire real song information corresponding to the first audio to be identified; The real song information represents the song to which the first audio to be identified belongs; A recognition unit, configured to recognize the first audio to be recognized according to a pre-trained first processing model, and obtain first predicted song information of the first audio to be recognized; An updating unit is used to update the first processing model according to the first audio to be identified and the real song information when it is determined that the first predicted song information is inconsistent with the real song information, until a convergence condition is reached to obtain a trained first processing model; the trained first processing model is used to perform song recognition on the audio.

9. A computer device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the audio recognition method according to any one of claims 1 to 7.

10. A computer program product, characterized in that includes executable computer instructions; When the computer instructions are executed by a processor, they are used to implement the audio recognition method according to any one of claims 1 to 7.

Images