Training methods for matching models, audio recognition methods, and related equipment

Abstract

This application discloses a training method for a matching model, an audio recognition method, and related equipment. The training method includes: calculating the model loss function of the matching model based on the melodic feature distances between each current original full audio segment and its respective modified audio segment and other original audio segments; adjusting the model parameters of the initial matching model based on the model loss function until a convergence condition is met to obtain a target matching model; the target matching model is used to output the target melodic features corresponding to the audio segment to be identified, so as to identify the original full audio segment to which the audio segment to be identified belongs through the target melodic features. This application helps to fully learn the intrinsic relationship between multiple modified segments and the current original full audio as a whole, ensuring that each modified audio segment of the same current original full audio can cluster in the feature space, and distancing the feature relationships between the current original full audio and other original audio segments, thereby improving the recognition accuracy of modified audio such as cover songs.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to training methods for matching models, audio recognition methods, and related devices. Background Technology

[0002] In daily life, there are many audio files played after the update. In the field of music, in order to meet the need for music recognition, users often use audio recognition technology to find out the audio file of a certain pirated recording.

[0003] Currently, existing song recognition technology relies on the similarity between a recorded cover song and the original song to identify the corresponding original track. However, in reality, cover song recordings are usually short, and relying solely on the audio information extracted from the recording is insufficient, easily leading to inaccurate recognition or failure to identify the original song.

[0004] In this regard, the relevant technologies do not provide an effective solution. Summary of the Invention

[0005] This application provides a matching model training method, an audio recognition method, and related equipment to improve the recognition effect of modified audio segments.

[0006] The first aspect of this application provides a method for training a matching model, comprising:

[0007] Obtain the frequency domain features of each group of sample audio in the sample set; each group of sample audio includes the current original full audio, multiple modified audio segments corresponding to the current original full audio, and other original audio that is different from the current original full audio, and the current original full audio in each group of sample audio is different;

[0008] Each of the frequency domain features is input into the initial matching model to obtain the melody features of each of the sample audios;

[0009] The model loss function of the matching model is calculated based on the first melody feature distance between each of the current original full audio and each of the modified audio segments, and the second melody feature distance between each of the current original full audio and the other original audio in the same group;

[0010] The model parameters of the initial matching model are adjusted based on the model loss function until the convergence condition is met to obtain the target matching model. The convergence condition includes ensuring that the distances of each of the first melody features are less than or equal to a first threshold due to model parameter tuning, and that the distances of each of the second melody features are greater than or equal to a second threshold due to model parameter tuning, wherein the first threshold is less than the second threshold. The target matching model is used to output the target melody features corresponding to the audio segment to be identified, so that the original full audio to which the audio segment to be identified belongs can be identified through the target melody features.

[0011] Optionally, the process of calculating the model loss function of the matching model includes:

[0012] For each set of sample audio, calculate the first melodic feature distance between each modified audio segment and the corresponding original sub-segment, and the second melodic feature distance between each other original audio and the current original full audio within a preset length; the other original audio includes other original full audio or sub-segments of other original full audio.

[0013] The distances of each melody feature in the sample audio that meet the preset conditions are summed to obtain the sample loss function of the sample audio.

[0014] The loss function of each group of samples is used as the model loss function, or the loss function of each group of samples is calculated in a weighted manner to obtain the model loss function.

[0015] Optionally, the step of summing the distances of each melody feature in the sample audio that meets the preset conditions includes:

[0016] Filter out a preset number of modified audio segments whose first melody feature distance is greater than a preset distance;

[0017] Based on the triplet loss function rule, the distances of the first melody features and each of the second melody features filtered out from the sample audio are calculated to obtain the sample loss function of the sample audio.

[0018] Optionally, the step of summing the distances of each melody feature in the sample audio that meets the preset conditions includes:

[0019] Filter out a preset number of modified audio segments whose first melody feature distance is greater than a preset distance;

[0020] Based on the comparison results of text feature content and / or spectral peaks among the preset number of the modified audio segments, a weighting coefficient is assigned to each of the filtered first melody feature distances;

[0021] The sample loss function of the sample audio is calculated by weighting and fusing the first melody feature distance based on the weighted coefficients and adding the second melody feature distance of the same group.

[0022] Optionally, obtaining the frequency domain features of each group of sample audio in the sample set includes:

[0023] According to the preset slice length and slice offset, the modified full audio, the current original full audio, and the other original full audio are sliced ​​to obtain multiple modified audio segments, original sub-segments used to constitute the current original full audio, and at least one other sub-segment as the other original audio; wherein, the slice length and slice offset are determined according to the audio segment playback duration or lyrics arrangement rules;

[0024] Extract the frequency domain features of each group of original sub-segments, other original audio files, and modified audio segments.

[0025] Optionally, obtaining the target matching model until the convergence condition is met includes:

[0026] If, after multiple rounds of model parameter tuning, the distance of the first melody feature in each round is less than or equal to the first threshold, and the distance of the second melody feature in each round is greater than or equal to the second threshold, then the model obtained from the latest parameter tuning is considered to be the target matching model.

[0027] A second aspect of this application provides an audio recognition method, including:

[0028] The audio segment to be identified is input into the target matching model to obtain the target melody features corresponding to the audio segment to be identified; wherein the target matching model is trained according to the training method described in the first aspect;

[0029] Calculate the melody feature distance between the target melody feature and the melody features of each original audio file;

[0030] The original full audio corresponding to the one with the smallest distance among the melodic feature distances is determined as the target full audio to which the audio segment to be identified belongs.

[0031] Optionally, if the feature distance is a multi-type distance metric including cosine distance or Euclidean distance, then determining the original full audio corresponding to the smallest distance among the melodic feature distances as the target full audio to which the audio segment to be identified belongs includes:

[0032] For each original audio source, the melody feature distance is weighted and calculated to obtain the distance fusion result for each original full audio source.

[0033] The original full audio corresponding to the smallest result among the various distance fusion results is determined as the target full audio.

[0034] A third aspect of this application provides an electronic device, including:

[0035] Central processing unit, memory, and input / output interfaces;

[0036] The memory is either a short-term storage memory or a persistent storage memory;

[0037] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect of the embodiments of this application or any specific implementation thereof.

[0038] A fourth aspect of this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect of this application or any specific implementation thereof.

[0039] The fifth aspect of this application provides a computer program product comprising instructions or a computer program, which, when run on a computer, causes the computer to perform the method described in the first aspect of this application or any specific implementation thereof.

[0040] As can be seen from the above technical solutions, the embodiments of this application have at least the following advantages:

[0041] From the perspective of sample input, extracting the current original full audio and its corresponding multiple modified audio segments as samples can effectively prevent the randomness of training results caused by a single modified segment. Furthermore, calculating the feature distance between the melodic features of each sample audio helps to fully learn the intrinsic relationship between multiple modified segments and the current original full audio as a whole. This ensures that modified audio segments of the same current original full audio can cluster in the feature space and distance the feature connections between the current original full audio and other original audio, thereby improving the accuracy of recognizing modified audio such as covers as belonging to the current original full audio. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0043] It should be noted that although the steps in the flowcharts (if any) involved in the embodiments are drawn sequentially according to the direction of the arrows, unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0044] Figure 1 This is a schematic diagram of a system architecture for a method according to an embodiment of this application;

[0045] Figure 2 This is a schematic flowchart illustrating the training method of the matching model in an embodiment of this application.

[0046] Figure 3 This is another flowchart illustrating the training method of the matching model in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of a slice of an embodiment of this application;

[0048] Figure 5 This is a flowchart illustrating the audio recognition method according to an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] In the following description, expressions such as "one specific implementation" or "one specific example" describe a subset of all possible embodiments. However, it is understood that "one specific implementation" or "one specific example" can be the same or different subset of all possible embodiments and can be combined with each other without conflict. In the following description, the term "multiple" means at least two. When a certain value mentioned in this application reaches a threshold (if it exists), in some specific examples, it may include the former being greater than the latter. When "any" or "at least one" or similar expressions are mentioned, it specifically refers to any one of the listed examples or any combination of these examples.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0054] Please see as follows Figure 1 The diagram shows a system architecture of an embodiment of this application. The audio recognition method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 101 via a network, and data storage system 100 stores the data that server 101 needs to process. Data storage system 100 can be integrated onto server 101 or placed in the cloud or on other network servers. Terminal 102 can acquire a user-recorded audio segment to be identified and transmit it to server 101. Server 101 can obtain the target melody features of the audio segment to be identified through a trained target matching model, and identify the original full audio file to which the audio segment belongs based on these target melody features. Then, server 101 can return the audio information of the original full audio file (such as the original song playback resource, the original song name, and the original singer, etc.) to terminal 102, thereby fulfilling the need for song recognition based on modified sub-segments such as cover songs, enhancing the user experience. Taking songs (or crosstalk, recitation, etc.) as audio as an example, the sample audio used to train the target matching model can be the original full song and multiple modified audio segments corresponding to the original full song, such as a live version segment of the original song "Seven Mile Fragrance" or a new cover version segment. Alternatively, the sample audio can also be other original songs such as "Later".

[0055] The aforementioned terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. The server 101 can be implemented using a standalone server or a server cluster composed of multiple servers. It should be noted that the method provided in this application embodiment can be implemented jointly by the terminal device and the server as described above, or it can be implemented entirely on the server side, or it can be implemented entirely on the terminal device side. The specific implementation can be determined according to the actual application scenario, and no limitation is made here.

[0056] The method described in this application will be further explained in detail below.

[0057] Please see Figure 2 The first aspect of this application provides a specific embodiment of a training method for a matching model, which includes the following steps:

[0058] Step 21: Obtain the frequency domain features of each group of sample audio in the sample set.

[0059] Using songs as an audio example, in song recognition scenarios, when the recorded audio sample is the original song, even with some background noise, existing song recognition technologies can still accurately match the original song using audio fingerprints. However, when users need to identify new modified works (such as cover songs or melody adaptations), fingerprint matching struggles to achieve ideal recognition results. This is because the fingerprints for new works are not extracted and added to the database in a timely manner, and because there are certain differences between modified and original works, making it difficult to match the modified work with the corresponding original full song, or resulting in a vague match to other full songs. Therefore, it is necessary to provide a method that can accurately identify modified audio segments to enhance the recall effect of the original full audio.

[0060] Specifically, the sample audio can include the current original full audio, multiple modified audio segments corresponding to the current original full audio, and other original audio that is different from the current original full audio. The current original full audio in each group of sample audio is different. Taking a song as an example, here we select multiple modified cover segments from each current original full audio (which can be called the whole song) as sample audio, such as multiple segments broken down by lyrics from the live version of "Qilixiang". This helps to introduce the inherent connection between the whole song and different cover segments during the training process, enrich the audio melody information in the training set, and associate different segments within the same song, thereby improving the accuracy of cover segment recognition. Among them, other original audio refers to audio that is different from the current original full audio of "Qilixiang", such as the original full audio of "Later" or its sub-segments.

[0061] The aforementioned frequency domain features include, but are not limited to, one or more of the following audio features: Mel-Frequency Cepstral Coefficient (MFCC), logarithmic Mel-spectral features, spectral difference amplitude, and spectral centroid. These frequency domain features can be derived from audio time-domain features, etc.

[0062] Step 22: Input each frequency domain feature into the initial matching model to obtain the melody features of each sample audio.

[0063] After obtaining the frequency domain features of the original full song and its multiple cover versions, these features can be input into the initial matching model to obtain the melodic features of each sample audio. These melodic features can be called embedding features, and can specifically be any of the features of pitch, duration, and volume. They can be used to express the characteristics of the audio in terms of mode, rhythm, meter, dynamics, timbre, and other performance styles. In practical applications, embedding features can be represented in vector or scalar form.

[0064] Specifically, a residual neural network (ResNet) model can be used as the initial matching model, such as the ResNet18 model, VGG network, or Inception network, which use residual structures. The use of residual networks here can ensure that the network will not have gradient problems, thereby enhancing the model's learning ability and improving its transferability.

[0065] Step 23: Calculate the model loss function of the matching model based on the first melody feature distance between each current original full audio and its respective modified audio segment, and the second melody feature distance between each current original full audio and other original audios in the same group.

[0066] After obtaining the melodic features of the audio, the similarity of melodic features between the current original full-length song and its respective cover segments, as well as other original audio tracks in the same group, can be compared. This allows for the determination of the model's loss function, which can be used to adjust the model parameters of the initial matching model to train a target matching model with ideal matching results. It should be noted that the corresponding sample audio tracks can be grouped into groups such as Group 1, Group 2, etc., based on the different current original full-length audio tracks; that is, the group number within the sample set can be determined based on the current original full-length audio tracks.

[0067] Step 24: Adjust the model parameters of the initial matching model based on the model loss function until the convergence condition is met to obtain the target matching model.

[0068] The convergence conditions include ensuring that the distances of each first melody feature are less than or equal to a first threshold due to model parameter tuning, and that the distances of each second melody feature are greater than or equal to a second threshold due to model parameter tuning, with the first threshold being less than the second threshold. The target matching model is used to output the target melody features corresponding to the audio segment to be identified (such as a cover song segment recorded), so that the original full audio to which the audio segment to be identified belongs can be identified through the target melody features, thereby enhancing the accurate positioning effect of the full audio to which the recorded segment belongs, such as helping to locate the corresponding position of the cover song segment in the original full song.

[0069] In summary, from the perspective of sample input, this application's embodiments extract the current original full audio and its corresponding multiple modified audio segments as samples. This helps to associate the inherent melodic characteristics between the modified audio segments and the original song, thereby effectively preventing the randomness of training results caused by a single modified segment. Furthermore, calculating the feature distance between the melodic features of each sample audio helps to fully learn the inherent connections between multiple modified segments and the current original full audio as a whole. This ensures that modified audio segments of the same current original full audio can cluster in the feature space and distance the feature connections between the current original full audio and other original audio, thereby improving the accuracy of identifying the current original full audio to which modified audio such as cover songs belongs.

[0070] Based on the examples above, some specific possible implementation examples will be provided below. In practical applications, the implementation details of these examples can be combined as needed according to the corresponding functional principles and application logic.

[0071] Please see Figures 2 to 4 This application provides another specific embodiment of a training method for a matching model, which includes the following steps:

[0072] Step 21: Obtain the frequency domain features of each group of sample audio in the sample set.

[0073] In some specific examples, the specific operation process of step 21 may include: slicing the modified full audio, the current original full audio, and other original full audio according to a preset slice length and slice offset, to obtain multiple modified audio segments, original sub-segments used to constitute the current original full audio, and at least one other sub-segment as other original audio; wherein, the slice length and slice offset are determined according to the audio segment playback duration or lyrics arrangement rules; and extracting the frequency domain features of each group of original sub-segments, other original audio, and modified audio segments.

[0074] Taking one set of sample audio from the sample set as an example, the extracted original full audio, such as the original full song of "Qilixiang," can be recorded as the anchor sample, while multiple modified full audios, such as the live version of "Qilixiang" and Li's cover version, can be recorded as positive samples. Then, the original full audio and each modified full audio (which can be referred to as the modified full song) can be sliced ​​(or segmented). The slice length and slice offset of different sets of sample audio can be the same or different, and can be set by any actual factors such as the playback duration of the original full audio and / or the modified full audio, and the lyrics arrangement rules. The lyrics arrangement rules can be word / sentence length or the order of main and secondary lyrics, etc. Figure 4 As shown, the slice offset can be used to provide feedback on the overlap length formed by the beginning and end of adjacent segments. It is understandable that, since it is usually unknown from which playback moment the user started recording the modified audio segment (such as a cover song segment), in order to ensure that the segment information is as complete as possible without loss, or to prevent the difference in results caused by adjacent segments, the slice offset can be set specifically. Among them, considering that the smaller the slice offset, the more accurate the output result, but the larger the calculation, the slice offset can be set as needed according to the slice length, etc. For example, the slice length and slice offset can be preset to 15 seconds and 7.5 seconds, respectively.

[0075] In some specific examples, slicing the original full audio file may not be necessary. However, comparatively, the feature distance between the sliced ​​original sub-segments and the corresponding modified audio segments provides better performance for model training than the unsliced ​​version. Similarly, other original audio files can be sliced ​​or left unsliced ​​as described above.

[0076] Step 22: Input each frequency domain feature into the initial matching model to obtain the melody features of each sample audio.

[0077] Step 23: Calculate the model loss function of the matching model based on the melodic feature distance between each current original full audio and its respective modified audio segment, and the second melodic feature distance between each current original full audio and other original audios in the same group.

[0078] In some specific examples, the specific operation process of step 23 may include: for each group of sample audio, calculating the first melodic feature distance between each modified audio segment and the original sub-segment of the corresponding length, and the second melodic feature distance between each other original audio and the current original full audio within a preset length; other original audio includes sub-segments in other original full audio or other original full audio; summing the melodic feature distances in the sample audio that meet the preset conditions to obtain the sample loss function of the sample audio; using each group of sample loss functions as the model loss function, or weighting each group of sample loss functions to obtain the model loss function.

[0079] When slicing the original full audio, a first melodic feature distance d(a, p) can be calculated between each modified audio segment (positive-x) and the corresponding original sub-segment (anchor-x), and a second melodic feature distance d(a, n) between each other original audio segment (negative-x) and the current original full audio within a preset segment length. In this embodiment, through the exemplary explanation between anchor and positive, similar exemplary explanations can be obtained regarding anchor and negative, such as slicing, similarity calculation, etc. As one possible implementation, all melodic feature distances can be applied to adjust the model parameters of the initial matching model; or, as another possible implementation, without slicing the original full audio, the melodic feature distance d(a, p) between each modified audio segment (positive-x) and the current original full audio (anchor) can be calculated. In comparison, feature comparison between segments is more granular and accurate than feature comparison between a segment and the whole song; therefore, the subsequent application effect of the former's comparison result, such as the feature distance (d), will be better.

[0080] Furthermore, in practical applications, the sample loss function of each set of sample audio can be used to tune the parameters of the initial matching model, meaning that the sample loss function of each set can be directly used as the model loss function. Alternatively, the sample loss functions of all sets of sample audio can be summed (e.g., by taking the mean or by weighting) to obtain a final loss function as the model loss function, which can then be used for unified model tuning. The timing or number of model tuning iterations can be selected as needed. The weight coefficients here can be determined based on the text feature content (e.g., the amount of lyrics) of each audio segment. For details, please refer to the process of assigning weights to each revised audio segment below; it will not be elaborated here.

[0081] In some specific examples, the specific operation process of "summing up the distances of each melody feature in the sample audio that meet the preset conditions" may include: filtering out a preset number of modified audio segments whose first melody feature distance is greater than the preset distance; assigning weight coefficients to each filtered first melody feature distance based on the comparison results of text feature content and / or spectral peaks among the preset number of modified audio segments; and weighting and fusing the filtered first melody feature distances based on the weight coefficients, and adding the second melody feature distances of the same group to calculate the sample loss function of the sample audio.

[0082] For this approach, if only one segment (positive-x) is selected, while the anchor is the entire song, it is difficult to learn the local correspondence between the features of the entire song and the segment. Therefore, in order to comprehensively learn the intrinsic relationship between the entire song and the segments, multiple positive-x segments can be introduced by slicing, such as m (m is an integer greater than 1) cover segments of different covers of the entire song. In some specific examples, loss may not be calculated for all positive segments because the embedding distance d (such as Euclidean distance or cosine distance) between some positive-x segments and the entire song is very close and does not require optimization. Therefore, some segment positive-x segments that meet the preset conditions can be selected to calculate the loss and optimize. For example, for each group of sample audio (which can be grouped according to the original full song), the embedding distance d between each positive-x segment and the anchor can be calculated, and based on the pre-set distance threshold a0, i (i is greater than 1) positive embedding sets (positive_emb1, positive_emb1, ..., positive_embi) with a distance greater than a0 can be filtered out. As one possible implementation, the mean distance between the embeddings of the i embeddings and the anchor can be calculated, such as (d1+…+di) / i. In some examples, this mean distance (d1+…+di) / i can be used as the distance between the original full track (anchorh) and the modified full track (positive), and as the sample loss function for each group of sample audios. Di represents the feature distance between the melody feature of the filtered i-th positive segment and the melody feature of the corresponding anchorh audio (i.e., the original full track or a segment of the original track of the corresponding length). Of course, without considering the computational burden, the loss can also be calculated and optimized for all positive-x segments in each group of sample audios. As another possible implementation, among the filtered i positive (i.e. multi-positive) segments, those segments with a high content of text (such as lyrics) features and / or high spectral peaks can be assigned larger weight coefficients. Then, the sample loss function of each group of sample audio can be calculated based on the weight coefficients assigned to each segment and the distance d(a,n) of each second melody feature. For example, the sample loss function of each group of sample audio = d1*weight coefficient_1 + ... + di*weight coefficient_i + each d(a,n). The d(a,n) can also be similarly weighted and fused with each other.

[0083] In practical applications, although two songs may have different melodies, there may be instances where a short segment of the melody is similar. This can easily lead to the false recall of other original songs. Therefore, during the training phase, other original audio files can be introduced as sample audio files to learn the differences between different song segments and avoid false recalls caused by local similarities between segments of different songs. Specifically, the similarity between other original full audio files and the current original full audio file can be determined by whether the similarity of text features (such as song title or lyrics) and / or rhythm features (such as time signature) exceeds a preset similarity threshold, thereby distinguishing different original audio files, such as distinguishing between the original versions of "Later" and "Rice Fragrance".

[0084] For example, if the original full-audio anchor is the original version of "Qilixiang," other original full-audio tracks (which can be denoted as negative samples) could be the original versions of other songs such as "Later" and "Ten Years." These three types of audio, along with the modified full-audio positive tracks, can be considered a triple (a, p, n), i.e., a set of sample audio. Multiple triples can be extracted to form a sample set. The melody features (embedding) of the negative samples can also be output through the sample frequency domain features, such as MFCC features, input into the matching model. Similar to the slicing process for the modified full-audio tracks described above, multiple sub-segments (i.e., other sub-segments) can be obtained for each other original full-audio track. It should be noted that the slice length and slice offset between the original full-audio tracks, modified full-audio tracks, and other original full-audio tracks (i.e., triples) can be the same or different, depending on the specific situation; of course, it is also possible to choose not to slice the other original full-audio tracks. Similarly, the second melody feature distance can be the embedding distance between the anchor-x segment and the negative-x segment or the entire negative song, or the embedding distance d(a,n) between the entire anchor audio and the entire negative audio. For details, please refer to the above explanation of d(a,p), which will not be repeated here.

[0085] Euclidean distance or cosine distance, etc., can be understood as a comparison of the loss caused by two parameters. Therefore, it is understandable that the feature distance between the anchor and each positive segment, and between the anchor and the negative segment can be characterized by squared loss function, absolute value loss function, etc.

[0086] Step 24: Adjust the model parameters of the initial matching model based on the model loss function until the convergence condition is met to obtain the target matching model.

[0087] When other (negative) audio is applied, step 24 may include the following: if, after multiple rounds of model parameter tuning, the first melody feature distance in each round is less than or equal to the first threshold, and the second melody feature distance in each round is greater than or equal to the second threshold, then the model obtained from the latest parameter tuning is considered a target matching model. The specific number of rounds can be determined based on practical experience, such as setting it to 3 rounds. Referring to the loss function across multiple rounds helps improve the model's prediction accuracy and reliability.

[0088] like Figure 3 As shown, as another possible implementation, the specific operation process of "summing up the distances of each melody feature in the sample audio that meet the preset conditions" may include: filtering out a preset number of modified audio segments whose first melody feature distance is greater than the preset distance; and calculating the first melody feature distance and each second melody feature distance filtered out in the sample audio according to the triplet loss function rule to obtain the sample loss function of the sample audio.

[0089] A model loss function suitable for this task can be constructed using the triplet-loss concept (which can be called multi-positive-triplet-loss), for example:

[0090] L=max(d(a,p)-d(a,n)+margin,0);

[0091] Where d(a,p) is the average distance between the anchor and the i filtered positive segments regarding the embedding features, d(a,n) is the embedding feature distance between the anchor and the negative segments, and margin is an adjustable coefficient (which can be a user-defined positive number). Appropriately determining the margin value can ensure network convergence and give the model good predictive performance. Here, anchor can specifically refer to the original full song or its sub-segments, and negative can specifically refer to other full songs or their sub-segments. The purpose of the triplet loss function is to make the embedding features between the original song's anchor and the positive segments of the cover song closer together, such as causing the embeddings of each segment of the same song to converge more towards the center of the original song, while simultaneously making the embedding features of the negative segments of the original song more distant from those of other songs. Of course, without considering computational power, positive segments can also be left unfiltered, i.e., d(a,p) corresponding to all positive segments can be applied. The triplet loss value L in the above equation can be regarded as a sample loss function generated by a set of sample audios, where d(*,*) can refer to the Euclidean distance or cosine distance equidistance expression; correspondingly, the sample loss function generated by all sets of sample audios in the sample set can be summed and expressed as the following triplet-loss formal formula:

[0092]

[0093] Wherein, the mapping function of the i-th sample x is f(x), which can be regarded as the output result of the matching model on x, N can be the size of the entire sample set, the input x of each triplet is (a, p, n), and α is the margin value; It represents the square of the 2-norm, which is used to calculate the spatial length of the Euclidean distance or the magnitude of a vector.

[0094] Finally, the initial matching model can be trained using the above multi-positive-triplet-loss results until the loss function converges to obtain the target matching model, so that the matching model can accurately output the embedding features of the audio to be identified.

[0095] In summary, this application proposes a multi-positive triplet segment training method, primarily targeting triplet loss optimization in adaptation scenarios such as cover songs. By introducing the original song's entire anchor and multiple positive cover segments, the feature distance between the original song and multiple cover segments is calculated. A set distance threshold is used to select the more distant positive segments for optimization, ensuring that cover segments of the same song cluster within the feature space. Simultaneously, negative segments from other songs are selected to ensure that the training process incorporates the differences between segments from different songs. In other words, the training objective or advantage of this application is to bring the embedding features of the original song and the cover song closer together, and to make the embedding features of the original song and other (cover) songs more distant. Therefore, this application considers the melodic information of the entire song, rather than learning solely through direct associations between segments, thereby associating the melodic characteristics between the cover segments and the original song and improving the accuracy of cover song recognition. Among them, the reconstruction of the loss function for the special scenario of cover songs solves the problem of isolation between different segments of the same song, and makes up for the gap in the original song recognition of a large number of modified audio fingerprints that cannot be included in the database in time, thereby improving the effective recall rate of the song recognition function.

[0096] Please see Figure 5 The second aspect of this application provides a specific embodiment of an audio recognition method, which includes the following operational steps:

[0097] 51. Input the audio segment to be identified into the target matching model to obtain the target melody features corresponding to the audio segment to be identified.

[0098] The target matching model is trained according to the training method in the first aspect. The audio segment to be identified can be a cover song segment or the original song segment; for example, the target matching model can output the target melody features of a recorded cover song segment.

[0099] 52. Calculate the melodic feature distance between the target melody feature and the melody features of each original audio file.

[0100] Specifically, the target melody features can be compared with the melody features (embedding) of each original audio file in the library to obtain the melody feature distance, such as Euclidean distance or cosine distance; the original audio file here can specifically refer to the original full song or a sub-segment of the original full song.

[0101] 53. The original full audio corresponding to the one with the smallest distance among the melodic feature distances is determined as the target full audio corresponding to the audio segment to be identified.

[0102] In some specific examples, if the feature distance is a multi-type distance metric including cosine distance or Euclidean distance, then step 53 may include: for each original audio melody feature distance, weighted calculation of different types of feature distances to obtain the distance fusion result of each original full audio; and determining the original full audio corresponding to the smallest result among the distance fusion results as the target full audio. For example, the distance fusion result = cosine distance * weight coefficient_1 + Euclidean distance * weight coefficient_2, where weight coefficient_1 and weight coefficient_2 can be set according to practical experience.

[0103] In summary, this application proposes a multi-positive triplet-based method for identifying cover songs. It optimizes the method for short cover songs, addressing the challenge of matching original songs based on melodic characteristics. By optimizing the triplet construction and loss calculation strategy of the triplet loss function, the model can learn the differences between different song segments, as well as the connections between different segments of the same song and between segments of the same song and the original song. This allows the model to recall the original song corresponding to the segment with the highest similarity, thereby further improving the accuracy of the cover song identification model and providing a better user experience for song recognition.

[0104] Please see Figure 6 The electronic device 600 of this application embodiment may include one or more central processing units (CPUs) 601 and a memory 605, wherein the memory 605 stores one or more applications or data.

[0105] The memory 605 can be volatile or persistent storage. The program stored in the memory 605 can include one or more modules, each module including a series of instruction operations on the electronic device. Furthermore, the central processing unit 601 can be configured to communicate with the memory 605 and execute the series of instruction operations stored in the memory 605 on the electronic device 600.

[0106] Electronic device 600 may also include one or more power supplies 602, one or more wired or wireless network interfaces 603, one or more input / output interfaces 604, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0107] The central processing unit 601 can perform the operations performed by the first aspect or any specific method embodiment of the first aspect, which will not be described in detail here.

[0108] This application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method as described in any specific implementation of the first or second aspect above.

[0109] This application provides a computer program product containing instructions or computer programs, which, when run on a computer, causes the computer to perform the method as described in any specific implementation of the first or second aspect above.

[0110] It is understood that, in the various embodiments of this application, the sequence number of each step does not imply the order of execution. The execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system (if it exists) and device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system or apparatus, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product (computer program product) is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a business server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for training a matching model, characterized in that, include: Obtain the frequency domain features of each group of audio samples in the sample set; Each group of sample audios includes the current original full audio, multiple modified audio segments corresponding to the current original full audio, and other original audios that are different from the current original full audio. The current original full audio in each group of sample audios is different. Each of the frequency domain features is input into the initial matching model to obtain the melody features of each of the sample audios; The model loss function of the matching model is calculated based on the first melody feature distance between each of the current original full audio and each of the modified audio segments, and the second melody feature distance between each of the current original full audio and the other original audio in the same group; wherein, the first melody feature distance is the distance between the melody feature of the modified audio segment and the melody feature of the original sub-segment of the corresponding length of the modified audio segment in the current original full audio, and the second melody feature distance is the distance between the melody feature of the other original audio within a preset length and the melody feature of the current original full audio within a preset length; The model parameters of the initial matching model are adjusted based on the model loss function until the convergence condition is met to obtain the target matching model. The convergence condition includes ensuring that the distances of each of the first melody features are less than or equal to a first threshold due to model parameter tuning, and that the distances of each of the second melody features are greater than or equal to a second threshold due to model parameter tuning, wherein the first threshold is less than the second threshold. The target matching model is used to output the target melody features corresponding to the audio segment to be identified, so that the original full audio to which the audio segment to be identified belongs can be identified through the target melody features.

2. The training method according to claim 1, characterized in that, The process of calculating the model loss function of the matching model includes: For each set of sample audio, calculate the first melodic feature distance between each modified audio segment and the corresponding original sub-segment, and the second melodic feature distance between each other original audio and the current original full audio within a preset length; the other original audio includes other original full audio or sub-segments of other original full audio. The distances of the first melody features and the distances of the second melody features that meet the preset conditions in the sample audio are summed to obtain the sample loss function of the sample audio. The loss function of each group of samples is used as the model loss function, or the loss function of each group of samples is calculated in a weighted manner to obtain the model loss function.

3. The training method according to claim 2, characterized in that, The summation calculation of the distances of the first melody features and the distances of the second melody features in the sample audio that meet the preset conditions includes: Filter out a preset number of modified audio segments whose first melody feature distance is greater than a preset distance; Based on the triplet loss function rule, the distances of the first melody features and each of the second melody features filtered out from the sample audio are calculated to obtain the sample loss function of the sample audio.

4. The training method according to claim 2, characterized in that, The summation calculation of the distances of each melody feature in the sample audio that meets the preset conditions includes: Filter out a preset number of modified audio segments whose first melody feature distance is greater than a preset distance; Based on the comparison results of text feature content and / or spectral peaks among the preset number of the modified audio segments, a weighting coefficient is assigned to each of the filtered first melody feature distances; The sample loss function of the sample audio is calculated by weighting and fusing the first melody feature distance based on the weighted coefficients and adding the second melody feature distance of the same group.

5. The training method according to claim 1, characterized in that, The acquisition of the frequency domain features of each group of audio samples in the sample set includes: According to the preset slice length and slice offset, the modified full audio, the current original full audio, and the other original audio are sliced ​​to obtain multiple modified audio segments, original sub-segments used to constitute the current original full audio, and at least one other sub-segment as the other original audio; wherein, the slice length and slice offset are determined according to the audio segment playback duration or lyrics arrangement rules; Frequency domain features are extracted from each group of original sub-segments, other original audio files, and modified audio segments.

6. The training method according to claim 1, characterized in that, The process of obtaining the target matching model until the convergence condition is met includes: If, after multiple rounds of model parameter tuning, the distance of the first melody feature in each round is less than or equal to the first threshold, and the distance of the second melody feature in each round is greater than or equal to the second threshold, then the model obtained from the latest parameter tuning is considered to be the target matching model.

7. An audio recognition method, characterized in that, include: The audio segment to be identified is input into the target matching model to obtain the target melody features corresponding to the audio segment to be identified; wherein, the target matching model is trained according to the training method described in any one of claims 1 to 6; Calculate the melody feature distance between the target melody feature and the melody features of each original audio file; The original full audio corresponding to the one with the smallest distance among the melodic feature distances is determined as the target full audio to which the audio segment to be identified belongs.

8. The audio recognition method according to claim 7, characterized in that, If the feature distance is a multi-type distance metric including cosine distance or Euclidean distance, then determining the original full audio corresponding to the smallest distance among the melodic feature distances as the target full audio to which the audio segment to be identified belongs includes: For each original audio source, the melody feature distance is weighted and calculated to obtain the distance fusion result for each original full audio source. The original full audio corresponding to the smallest result among the various distance fusion results is determined as the target full audio.

9. An electronic device, characterized in that, include: Central processing unit, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images