Song recognition method, apparatus, medium, and computing device

Abstract

Embodiments of the present application provide a song recognition method, device, medium and computing equipment, which converts the audio to be recognized into a phonetic symbol sequence to be recognized, then determines a candidate phonetic symbol sequence from the phonetic symbol sequence corresponding to each candidate song through phonetic symbol comparison, and further sorts the candidate phonetic symbol sequence in combination with phonetic symbol alignment scoring, so as to obtain a final song recognition result. The song recognition based on the method can effectively avoid recognition errors caused by homophonic words, and a large number of mis-matches are filtered through phonetic symbol alignment, so as to improve the accuracy of phonetic symbol retrieval and song recognition.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of song recognition, and more particularly, embodiments of the present application relate to a song recognition method, device, medium and computing device. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the application recited in the claims. The description herein does not constitute admission that the information provided herein is prior art merely because it is included herein.

[0003] With the popularity of digital music and the rapid development of Internet technology, song recognition technology is widely used in user life scenarios. At present, the traditional song recognition method mainly relies on the extraction and comparison of audio features, and identifies songs by matching the melody, rhythm, lyrics and other features of the audio segment. However, this method is easily affected by the accuracy of lyrics recognition when identifying audio segments containing lyrics, for example, for homophonic words or dialect, small language song recognition, the error rate of lyrics recognition will increase significantly, thereby reducing the accuracy of song recognition.

[0004] Therefore, how to further improve the accuracy of song recognition and reduce the cost of supporting new languages has become a problem to be solved in the current song recognition technology field. SUMMARY

[0005] In view of this, to solve the above problems, embodiments of the present application aim to provide a song recognition method, device, medium and computing device.

[0006] In a first aspect of the embodiments of the present application, a song recognition method is provided, the method comprising:

[0007] obtaining a to-be-recognized phonetic symbol sequence corresponding to a to-be-recognized audio;

[0008] performing phonetic symbol comparison on the to-be-recognized phonetic symbol sequence and a first phonetic symbol sequence of each candidate song to determine a candidate sub-sequence in which the phonetic symbols are the same and the sequence length meets the requirements in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence;

[0009] determining a candidate phonetic symbol sequence that is time-matched with the to-be-recognized phonetic symbol sequence from the first phonetic symbol sequence containing the candidate sub-sequence according to the position of the candidate sub-sequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence;

[0010] aligning the to-be-recognized audio with the phonetic symbols in the candidate phonetic symbol sequence to obtain a phonetic symbol alignment score of the to-be-recognized audio according to the candidate phonetic symbol sequence;

[0011] determining a song recognition result of the to-be-recognized audio according to the phonetic symbol alignment score corresponding to each candidate phonetic symbol sequence.

[0012] Optionally, the phonetic symbol sequence to be identified and the first phonetic symbol sequence of each candidate song are subjected to phonetic symbol comparison, comprising:

[0013] The phonetic symbol sequence to be identified is divided into a plurality of to-be-identified sub-sequences according to a set sequence length.

[0014] Each to-be-identified sub-sequence is matched with an index key of the inverted index; wherein the index key is a first phonetic symbol sequence; the first phonetic symbol sequence is each sub-sequence obtained by dividing the complete phonetic symbol sequence of each candidate song according to the set sequence length; and an index value corresponding to the index key is candidate song information containing the index key.

[0015] An index key identical in phonetic symbol to the to-be-identified sub-sequence is taken as the candidate sub-sequence.

[0016] Optionally, the phonetic symbol sequence to be identified is divided into a plurality of to-be-identified sub-sequences according to a set sequence length, comprising:

[0017] Starting from a starting phonetic symbol of the phonetic symbol sequence to be identified, a window is used to slide on the phonetic symbol sequence to be identified according to a set step length and an adjacent window overlap value, so as to divide the phonetic symbol sequence to be identified into a plurality of to-be-identified sub-sequences; wherein the length of the window is set as the set sequence length.

[0018] Optionally, the candidate phonetic symbol sequence that is time-matched with the phonetic symbol sequence to be identified is determined from the first phonetic symbol sequence containing the candidate sub-sequence, comprising:

[0019] A playback time speed scaling ratio of the candidate sub-sequence on the phonetic symbol sequence to be identified and the first phonetic symbol sequence is obtained.

[0020] According to the playback time speed scaling ratio and positions of the candidate sub-sequence in the phonetic symbol sequence to be identified and the first phonetic symbol sequence, a candidate phonetic symbol sequence that is time-matched with the phonetic symbol sequence to be identified is determined.

[0021] Optionally, the playback time speed scaling ratio of the candidate sub-sequence on the phonetic symbol sequence to be identified and the first phonetic symbol sequence is obtained, comprising:

[0022] A first playback time length is determined according to a starting time stamp and an ending time stamp of the candidate sub-sequence in the phonetic symbol sequence to be identified.

[0023] A second playback time length is determined according to a starting time stamp and an ending time stamp of the candidate sub-sequence in the first phonetic symbol sequence.

[0024] The playback time speed scaling ratio is determined according to the first playback time length and the second playback time length.

[0025] Optionally, the determining the candidate phonetic sequence time-matching the to-be-recognized phonetic sequence from the first phonetic sequence comprising candidate sub-sequences comprises:

[0026] obtaining a first time interval between a start time stamp of the to-be-recognized phonetic sequence and a start time stamp of a candidate sub-sequence in the to-be-recognized phonetic sequence;

[0027] obtaining a second time interval between an end time stamp of the to-be-recognized phonetic sequence and an end time stamp of a candidate sub-sequence in the to-be-recognized phonetic sequence;

[0028] determining a time stamp range of the candidate phonetic sequence in the first phonetic sequence according to the start time stamp and the end time stamp of the candidate sub-sequence in the first phonetic sequence, the first time interval, the second time interval, and the playback speed scaling ratio;

[0029] determining the candidate phonetic sequence in the first phonetic sequence corresponding to the time stamp range as the candidate phonetic sequence.

[0030] Optionally, the determining the time stamp range of the candidate phonetic sequence in the first phonetic sequence comprises:

[0031] determining a start time stamp of the candidate phonetic sequence in the first phonetic sequence according to the start time stamp of the candidate sub-sequence in the first phonetic sequence, the playback speed scaling ratio, and the first time interval;

[0032] determining an end time stamp of the candidate phonetic sequence in the first phonetic sequence according to the end time stamp of the candidate sub-sequence in the first phonetic sequence, the playback speed scaling ratio, and the second time interval.

[0033] Optionally, the aligning the to-be-recognized audio with the phonetic symbols in the candidate phonetic sequence comprises:

[0034] generating, for each frame of the to-be-recognized audio, a probability of each international phonetic symbol belonging to each frame by using a pre-trained phonetic symbol recognition network;

[0035] constructing a decoding sequence according to the candidate phonetic sequence; a node in the decoding sequence corresponds to a phonetic symbol in the candidate phonetic sequence;

[0036] decoding, according to the probability of each international phonetic symbol belonging to each frame, by using the decoding sequence, to obtain a triple sequence of phonetic symbols in the to-be-recognized audio aligning with the candidate phonetic sequence, the triple comprising a phonetic symbol, a start time of the phonetic symbol in the to-be-recognized audio, and an end time of the phonetic symbol in the to-be-recognized audio.

[0037] Optionally, the decoding sequence adopts a serial structure, and includes a plurality of sequence nodes; in a case where phonetic symbols corresponding to a plurality of continuous sequence nodes belong to a complete lyric,

[0038] The adjacent front node of a start node and the adjacent rear node of an end node in the plurality of continuous sequence nodes are nodes representing silence; the adjacent nodes of the plurality of continuous sequence nodes include a node representing silence and being skipable.

[0039] Optionally, the phonetic symbol alignment score of the candidate phonetic symbol sequence is obtained by:

[0040] For each triple, a probability mean of each international phonetic symbol in a triple time range is obtained according to a probability of each frame in the to-be-recognized audio belonging to each international phonetic symbol; the triple time range is determined according to a start time and an end time of a phonetic symbol in the triple in the to-be-recognized audio;

[0041] A posterior alignment probability of the triple is calculated according to the probability mean of the phonetic symbol in the triple and a sum of the probability mean of each international phonetic symbol in the triple time range;

[0042] A phonetic symbol alignment score corresponding to the candidate phonetic symbol sequence is obtained according to the posterior alignment probability of each triple.

[0043] Optionally, the to-be-recognized phonetic symbol sequence corresponding to the to-be-recognized audio is obtained by:

[0044] The to-be-recognized phonetic symbol sequence and time stamp information of each phonetic symbol in the to-be-recognized audio are obtained by performing phonetic symbol recognition on the to-be-recognized audio by using a pre-trained phonetic symbol recognition network;

[0045] The method further includes:

[0046] In a case where the length of the to-be-recognized phonetic symbol sequence is greater than a set threshold, the to-be-recognized phonetic symbol sequence is compared with the first phonetic symbol sequence for phonetic symbol alignment;

[0047] In a case where the length of the to-be-recognized phonetic symbol sequence is less than a set threshold, the song recognition is terminated.

[0048] In a second aspect of the embodiments of the present application, a song recognition device is provided, and the device includes:

[0049] A to-be-recognized phonetic symbol sequence obtaining module is configured to obtain a to-be-recognized phonetic symbol sequence corresponding to a to-be-recognized audio;

[0050] The phonetic symbol comparison module is configured to compare the to-be-recognized phonetic symbol sequence with the first phonetic symbol sequence of each candidate song, and determine a candidate sub-sequence that is identical in phonetic symbol and meets a sequence length requirement.

[0051] The candidate phonetic symbol sequence acquisition module is configured to determine, from the first phonetic symbol sequence containing the candidate sub-sequence, a candidate phonetic symbol sequence that is time-matched with the to-be-recognized phonetic symbol sequence according to a position of the candidate sub-sequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence.

[0052] The phonetic symbol alignment module is configured to align the to-be-recognized audio with the phonetic symbols in the candidate phonetic symbol sequence, and obtain a phonetic symbol alignment score of pronunciation of the to-be-recognized audio according to the candidate phonetic symbol sequence.

[0053] The song recognition result generation module is configured to determine a song recognition result of the to-be-recognized audio according to the phonetic symbol alignment score corresponding to each candidate phonetic symbol sequence.

[0054] Optionally, the phonetic symbol comparison module is specifically configured to:

[0055] The division module is configured to divide the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized sub-sequences according to a set sequence length.

[0056] The index key matching module is configured to match each to-be-recognized sub-sequence with an index key of the inverted index; wherein the index key is a first phonetic symbol sequence; the first phonetic symbol sequence refers to each sub-sequence obtained by dividing a complete phonetic symbol sequence of each candidate song according to the set sequence length; and an index value corresponding to the index key is candidate song information containing the index key.

[0057] The candidate sub-sequence determination module is configured to determine, as the candidate sub-sequence, an index key that is identical in phonetic symbol to the to-be-recognized sub-sequence.

[0058] Optionally, the division module is specifically configured to:

[0059] The division module is configured to divide the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized sub-sequences by using a window to slide on the to-be-recognized phonetic symbol sequence according to a set step length and a neighboring window overlap value, starting from a starting phonetic symbol of the to-be-recognized phonetic symbol sequence; wherein a length of the window is set as the set sequence length.

[0060] Optionally, the candidate phonetic symbol sequence acquisition module is specifically configured to:

[0061] The scaling ratio determination module is configured to acquire a playback time scaling ratio of the candidate sub-sequence on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence.

[0062] determine the candidate phonetic symbol sequence time-matching the to-be-recognized phonetic symbol sequence according to the playback time scaling ratio and the position of the candidate sub-sequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence.

[0063] Optionally, the scaling ratio determining module is specifically configured to:

[0064] determine a first playback duration according to the start timestamp and the end timestamp of the candidate sub-sequence in the to-be-recognized phonetic symbol sequence;

[0065] determine a second playback duration according to the start timestamp and the end timestamp of the candidate sub-sequence in the first phonetic symbol sequence;

[0066] determine the playback time scaling ratio according to the first playback duration and the second playback duration.

[0067] Optionally, the candidate phonetic symbol sequence determining module is specifically configured to:

[0068] obtain a first time interval between a start timestamp of the to-be-recognized phonetic symbol sequence and a start timestamp of a candidate sub-sequence in the to-be-recognized phonetic symbol sequence;

[0069] obtain a second time interval between an end timestamp of the to-be-recognized phonetic symbol sequence and an end timestamp of the candidate sub-sequence in the to-be-recognized phonetic symbol sequence;

[0070] determine a timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence by using the first time interval, the second time interval, and the playback time scaling ratio according to the start timestamp and the end timestamp of the candidate sub-sequence in the first phonetic symbol sequence;

[0071] determine the candidate phonetic symbol sequence as a phonetic symbol sequence corresponding to the timestamp range in the first phonetic symbol sequence.

[0072] Optionally, the candidate phonetic symbol sequence determining module, when used to determine the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence, comprises:

[0073] determine a start timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence according to the start timestamp of the candidate sub-sequence in the first phonetic symbol sequence, the playback time scaling ratio, and the first time interval;

[0074] determine an end timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence according to the end timestamp of the candidate sub-sequence in the first phonetic symbol sequence, the playback time scaling ratio, and the second time interval.

[0075] Optionally, the phonetic symbol alignment module is specifically configured to:

[0076] For each frame of the audio to be recognized, a pre-trained phoneme recognition network is used to generate a probability that each frame belongs to each international phoneme;

[0077] According to the candidate phoneme sequence, a decoding sequence is constructed; nodes in the decoding sequence correspond to phonemes in the candidate phoneme sequence;

[0078] According to the probability that each frame belongs to each international phoneme, the decoding sequence is decoded to obtain a sequence of triples in which phonemes align with the candidate phoneme sequence in the audio to be recognized; the triple includes a phoneme symbol, a start time and an end time of the phoneme symbol in the audio to be recognized.

[0079] Optionally, the decoding sequence adopts a serial structure and includes a plurality of sequence nodes; in a case where phonemes corresponding to a plurality of continuous sequence nodes belong to a complete lyric,

[0080] A node adjacent to a front node of a start node and a node adjacent to a rear node of an end node in the plurality of continuous sequence nodes are nodes representing silence; nodes between adjacent nodes of the plurality of continuous sequence nodes include a node representing silence and being skipable.

[0081] Optionally, the phoneme alignment module is specifically configured to:

[0082] For each triple, according to the probability that each frame of the audio to be recognized belongs to each international phoneme, a probability mean of each international phoneme in a triple time range is obtained; the triple time range is determined according to a start time and an end time of a phoneme symbol in the triple in the audio to be recognized;

[0083] According to the probability mean of the phoneme symbol in the triple and a sum of the probability means of each international phoneme in the triple time range, a posterior alignment probability of the triple is calculated;

[0084] According to the posterior alignment probability of each triple, a phoneme alignment score corresponding to the candidate phoneme sequence is obtained.

[0085] Optionally, the audio to be recognized sequence obtaining module is specifically configured to:

[0086] A pre-trained phoneme recognition network is used to perform phoneme recognition on the audio to be recognized to obtain a corresponding audio to be recognized sequence and time stamp information of each phoneme in the audio to be recognized;

[0087] The apparatus further includes:

[0088] In a case that the length of the phonetic symbol sequence to be recognized is greater than a set threshold, the phonetic symbol sequence to be recognized is compared with the first phonetic symbol sequence; in a case that the length of the phonetic symbol sequence to be recognized is less than a set threshold, the song recognition is terminated.

[0089] In a third aspect of the embodiments of the present application, a readable storage medium is provided, and the readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the song recognition method.

[0090] In a fourth aspect of the embodiments of the present application, a computing device is provided, and the computing device comprises a processor and a memory, the memory is configured to store a computer program, and the processor is configured to execute the song recognition method by calling the computer program.

[0091] The song recognition method provided by the embodiments of the present application introduces the international phonetic alphabet, a cross-language standardized phonetic representation, converts the audio to be recognized into a phonetic symbol sequence to be recognized, and then determines a candidate phonetic symbol sequence from phonetic symbol sequences corresponding to each candidate song through phonetic symbol comparison, and further sorts the candidate phonetic symbol sequence in combination with phonetic symbol alignment scoring, so as to obtain a final song recognition result. The song recognition based on the method positions the song through phonetic symbol matching, can effectively avoid recognition errors caused by homophonic words, filters out a large number of mismatching through phonetic symbol alignment, and thus improves the accuracy of phonetic symbol retrieval and song recognition. BRIEF DESCRIPTION OF DRAWINGS

[0092] The above and other objects, features and advantages of the exemplary embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0093] FIG. 1 A step flowchart of a song recognition method according to an embodiment of the present application is schematically shown;

[0094] FIG. 2A A phonetic symbol comparison flowchart based on an inverted index according to an embodiment of the present application is schematically shown;

[0095] FIG. 2B A process example diagram of dividing a phonetic symbol sequence to be recognized into a plurality of phonetic symbol sub-sequences through a sliding window according to an embodiment of the present application is schematically shown;

[0096] FIG. 3A An example diagram flowchart of determining a play-time speed scaling ratio according to an embodiment of the present application is schematically shown;

[0097] FIG. 3BA flowchart of a step of determining a candidate phonetic sequence according to an embodiment of the application is schematically illustrated;

[0098] FIG. 3C An example diagram of determining a timestamp range based on a phonetic sequence according to an embodiment of the application is schematically illustrated;

[0099] FIG. 4A A flowchart of a phonetic alignment step according to an embodiment of the application is schematically illustrated;

[0100] FIG. 4B A general flow of training a phonetic recognition network according to an embodiment of the application is schematically illustrated;

[0101] FIG. 4C A diagram of a decoding sequence according to an embodiment of the application is schematically illustrated;

[0102] FIG. 4D Another diagram of a decoding sequence according to an embodiment of the application is schematically illustrated;

[0103] FIG. 5 A block diagram of a song recognition apparatus according to an embodiment of the application is schematically illustrated;

[0104] FIG. 6 A diagram of a medium according to an embodiment of the application is schematically illustrated;

[0105] FIG. 7 A diagram of a computing device according to an embodiment of the application is schematically illustrated.

[0106] In the drawings, same or similar reference numerals refer to same or similar parts throughout the several views. DETAILED DESCRIPTION

[0107] The principles and spirits of the present application will be described below with reference to several exemplary embodiments. It should be understood, however, that the embodiments are given solely for the purpose of illustration and are not meant to limit the scope of the present application in any way. On the contrary, the embodiments are intended to convey the scope of the present application to those skilled in the art. Therefore, the present application should be understood to include any variations of these embodiments.

[0108] Those skilled in the art will understand that the embodiments of the present application can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present application can have the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware.

[0109] According to an embodiment of the present application, a song recognition method, device, medium and computing equipment are provided.

[0110] In this document, it should be understood that the number of any elements in the figures is intended to be illustrative only and not limiting; and any naming convention is merely intended to differentiate between elements and not intended to be limiting.

[0111] The principles and spirit of the present application will be explained in detail below with reference to several representative embodiments of the present application.

[0112] With the popularity of digital music and the rapid development of Internet technology, song recognition technology has gradually become a popular research field. Whether in song search in music playing software, humming recognition, or in the background music recognition scene in public places, song recognition technology plays a crucial role. However, although existing song recognition technology has made significant progress in some aspects, it still faces many challenges.

[0113] The traditional song recognition method mainly relies on the extraction and comparison of audio features, and identifies songs by matching the melody, rhythm and other features of the audio segment and song lyrics recognition retrieval. However, this method is easily affected by the accuracy of song lyrics recognition, and when encountering homophonic words or dialect songs, the error rate of song lyrics recognition will increase significantly, resulting in a decrease in the accuracy of song recognition. For the recognition of new language songs, in order to increase the speech recognition ability of new language songs, it is usually necessary to prepare a large amount of audio data of new language songs for model training, which not only requires a large amount of time and resource investment, but also may face the difficulties of data acquisition and processing, and the cost of supporting new language song recognition is high.

[0114] Therefore, how to further improve the accuracy of song recognition, reduce the cost of supporting new languages, and simplify the maintenance work of song library has become a problem to be solved in the current song recognition technology field.

[0115] Therefore, the present application proposes a song recognition method based on phonetic symbol matching, which locates possible matching candidate song segments from candidate songs through international phonetic symbol comparison, and filters mis-matches between candidate song segments and audio to be recognized through phonetic symbol alignment, and finally obtains a song recognition result according to the phonetic symbol alignment score, which can effectively avoid recognition errors caused by homophonic words, thereby improving the accuracy of phonetic symbol retrieval and song recognition. The present application bypasses the language restriction, models from the phonetic phoneme level, uses international phonetic symbols as the training target, and combines pronunciation evaluation technology to improve the recognition accuracy through phonetic symbol alignment scoring, forming a song recognition scheme based on audio phonetic symbol recognition and retrieval technology. This scheme has the characteristics of high universality and song language independence, and is suitable for various language song recognition scenarios, and can be quickly extended to small language or dialect song recognition.

[0116] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail.

[0117] Exemplary Method

[0118] The song recognition method according to the exemplary embodiments of the present application will be described below with reference to FIGS. 1-7

[0119] It should be noted that the above application scenarios are only shown for the purpose of facilitating the understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0120] Reference FIG. 1 A step flowchart of a song recognition method according to an embodiment of the present application is schematically shown, and the phonetic symbols in each phonetic symbol sequence involved in the song recognition method provided in the present embodiment are all International Phonetic Alphabet (IPA).

[0121] The method can at least include the following steps:

[0122] S101, obtaining a to-be-recognized phonetic symbol sequence corresponding to to-be-recognized audio;

[0123] The to-be-recognized audio represents audio data that a user hopes to be able to recognize and determine the song name or lyrics content thereof, which can include but is not limited to audio recorded by the user through a microphone or loaded from a file, and can contain singing, humming, playing of musical instruments, etc.

[0124] The to-be-recognized phonetic symbol sequence is an ordered result obtained by converting the sound in the to-be-recognized audio into phonetic symbols, and the phonetic symbols are used to represent the pronunciation features in the audio. The phonetic symbols in each phonetic symbol sequence involved in the song recognition method provided in the present embodiment are all International Phonetic Alphabet (IPA), which is a phonetic notation designed by the International Phonetic Association, and is used to accurately record and describe the pronunciation of human language, and contains various phoneme symbols such as vowels, consonants, diphthongs, and semivowels, covering the phonetic features of various languages.

[0125] As for obtaining the to-be-recognized phonetic symbol sequence corresponding to the to-be-recognized audio, a vocal tract model and speech knowledge can be used, i.e., the frequency domain or time domain characteristics of the speech signal are compared with known speech primitives (i.e., International Phonetic Alphabet), so as to recognize different speech. In a specific implementation, the speech signal of the to-be-recognized audio can be segmented and labeled, and then the similar phonetic symbol labels are given according to the acoustic characteristics of each segment, and a speech primitive grid is obtained according to the phonetic symbol label sequence, combined with the grammatical semantic information of the dictionary and the sentence, and finally the to-be-recognized phonetic symbol sequence is determined. ​

[0126] Alternatively, the to-be-recognized audio can also be converted into a phonetic symbol sequence by using a neural network or a machine learning method, that is, the to-be-recognized audio can be subjected to phonetic symbol recognition by using a pre-trained phonetic symbol recognition network to obtain a corresponding to-be-recognized phonetic symbol sequence and time stamp information of each phonetic symbol in the to-be-recognized audio. The training sample of the phonetic symbol recognition network includes a large amount of audio data and a corresponding phonetic symbol sequence label. The phonetic symbol sequence label corresponding to each audio data can be manually labeled by a phonetics expert or a trained personnel, and represents the accurate phonetic symbol of the pronunciation in the audio data.

[0127] S102, performing phonetic symbol comparison between the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence of each candidate song to determine a candidate sub-sequence that is identical in phonetic symbol and meets a sequence length requirement.

[0128] Regarding the candidate song, in the song recognition scenario, all songs in the song library are regarded as candidate songs, and these candidate songs are pre-stored music works with known information such as song name, singer, lyrics, etc.

[0129] The first phonetic symbol sequence represents the phonetic symbol sequence corresponding to the lyrics of the candidate song, and can be obtained by converting the lyrics of the candidate song word by word into corresponding phonetic symbols, representing the pronunciation of the song lyrics. The first phonetic symbol sequence can include two forms of representation: the first phonetic symbol sequence can represent the complete phonetic symbol sequence corresponding to the candidate song, that is, the lyrics of the entire song are converted into a continuous phonetic symbol sequence; or the first phonetic symbol sequence can also include a plurality of first sub-sequences divided from the complete phonetic symbol sequence according to a set sequence length, each first sub-sequence belongs to a segment of the complete phonetic symbol sequence, and collectively constitutes the phonetic symbol representation of the entire candidate song.

[0130] Phonetic symbol comparison refers to comparing the phonetic symbol symbols contained in two phonetic symbol sequences one by one to find the same phonetic symbol part between them. In this embodiment, phonetic symbol comparison is used to find a candidate sub-sequence that exists in both the first phonetic symbol sequence and the to-be-recognized phonetic symbol sequence and meets a sequence length requirement. The sequence length can be used to represent the number of phonetic symbols contained in the candidate sub-sequence, or based on the case that one phonetic symbol includes a plurality of characters, the sequence length can also be used to represent the number of single characters contained in the candidate sub-sequence, which is not limited in this application. The sequence length meeting the requirement is that the sequence length is greater than or equal to a pre-set length threshold, that is, the number of international phonetic symbols / single characters included in the candidate sub-sequence is greater than or equal to the pre-set length threshold. For each candidate song, if the candidate sub-sequence exists in the candidate song, the candidate song is taken as a candidate recognition result, such as being added to the candidate recognition result set.

[0131] As for obtaining the candidate sub-sequence through phonetic symbol comparison, any applicable method can be used, such as dynamic programming algorithm (e.g., longest common sub-sequence algorithm), string matching algorithm (e.g., brute-force matching or KMP algorithm), hash method, and machine learning algorithm. In order to improve the efficiency of phonetic symbol comparison, the complete phonetic symbol sequence corresponding to each candidate song can be divided into multiple first sub-sequences according to a set sequence length. Based on the multiple first sub-sequences corresponding to each candidate song, an inverted index is constructed, taking the first sub-sequence as the index key and the candidate song containing the first sub-sequence as the index value, so as to facilitate quick search for the candidate sub-sequence. For specific implementation process, please refer to the related example description below.

[0132] For example, assuming that the phonetic symbol sequence to be recognized is "a b c d e f g h i j k l m n o p q r s t u v w x y z", the first phonetic symbol sequence of a certain candidate song A is "a b c d e f g h i j k l m n o p q r s t u v w x y z", the sub-sequence that exists in both the phonetic symbol sequence to be recognized and the first phonetic symbol sequence of A is "a b c d e f g h i j k l m n o p q r s t u v w x y z" determined through phonetic symbol comparison, taking the number of phonetic symbols contained in the sub-sequence as an example, the sequence length of the sub-sequence is 16, which meets the length requirement, so the sub-sequence is taken as the candidate sub-sequence.

[0133] S103, determining the candidate phonetic symbol sequence that matches the phonetic symbol sequence to be recognized in time from the first phonetic symbol sequence containing the candidate sub-sequence according to the position of the candidate sub-sequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence;

[0134] Time matching is used to represent that the candidate phonetic symbol sequence selected from the first phonetic symbol sequence is synchronous or close to synchronous with the phonetic symbol sequence to be recognized in pronunciation time. After determining the candidate sub-sequence, the time characteristics of the candidate sub-sequence in the first international phonetic symbol sequence and the phonetic symbol sequence to be recognized can be compared, such as by calculating the duration, pronunciation rate, or other related time parameters of the candidate sub-sequence in the two different sequences.

[0135] According to the time characteristics, the timestamp or frame number of the candidate sub-sequence in the phonetic symbol sequence to be recognized can be aligned with the time information of the candidate sub-sequence in the first phonetic symbol sequence, and then according to the position of the candidate sub-sequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence, the position of the candidate sub-sequence in the first phonetic symbol sequence is taken as the starting point to expand forward and backward, to determine the candidate phonetic symbol sequence that matches the playback duration of the phonetic symbol sequence to be recognized.

[0136] S104, aligning the phonetic symbol sequence to be recognized with the phonetic symbol sequence to be recognized, to obtain a phonetic symbol alignment score of the phonetic symbol sequence to be recognized according to the candidate phonetic symbol sequence;

[0137] Phonetic alignment refers to the process of matching the phonetic signal in the audio to be recognized with the phonemes in the candidate phonetic sequence, which involves phonetic recognition techniques such as dynamic time warping (DTW), hidden Markov models (HMM), or deep learning models such as recurrent neural networks (RNN), long short-term memory networks (LSTM), and Transformers, to find the best match between the audio signal and the phonetic sequence.

[0138] Phonetic alignment score is a quantitative indicator that measures the quality of alignment between the audio to be recognized and the candidate phonetic sequence. The phonetic alignment score is calculated based on the degree of matching, error rate, or other relevant indicators during the alignment process, reflecting the degree of pronunciation matching between the audio signal and the candidate phonetic sequence. The phonetic alignment score is positively correlated with the quality of alignment, i.e., the higher the phonetic alignment score, the higher the quality of alignment, i.e., the higher the degree of matching between the audio to be recognized and the candidate phonetic sequence.

[0139] Regarding phonetic alignment, the audio frames of the audio to be recognized can be matched with the phonemes in the candidate phonetic sequence through alignment algorithms such as dynamic time warping (DTW), hidden Markov models (HMM), and deep learning models. The goal of phonetic alignment is to find an optimal matching method that minimizes the overall difference between the audio frames of the audio to be recognized and the candidate phonetic sequence. During the alignment process, a time range is determined for each phoneme, indicating the start and end times of the phoneme in the audio to be recognized. This time range is derived from the optimal matching result calculated by the alignment algorithm.

[0140] S105, according to the phonetic alignment scores corresponding to each candidate phonetic sequence, determines the song recognition result of the audio to be recognized.

[0141] The song recognition result is the most likely song name or identifier determined based on the comparison results of the audio to be recognized and the candidate phonetic sequence. This result is the final output of song recognition, which tells the user the song to which the audio to be recognized belongs.

[0142] In this step, the candidate phonetic sequences can be ranked according to the phonetic alignment scores, i.e., all candidate phonetic sequences are ranked from high to low according to their phonetic alignment scores, and the higher the score, the higher the degree of matching between the candidate phonetic sequence and the audio to be recognized. The highest scoring candidate phonetic sequence or the top N candidate phonetic sequences are selected, and the song information of the selected candidate phonetic sequence is obtained to generate the song recognition result. For example, the song name and the start and end timestamps of the candidate phonetic sequence in the candidate song to which it belongs are obtained, a song recognition result interface is generated, and the corresponding lyrics of the candidate phonetic sequence are played.

[0143] Alternatively, in order to ensure the accuracy of song recognition, after the candidate phonetic symbol sequences are sorted according to the phonetic symbol alignment score, one or more candidate phonetic symbol sequences with a score ranking of N and greater than a score threshold can be selected, and the corresponding candidate song information is obtained to generate a song recognition result.

[0144] In the embodiments of the present disclosure, the international phonetic symbols, a standardized phonetic symbol form across languages, are used to achieve accurate recognition of song audio. The phonetic symbol sequence of the audio to be recognized is obtained first, and then compared with the first phonetic symbol sequence of the candidate song to determine the matching candidate subsequence. Then, the time-matched candidate phonetic symbol sequence is determined from the first phonetic symbol sequence in which the candidate subsequence is located. The phonetic symbol alignment further filters out a large number of mis-matches, improves the accuracy of phonetic symbol retrieval and final recognition, and sorts the candidate phonetic symbol sequences through phonetic symbol alignment scoring to obtain the final song recognition result. The song recognition process does not directly rely on a language model for lyric recognition, but locates the song through phonetic symbol matching, thereby effectively avoiding recognition errors caused by homophonic words and significantly improving the accuracy of song recognition. Moreover, the method is based on the international phonetic symbols for modeling. When expanding new language songs, only the corresponding relationship between the new language and the international phonetic symbols needs to be established, without the need to prepare a large amount of audio data of the new language to retrain, so that the method can quickly adapt to and support the recognition of new language songs, reducing the cost and time of new language support.

[0145] In some embodiments, in order to further optimize the accuracy and efficiency of song matching in audio recognition, especially when dealing with large-scale music libraries, the traditional linear scanning comparison method is relatively low in efficiency. Therefore, the present embodiment proposes a phonetic symbol comparison method based on inverted index, which can accelerate the comparison process while ensuring the accuracy of the comparison result, thereby improving the comparison efficiency. Referring to FIG. 2A An example of a phonetic symbol comparison flowchart based on inverted index, which can be implemented by the following steps to compare the phonetic symbol sequence to be recognized with the first phonetic symbol sequence of each candidate song:

[0146] S201, divide the phonetic symbol sequence to be recognized into a plurality of to-be-recognized subsequences according to a set sequence length;

[0147] The set sequence length can be used to represent the number of phonetic symbols contained in each subsequence to be identified. Alternatively, based on the case where a phonetic symbol includes multiple characters, the sequence length can also be used to represent the number of single characters contained in each subsequence to be identified. The value and meaning of the set sequence length can be determined according to the needs of the actual application and the expected comparison accuracy. Since this invention performs song recognition at the phoneme level, the description of the set sequence length is illustrated using the number of phonetic symbols contained in the sequence as an example. It is understood that this description is merely illustrative and does not constitute a limitation on the scope of this application.

[0148] Dividing a sequence of phonetic symbols into multiple subsequences can be achieved using a sliding window. The sliding window is a virtual frame or tool that moves progressively across the sequence to extract multiple subsequences. The window starts at the first phonetic symbol of the sequence and moves forward according to a set step size. The step size determines the distance the window moves each time. When the step size equals the set sequence length, the subsequences will not overlap; when the step size is less than the set sequence length, adjacent subsequences will partially overlap.

[0149] In this embodiment, starting from the initial phonetic symbol of the phonetic symbol sequence to be identified, a sliding window is used to slide on the phonetic symbol sequence to be identified according to a set step size and the overlap value of adjacent windows, so as to divide the phonetic symbol sequence to be identified into multiple sub-sequences to be identified; wherein, the length of the sliding window is set to the set sequence length.

[0150] The adjacent window overlap value is an optional parameter that specifies the number of overlapping phonetic symbols between adjacent subsequences. When the sliding step is less than the set sequence length, the adjacent window overlap value determines the degree of phonetic symbol overlap.

[0151] See FIG. 2B An exemplary diagram illustrates a process of dividing a sequence of phonetic symbols to be recognized into multiple sub-sequences using a sliding window, with the sequence of phonetic symbols to be recognized being " For example, a sequence containing 19 phonetic symbols is divided into five subsequences with phonetic symbol positions [1-15], [2-16], [3-17], [4-18], and [5-19].

[0152] S202, match each to-be-recognized sub-sequence with an index key of the inverted index; wherein the index key is each sub-sequence contained in the first phonetic sequence, which is obtained by dividing the complete phonetic sequence of each candidate song according to the set sequence length; and the index value corresponding to the index key is candidate song information containing the index key;

[0153] The inverted index is an index structure for full-text search, which records the occurrence position of a word (or phrase) in a document, and is suitable for quickly finding all documents containing a specific word or phrase. Unlike the traditional forward index (which records which words a document contains), the inverted index starts from a word and records all documents containing the word and their position information. In the present embodiment, the inverted index is used to record all candidate songs containing a phonetic sub-sequence and the position information of the phonetic sub-sequence in each candidate song.

[0154] In the inverted index, the index key refers to the word or phrase used for retrieval. In the present embodiment, for all candidate songs, after converting the lyrics of each candidate song into the International Phonetic Alphabet to obtain a complete phonetic sequence, the complete phonetic sequence is divided into multiple sub-sequences according to the set sequence length described in step S201, and after removing duplicates from the multiple sub-sequences of all candidate songs, the index keys are formed respectively. Among them, the way of dividing the complete phonetic sequence of a candidate song into each sub-sequence is the same as the way of dividing the to-be-recognized phonetic sequence into multiple to-be-recognized sub-sequences described in the foregoing step S201.

[0155] For example, candidate song A1 is divided into M1 sub-sequences according to the set sequence length, candidate song A2 is divided into M2 sub-sequences, …, and candidate song An is divided into Mn sub-sequences, then (M1+M2+…+Mn) sub-sequences are removed after duplicates are removed, and each sub-sequence forms an index key. Assuming that there are (M1+M2+…+Mn-100) sub-sequences after removing duplicates, then there are (M1+M2+…+Mn-100) index keys.

[0156] The index value corresponding to the index key refers to a list or set of a group of documents containing the index key, and in the present embodiment, the index value corresponding to the index key refers to candidate song information such as song ID containing the sub-sequence represented by the index key, and position information such as start timestamp and end timestamp of the sub-sequence in the candidate song.

[0157] Continuing with the aforementioned candidate song A1 as an example, M1 index keys are formed by the M1 subsequences divided from the candidate song A1, and the index value corresponding to each index key includes the candidate song A1 and the position of the subsequence represented by the index key in the candidate song A1. For example, the first subsequence divided from the candidate song A1 forms index key 1, and the value corresponding to the index key 1 is “candidate song A1: [00:07, 00:33]”. The second subsequence forms index key 2, and the value corresponding to the index key 2 is “candidate song A1: [00:40, 01:01]”. For the case of successful subsequence deduplication, it indicates that there are multiple candidate songs containing the subsequence, so the index value corresponding to the index key formed by the subsequence includes multiple candidate songs and the position information of the subsequence, for example, the value corresponding to a certain index key n1 is “candidate song A3: [01:30, 01:55], candidate song A9: [02:15, 02:30]”.

[0158] In the process of matching each to-be-recognized subsequence with the index keys of the inverted index, since the to-be-recognized phonetic sequence is divided into multiple independent subsequences for matching, in order to improve the matching efficiency, the matching of the to-be-recognized subsequences with the index keys can be processed in parallel.

[0159] Regarding the inverted index, a phonetic sequence library can be established in advance, including the data structure of the inverted index, that is, according to the set sequence length, the complete phonetic sequence corresponding to each candidate song can be divided into multiple first subsequences, and the deduplication of the first subsequences is performed.

[0160] The first subsequences are taken as the keys of the inverted index, and the song information containing the first subsequences is taken as the values corresponding to the keys, to construct the phonetic sequence library.

[0161] S203, taking the index key with the same to-be-recognized subsequence phonetic sequence as the candidate subsequence.

[0162] The embodiment is used to perform phonetic sequence comparison between the to-be-recognized phonetic sequence and the first phonetic sequence of each candidate song, to determine the candidate subsequence with the same phonetic sequence and the sequence length meeting the requirements in the to-be-recognized phonetic sequence and the first phonetic sequence. By dividing the to-be-recognized phonetic sequence into multiple to-be-recognized subsequences and comparing them with each index key in the inverted index, if the phonetic sequence of any to-be-recognized subsequence is the same as that of any index key, the subsequence represented by the index key is the candidate subsequence with the same phonetic sequence and the sequence length meeting the requirements in the to-be-recognized phonetic sequence and the first phonetic sequence.

[0163] In the embodiments of the present disclosure, by dividing the phonetic symbol sequence to be recognized into a plurality of shorter subsequences and matching with the index keys in the inverted index, the characteristics of local matching are utilized, the number of phonetic symbols that need to be compared can be significantly reduced, the matching process is more efficient, the candidate subsequence and the plurality of candidate songs containing the candidate subsequence are quickly determined as the candidate recognition result, and the song recognition efficiency is improved.

[0164] In some embodiments, after determining that the candidate subsequence has the same phonetic symbols as the first phonetic symbol sequence and meets the sequence length requirement, based on the fact that the candidate subsequence belongs to a segment in the phonetic symbol sequence to be recognized, based on local segment matching, and through a local-to-global approach, a candidate phonetic symbol sequence that is time-matched with the entire phonetic symbol sequence to be recognized can be located from the first phonetic symbol sequence containing the candidate subsequence, which can be achieved through steps a1-a2 as follows:

[0165] a1, obtaining a playback time scaling ratio of the candidate subsequence on the phonetic symbol sequence to be recognized and the first phonetic symbol sequence;

[0166] The playback time scaling ratio is used to reflect the difference in singing speed of the candidate subsequence in the two different sequences. Based on the fact that the singing speed of the candidate subsequence with the same phonetic symbols under different audio sources is different, for example, the timestamp information of the candidate subsequence in the phonetic symbol sequence to be recognized is [01:45-02:00], i.e., 15 seconds of singing, and the timestamp information of the first phonetic symbol sequence corresponding to the candidate song is [01:27-01:57], i.e., 30 seconds of singing, in the case of local segment matching, in order to ensure the accuracy and reliability of audio recognition, the playback time scaling ratio of the candidate subsequence on the phonetic symbol sequence to be recognized and the first phonetic symbol sequence can be obtained based on the timestamp information of the local segment, i.e., the candidate subsequence, in the two sequences, so as to realize that the finally determined candidate phonetic symbol sequence can be matched with the phonetic symbol sequence to be recognized.

[0167] Regarding the calculation of the playback time scaling ratio, the playback time of the candidate subsequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence containing the candidate subsequence is determined. That is, the first playback time can be determined according to the start timestamp and the end timestamp of the candidate subsequence in the phonetic symbol sequence to be recognized, and the second playback time can be determined according to the start timestamp and the end timestamp of the candidate subsequence in the first phonetic symbol sequence; and the playback time scaling ratio is determined according to the first playback time and the second playback time.

[0168] For example, referring to FIG. 3AAn example diagram of determining a playback speed scaling ratio is shown. The to-be-recognized audio corresponds to a to-be-recognized phonetic symbol sequence containing m phonetic symbols, denoted as R1...Rm, and the pronunciation time stamp of each phonetic symbol in the to-be-recognized audio is denoted as T1...Tm. When a candidate sub-sequence R_h1...R_hn in the to-be-recognized audio is matched to a candidate sub-sequence in the candidate song S, and the candidate sub-sequence R_h1...R_hn is completely identical to the phonetic symbol sequence in the candidate song S, the time stamp of the candidate sub-sequence R_h1...R_hn in the to-be-recognized audio is denoted as T_r1...T_rn, and the pronunciation time stamp in the candidate song S is denoted as T_s1...T_sn, the playback speed scaling ratio can be represented as: scale=(the playback time length in the candidate song S) / (the playback time length in the to-be-recognized audio)=(T_sn-T_s1) / (T_rn-T_r1).

[0169] Suppose the candidate sub-sequence R_h1...R_hn is played for 30 s in the first phonetic symbol sequence and for 15 s in the to-be-recognized audio, the playback speed scaling ratio can be calculated as 2. The playback speed scaling ratio is used when the entire to-be-recognized audio sequence is matched to the matching sequence range in the first phonetic symbol sequence, and the matching sequence time range is expanded by 2 times the total time length of the to-be-recognized audio sequence. If the playback time length of the complete to-be-recognized phonetic symbol sequence is 25 s, the playback time length of the candidate phonetic symbol sequence corresponding to the to-be-recognized phonetic symbol sequence in the first phonetic symbol sequence is 50 s.

[0170] a2, determining a candidate phonetic symbol sequence that is time matched to the to-be-recognized phonetic symbol sequence according to the playback speed scaling ratio and the positions of the candidate sub-sequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence.

[0171] In this step, the playback time length of the to-be-recognized phonetic symbol sequence in the first phonetic symbol sequence can be determined according to the playback speed scaling ratio. The starting point of the candidate phonetic symbol sequence is expanded according to the candidate sub-sequence. The matching range is expanded from the candidate sub-sequence to both sides. The candidate phonetic symbol sequence that is time matched to the entire to-be-recognized phonetic symbol sequence can be accurately located by combining the relative position relationship with the playback time length.

[0172] In this way, the influence of the playback speed difference on audio recognition can be effectively overcome, and the accuracy and robustness of recognition can be improved. Meanwhile, the local-to-global matching idea makes it possible to more flexibly process audio sequences of different lengths and complexities, providing more extensive possibilities for the application of song recognition technology.

[0173] For the foregoing step a2, the candidate phonetic symbol sequence that is time matched to the to-be-recognized phonetic symbol sequence is determined from the first phonetic symbol sequence containing the candidate sub-sequence, as shown inFIG. 3B The flowchart shown in the example can be implemented by the following steps:

[0174] S301, obtaining a first time interval between a start timestamp of the to-be-recognized phoneme sequence and a start timestamp of a candidate sub-sequence in the to-be-recognized phoneme sequence;

[0175] The start timestamp of the to-be-recognized phoneme sequence is used to represent the pronunciation time of the first phoneme in the to-be-recognized phoneme sequence in the to-be-recognized audio. As shown in the example, FIG. 3C The example of the timestamp range determination based on the phoneme sequence shown in the example is shown in the example. The to-be-recognized phoneme sequence corresponding to the to-be-recognized audio contains m phonemes, denoted as R1...Rm, and the pronunciation time stamp of each phoneme corresponding to the to-be-recognized audio is denoted as T1...Tm. The start timestamp of the to-be-recognized phoneme sequence is T1.

[0176] The start timestamp of the candidate sub-sequence in the to-be-recognized phoneme sequence is used to represent the pronunciation time of the first phoneme in the candidate sub-sequence in the to-be-recognized phoneme sequence in the to-be-recognized audio. As shown in the example, FIG. 3C As shown in the example, the timestamp of the candidate sub-sequence R_h1...R_hn in the to-be-recognized audio is denoted as T_r1...T_rn, and the start timestamp of the candidate sub-sequence is T_r1.

[0177] The first time interval refers to the difference between the start timestamp of the to-be-recognized phoneme sequence and the start timestamp of the candidate sub-sequence, which represents the time offset of the candidate sub-sequence in the to-be-recognized phoneme sequence relative to the start position of the entire to-be-recognized phoneme sequence. For example, FIG. 3C The first time interval is the time interval L1= | T_r1 -T1| shown in the example.

[0178] S302, obtaining a second time interval between an end timestamp of the to-be-recognized phoneme sequence and an end timestamp of the candidate sub-sequence in the to-be-recognized phoneme sequence;

[0179] The end timestamp of the to-be-recognized phoneme sequence is used to represent the pronunciation time of the last phoneme in the to-be-recognized phoneme sequence in the to-be-recognized audio. As shown in the example, FIG. 3C As shown in the example, the pronunciation time stamp of each phoneme corresponding to the to-be-recognized phoneme sequence in the to-be-recognized audio is denoted as T1...Tm, and the end timestamp of the to-be-recognized phoneme sequence is Tm.

[0180] The end timestamp of the candidate sub-sequence in the to-be-recognized phoneme sequence is used to represent the pronunciation time of the last phoneme in the candidate sub-sequence in the to-be-recognized phoneme sequence in the to-be-recognized audio. As shown in the example, FIG. 3CAs shown, the timestamps of the candidate sub-sequence R_h1...R_hnin the to-be-recognized audio are represented as T_r1...T_rn, and the end timestamp of the candidate sub-sequence is T_rn.

[0181] The second time interval refers to the difference between the end timestamp of the to-be-recognized phoneme sequence and the end timestamp of the candidate sub-sequence, which represents the time offset of the candidate sub-sequence relative to the end position of the entire sequence within the to-be-recognized phoneme sequence, and can also be understood as the time distance between the end point of the candidate sub-sequence and the end point of the to-be-recognized phoneme sequence. For example, FIG. 3C The first time interval is the time interval L2 = |Tn - T_rn | shown in the figure.

[0182] S303, according to the start timestamp and the end timestamp of the candidate sub-sequence in the first phoneme sequence, the first time interval, the second time interval, and the playback speed scaling ratio, determine the timestamp range of the candidate phoneme sequence in the first phoneme sequence;

[0183] The start timestamp and the end timestamp of the candidate sub-sequence in the first phoneme sequence identify the start position and the end position of the candidate sub-sequence in the first phoneme sequence.

[0184] In this embodiment, according to the start timestamp of the candidate sub-sequence in the first phoneme sequence, the playback speed scaling ratio, and the first time interval, the start timestamp of the candidate phoneme sequence in the first phoneme sequence can be determined; similarly, according to the end timestamp of the candidate sub-sequence in the first phoneme sequence, the playback speed scaling ratio, and the second time interval, the end timestamp of the candidate phoneme sequence in the first phoneme sequence can be determined.

[0185] Wherein, the first time interval reflects the time offset of the candidate sub-sequence relative to the start position of the entire to-be-recognized phoneme sequence within the to-be-recognized phoneme sequence, and then according to the playback speed scaling ratio and the first time interval, the time offset of the candidate sub-sequence relative to the start position of the candidate phoneme sequence within the first phoneme sequence can be calculated; according to the calculated time offset and the start timestamp of the candidate sub-sequence in the first phoneme sequence, the expected start timestamp of the candidate phoneme sequence can be calculated. For example, FIG. 3CFor example, for the time interval L1 on the phoneme sequence to be recognized, to determine the corresponding time interval L1' on the first phoneme sequence, the playback time scaling ratio can be used for determination. In the case where the playback time scaling ratio represents (the playback time length in the candidate song S) / (the playback time length in the audio to be recognized), the time interval L1' can be represented as L1'=L1*scale, and the start time stamp of the candidate phoneme sequence in the first phoneme sequence can be represented as Seg_start=T_s1-L1'.

[0186] Based on similar principles, for the time interval L2 on the phoneme sequence to be recognized, to determine the corresponding time interval L2' on the first phoneme sequence, the playback time scaling ratio can be used for determination. In the case where the playback time scaling ratio represents (the playback time length in the candidate song S) / (the playback time length in the audio to be recognized), the time interval L2' can be represented as L2'=L2*scale, and the end time stamp of the candidate phoneme sequence in the first phoneme sequence can be represented as Seg_end=T_s+L2'.

[0187] According to the expected start time stamp and the expected end time stamp of the candidate phoneme sequence in the first phoneme sequence determined in this step, the time stamp range of the candidate phoneme sequence in the first phoneme sequence can be determined, as shown in [Seg_start, Seg_end]. FIG. 3C

[0188] In S304, the phoneme sequence corresponding to the time stamp range in the first phoneme sequence is determined as the candidate phoneme sequence.

[0189] Referring to FIG. 4A , according to the start position Seg_start and the end position Seg_end of the segment in the candidate song S, the phoneme sequence in the time range is searched out as the candidate phoneme sequence corresponding to the candidate song, and the candidate phoneme sequence corresponding to the audio to be recognized satisfies the time matching.

[0190] In the embodiments of the present disclosure, according to the phoneme sequence to be recognized and the corresponding time stamp, and the candidate sub-sequence and the time stamp matched in the candidate song set, the time scaling ratio can be calculated, and the candidate phoneme sequence matched in time with the entire phoneme sequence to be recognized, the start position and the end position in the candidate song are determined, so as to realize the accurate alignment in time and provide a data basis for subsequent phoneme alignment.

[0191] In some embodiments, for the alignment of the audio to be recognized with the phonemes in the candidate phoneme sequence in the foregoing step S104, referring to the phoneme alignment step flowchart shown in FIG. 4B , the alignment can be realized in the following manner:​

[0192] S401, for each frame of the audio to be identified, using a pre-trained phoneme recognition network to generate the probability that each frame belongs to each international phonetic symbol;

[0193] Each frame of the audio to be identified represents a series of small time periods into which the audio signal is divided, each frame typically contains a fixed number of audio samples for subsequent processing and analysis. The length of each frame can be adjusted according to actual conditions, but should be short enough to capture subtle changes in speech.

[0194] The pre-trained phoneme recognition network refers to a neural network model that has been trained on a large amount of data for recognizing phonemes in audio signals, which can be constructed using deep learning techniques, capable of automatically extracting features from audio signals and mapping them to corresponding phonemes. This phoneme recognition network is the same network as the one used to obtain the phoneme sequence to be identified in the aforementioned embodiments.

[0195] The phoneme recognition network is used to convert audio into an international phonetic symbol sequence, and the training goal is to associate the speech features in the audio with the corresponding international phonetic symbols, and to learn the audio representation of each phonetic symbol. In the present application, 108 phonetic symbols can be used as training targets, including 107 international phonetic symbols, plus one other pronunciation or silence symbol sil. The phoneme recognition network can be modeled using traditional hidden Markov models, or end-to-end deep learning models. The model structure of the phoneme recognition network can use neural networks such as convolutional neural networks or recurrent neural networks, etc., which can learn deep features of phonemes from audio and improve robustness to non-speech interference.

[0196] The overall training process of the phoneme recognition network can be seen in FIG. 4C For song audio with actual phoneme sequence labels in the training sample, the song audio can be Fourier transformed by frame, such as 25ms frame length and 10ms frame shift, to obtain the original spectral features; Next, the original spectral features are converted into mel-spectral features or MFCC features, and the features are input into the phoneme recognition network for inference. The target label for training is the actual phoneme sequence of the song, and the loss function such as the CTC function is calculated as the error, i.e. the maximum probability when the output is the target phoneme sequence given the input features. Gradient descent is used for error backpropagation to continuously update the neural network model until the error calculated by the loss function no longer decreases or the specified training conditions are met to end training.

[0197] Before inputting the audio into the phoneme recognition network, the audio to be identified can be preprocessed, such as noise reduction, echo removal, volume normalization, etc., to improve the accuracy of phoneme recognition.

[0198] For each frame, its audio features are extracted by the phonetic recognition network, which can be mel-frequency cepstral coefficients, spectral envelope, fundamental frequency, etc., for describing the spectral and temporal characteristics of the audio signal. The extracted features are automatically processed by the phonetic recognition network, and the probability of each frame belonging to each international phonetic symbol is output, which represents the possibility of the pronunciation in the frame belonging to the international phonetic symbol. For a certain frame in the audio, after network calculation, the probability of phonetic symbol [p] is 0.1, the probability of phonetic symbol [b] is 0.05, etc. Using 108 international phonetic symbols, 108 probability values corresponding to each international phonetic symbol will be output for each frame.

[0199] S402, constructing a decoding sequence according to the candidate phonetic symbol sequence; the nodes in the decoding sequence correspond to the phonetic symbols in the candidate phonetic symbol sequence;

[0200] The decoding sequence is used to represent the state transition relationship of the phonetic symbols in the candidate phonetic symbol sequence at different time points, and is used for decoding according to the probability of each frame belonging to each international phonetic symbol, so as to determine the accurate position (start time and end time) of the phonetic symbol in the audio.

[0201] Referring to FIG. 4B An illustrative decoding sequence diagram is shown, taking the candidate phonetic symbol sequence “ ” as an example, which contains phonetic symbols in turn [m], [ɪ], [ŋ], [tʰ], [j], [ɛ], [n], [i], [x], [ ], a total of 10 phonetic symbols. The constructed decoding sequence can be represented as FIG. 4D , the state transition order represented by the arrow is consistent with the order of the contained phonetic symbols, and the 10 phonetic symbols form at least 10 nodes.

[0202] Based on the characteristics that most song performances will pause between sentences and may not pause within a sentence, the decoding sequence can adopt a serial structure, including a plurality of sequence nodes; in the case that the phonetic symbols corresponding to a plurality of continuous sequence nodes belong to a complete sentence, the adjacent front node of the start node and the adjacent rear node of the end node in the plurality of continuous sequence nodes are nodes representing silence; the adjacent nodes of the plurality of continuous sequence nodes include skipable nodes representing silence. Through such a decoding sequence setting, the front and rear of each sentence must have a sil node, and the phonetic symbols between the internal nodes of the sentence are skipable sil nodes, thereby improving the accuracy of phonetic symbol alignment.

[0203] As FIG. 4D illustratively shown, another decoding sequence diagram is shown, taking the candidate phonetic symbol sequence “ ” as an example, which belongs to a complete sentence, has sequence nodes representing silence before and after it, and sets skipable sequence nodes representing silence between the internal phonetic symbols of the sentence.

[0204] S403, according to the probability of each frame belonging to each international phonetic symbol, decoding the decoding sequence to obtain a trigram sequence of the phonetic symbol alignment of the candidate phonetic symbol sequence in the audio to be recognized, the trigram including a phonetic symbol, a start time and an end time of the phonetic symbol in the audio to be recognized.

[0205] The trigram sequence is the output of the decoding process, and each trigram includes a phonetic symbol, a start time and an end time of the phonetic symbol in the audio to be recognized. The trigram sequence provides complete information about the exact position and duration of each phonetic symbol in the candidate phonetic symbol sequence in the audio to be recognized.

[0206] Regarding decoding using the decoding sequence, Viterbi decoding can be performed using the phonetic symbol probability of each frame to obtain the alignment result of the final candidate phonetic symbol sequence with the input audio to be recognized. Viterbi decoding can find the most likely phonetic symbol alignment result according to the phonetic symbol probability and the structure of the decoding sequence, and calculate the corresponding decoding path score.

[0207] To better understand the decoding process, the present embodiment takes the decoding sequence of the Viterbi decoding and the diagram D as an example to exemplarily describe the decoding process:

[0208] Suppose the phonetic symbol probability of the audio to be recognized frame by frame is as follows:

[0209] frame 1: [0.01, 0.02,..., 0.9,...]

[0210] frame 2: [0.01, 0.02,..., 0.8,...]

[0211] frame 3: [0.01, 0.02,..., 0.7,...]

[0212] frame 4: [0.01, 0.02,..., 0.6,...]

[0213] frame 5: [0.01, 0.02,..., 0.6,...]

[0214] frame N: [0.01, 0.02,..., 0.5,...]

[0215] Based on FIG. 4D Viterbi decoding of the example decoding sequence (f:1 represents the first frame, f:1-2 represents the 1-2 frames, and the others are the same):

[0216] Possible path of frame 1: sil (f:1)

[0217] Possible paths of frame 2: sil(f:1-2), sil(f:1)->m(f:2)

[0218] Possible paths of frame 3: sil(f:1-3), sil(f:1-2)->m(f:3), sil(f:1)->m(f:2)->I(f:3)

[0219] …

[0220] By analogy, possible paths of the Nth frame: …, sil(f:1)->m(f:2)->I(f:3)->ŋ(f:4)->… (f:N-1)->sil(f:N).

[0221] The obtained triad sequence is the path with the maximum path probability from all decoding paths, assuming that the path is sil(f:1-2)->m(f:3)->I(f:4)->ŋ(f:5)->… (f:N-1)->sil(f:N), the path reaching the last node must pass through all phonemes.

[0222] The obtained triad sequence is a sequence composed of phoneme symbols, start times and end times of the phoneme symbols in the audio to be recognized, indicating the final phoneme alignment result, and clearly indicating the accurate time range of each phoneme in the audio to be recognized. For example, the decoding sequence shown in Exemplary Apparatus The corresponding triad sequence can be represented as (sil, 0, 10), (m, 10, 12), (I, 12, 13), (ŋ, 13, 15), (t, 15, 16), (j, 16, 17)…

[0223] In the embodiments of the present disclosure, by utilizing the pre-trained phoneme recognition network, the probability of each international phoneme for each frame of the audio to be recognized is generated, which can more accurately capture the pronunciation details in the audio, and a decoding sequence is constructed based on the candidate phoneme sequence in the candidate song that is time-matched with the phoneme sequence to be recognized, and the sequence is utilized for decoding, which can more flexibly process the pronunciation variation and coarticulation phenomenon in the audio, thereby realizing the phoneme alignment of the audio to be recognized and the candidate phoneme sequence, which can reduce the mis-matching condition, thereby improving the accuracy of song recognition.

[0224] In some embodiments, based on the triple obtained from the aforementioned phonetic alignment, the phonetic alignment score of the phonetic sequence in which the to-be-recognized audio is pronounced according to the to-be-recognized audio obtained in the aforementioned step S104, the phonetic alignment score can be calculated by the posterior alignment probability, and the phonetic alignment score can be calculated by the following steps b1-b3:

[0225] b1, for each triple in the decoded triple sequence, first obtain the probability mean of each international phonetic symbol in the triple time range according to the probability of each frame in the to-be-recognized audio belonging to each international phonetic symbol; the triple time range is determined according to the start time and end time of the phonetic symbol in the to-be-recognized audio in the triple.

[0226] For any international phonetic symbol, the probability mean is used to represent the average value of the probability of each frame in the to-be-recognized audio in the time range represented by the triple being identified as belonging to the international phonetic symbol, reflecting the stability and consistency of the audio content being identified as a certain phonetic symbol in the time range.

[0227] Taking the triple (j, 16, 17) as an example, in the triple, j represents an international phonetic symbol, and 16 and 17 represent the start time and end time of the phonetic symbol in the to-be-recognized audio in seconds. The probabilities of all frames from 16s to 17s being identified as each international phonetic symbol are obtained, for example, Frame 16: [0.01, 0.02, 0.05,..., 0.90,...], Frame 17: [0.02, 0.03, 0.06,..., 0.85,...], the correspondence between the index in the array and the phonetic symbol has been determined, and it is assumed that the scores of the phonetic symbol j are 0.90 and 0.85 respectively, then the probability mean of the phonetic symbol j in the triple time range is the sum of the probabilities of the phonetic symbol j in the two frames, and then divided by the number of frames 2, that is, the probability mean of the phonetic symbol j = (0.90+0.85) / 2 = 0.875. Based on the same principle, the calculation method of each international phonetic symbol is the same as that of the phonetic symbol j.

[0228] b2, according to the probability mean of the phonetic symbol in the triple and the sum of the probability means of each international phonetic symbol in the triple time range, the posterior alignment probability of the triple is calculated;

[0229] That is, after obtaining the probability mean of each international phonetic symbol in the triple time range, the sum of the probability means of the international phonetic symbol is first calculated where M represents the total number of international phonetic symbols (such as 108), represents the probability mean.

[0230] The posterior alignment probability of the triple can be represented as P = char / where char represents the probability mean of the phonetic symbol in the triple within the triple time range. For example, taking the triple (j, 16, 17) as an example, the posterior alignment probability represents the probability mean of the phonetic symbol j / the sum of the probability means of all IPA symbols within the 16-17s.

[0231] The posterior alignment probability ranges from [0, 1], and the greater the value of the posterior alignment probability, the greater the probability that the pronunciation in the time period is the phonetic symbol in the triple.

[0232] b3, obtaining a phonetic symbol alignment score corresponding to the candidate phonetic symbol sequence according to the posterior alignment probability of each triple.

[0233] After obtaining the posterior alignment probability of each triple, the posterior alignment probabilities of all triples can be summed up, and the sum is taken as the phonetic symbol alignment score. Alternatively, a weight can be assigned to each triple according to the importance of the phonetic symbol in the song, the difficulty of pronunciation, and other factors, and then the posterior alignment probability of each triple is multiplied by the corresponding weight, and the sum is taken to obtain the phonetic symbol alignment score, so as to more accurately reflect the matching degree of the audio and the candidate phonetic symbol sequence. Alternatively, any applicable phonetic symbol alignment score calculation method can be used, and the present application does not limit this.

[0234] In the embodiments of the present disclosure, the posterior alignment probability reflects the matching degree of a certain phonetic symbol with the audio content within a given audio frame range. By calculating the posterior alignment probability of each triple, the matching degree of the candidate phonetic symbol sequence and the audio to be recognized can be more accurately evaluated, and then the phonetic symbol alignment score can be obtained according to the posterior alignment probability, which can more objectively reflect the similarity between the candidate phonetic symbol sequence and the audio to be recognized, and provide strong support for subsequent song recognition result sorting or selection.

[0235] In some embodiments, song recognition based on phonetic symbols relies on phonetic symbol matching. In order to effectively match the phonetic symbol information of the audio to be recognized with the information in the database and optimize the efficiency and accuracy of the recognition process, after obtaining the phonetic symbol sequence to be recognized, it can be detected whether the amount of phonetic symbol information contained in the phonetic symbol sequence to be recognized is sufficient to avoid the problem that the amount of phonetic symbol information is too small to effectively match. That is, for the obtained phonetic symbol sequence to be recognized, the length of the phonetic symbol sequence to be recognized can be obtained, and it can be detected whether the length of the phonetic symbol sequence to be recognized is greater than a set threshold; in the case where the length of the phonetic symbol sequence to be recognized is greater than the set threshold, the phonetic symbol sequence to be recognized is compared with the first phonetic symbol sequence; in the case where the length of the phonetic symbol sequence to be recognized is less than the set threshold, the song recognition is terminated.

[0236] The length of the to-be-recognized phonetic symbol sequence is used to represent the number of international phonetic symbols contained in the to-be-recognized phonetic symbol sequence. The longer the length of the to-be-recognized phonetic symbol sequence is, the more pronunciation features contained in the to-be-recognized audio are, thereby providing more information for comparison and recognition. The length of the to-be-recognized phonetic symbol sequence can be obtained by counting each phonetic symbol through traversing the to-be-recognized phonetic symbol sequence, or can be obtained by a string processing function or other statistical tools.

[0237] The threshold is used to represent the minimum number of phonetic symbols that can be used for song recognition. The value of the threshold can be determined by the developer according to experience or through multiple tests.

[0238] When the length of the to-be-recognized phonetic symbol sequence is less than the threshold, it is indicated that the to-be-recognized audio segment is too short and contains limited phonetic symbol information, which is insufficient to support effective matching with a complete song sequence. Therefore, it is difficult to obtain a reliable recognition result through phonetic symbol matching. Therefore, in order to save computing resources and avoid unnecessary processing, the song recognition process is terminated when the length of the to-be-recognized phonetic symbol sequence is less than the threshold, thereby ensuring the accuracy of the recognition result while improving the overall running efficiency.

[0239] When the song recognition is terminated, prompt information such as "audio is too short, please re-enter" can be displayed to the user through the song recognition interactive interface, so as to prompt the user of the specific reason for the current recognition failure and guide the user to perform corresponding operations, thereby helping the user to quickly understand the problem and enabling the user to more smoothly complete the song recognition process, thereby improving the user experience.

[0240] In the embodiments of the present disclosure, by detecting the length of the to-be-recognized phonetic symbol sequence and directly terminating the song recognition when the length of the to-be-recognized phonetic symbol sequence is insufficient, unnecessary comparison processes and long waiting time of the user for inaccurate or failed results are avoided, the user experience is optimized, the waste of computing resources is reduced, and the efficiency of the entire recognition process is improved. At the same time, this method enables sequences containing sufficient phonetic symbol information to participate in comparison, which can effectively avoid misrecognition caused by insufficient information. The rich phonetic symbol information can more accurately reflect the features of the audio content, thereby improving the accuracy of song recognition.

[0241] In some embodiments, based on the above song recognition method, when a new language song needs to be incorporated into the database to facilitate subsequent song recognition, the new language song can be converted into an international phonetic symbol sequence and stored. In the way of phonetic symbol comparison based on inverted index, the new language song can be converted into an international phonetic symbol sequence, divided into multiple subsequences according to the set sequence length, formed into a new index key, and added to the data structure of the existing inverted index. The index value corresponding to the new index key is the new language song, and the time position of each index key in the new language song.

[0242] In the embodiments of the present disclosure, in the case that there is a new language song, only the phoneme correspondence relationship between the new language and the International Phonetic Alphabet needs to be established, so that the recognition of the new language song can be quickly supported, and a large amount of audio data of the new language is not needed to retrain the model, the expansibility is stronger, and the support cost of the new language is reduced.

[0243] FIG. 5

[0244] After the method of the exemplary embodiments of the present application is introduced, next, the device of the exemplary embodiments of the present application is described with reference to the drawings. FIG. 5 The device of the exemplary embodiments of the present application is described.

[0245] The implementation process of the functions and roles of each module in the device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here. For the device embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment.

[0246] Exemplary Medium A song recognition device according to an embodiment of the present application is schematically shown, which comprises:

[0247] The phoneme sequence to be recognized acquisition module 501 is configured to acquire the phoneme sequence to be recognized corresponding to the audio to be recognized;

[0248] The phoneme comparison module 502 is configured to compare the phonemes in the phoneme sequence to be recognized and the first phoneme sequence of each candidate song, and determine the candidate subsequence in which the phonemes are the same and the sequence length meets the requirement.

[0249] The candidate phoneme sequence acquisition module 503 is configured to determine the candidate phoneme sequence that is time-matched with the phoneme sequence to be recognized from the first phoneme sequence containing the candidate subsequence according to the position of the candidate subsequence in the phoneme sequence to be recognized and the first phoneme sequence.

[0250] The phoneme alignment module 504 is configured to align the phonemes in the audio to be recognized and the candidate phoneme sequence, and obtain the phoneme alignment score of the audio to be recognized according to the candidate phoneme sequence.

[0251] The song recognition result generation module 505 is configured to determine the song recognition result of the audio to be recognized according to the phoneme alignment score corresponding to each candidate phoneme sequence.

[0252] In some embodiments, the phoneme comparison module is specifically configured to:

[0253] The division module is configured to divide the phoneme sequence to be recognized into a plurality of phoneme subsequences to be recognized according to the set sequence length.

[0254] An index key matching module is configured to match each to-be-identified sub-sequence with an index key of the inverted index; the index key is a first phonetic sequence; the first phonetic sequence refers to each sub-sequence obtained by dividing a complete phonetic sequence of each candidate song according to the set sequence length; an index value corresponding to the index key is candidate song information containing the index key;

[0255] A candidate sub-sequence determining module is configured to take an index key identical to a phonetic sequence of the to-be-identified sub-sequence as the candidate sub-sequence.

[0256] In some embodiments, the dividing module is specifically configured to:

[0257] Starting from a starting phonetic of the to-be-identified phonetic sequence, a window is used to slide on the to-be-identified phonetic sequence according to a set step length and an adjacent window overlap value, so as to divide the to-be-identified phonetic sequence into a plurality of to-be-identified sub-sequences; a length of the window is set as the set sequence length.

[0258] In some embodiments, the candidate phonetic sequence obtaining module is specifically configured to:

[0259] A scaling ratio determining module is configured to obtain a playing time speed scaling ratio of a candidate sub-sequence on a to-be-identified phonetic sequence and a first phonetic sequence;

[0260] According to the playing time speed scaling ratio and positions of the candidate sub-sequence in the to-be-identified phonetic sequence and the first phonetic sequence, a candidate phonetic sequence time-matched with the to-be-identified phonetic sequence is determined.

[0261] In some embodiments, the scaling ratio determining module is specifically configured to:

[0262] According to a starting time stamp and an ending time stamp of the candidate sub-sequence in the to-be-identified phonetic sequence, a first playing time length is determined;

[0263] According to a starting time stamp and an ending time stamp of the candidate sub-sequence in the first phonetic sequence, a second playing time length is determined;

[0264] According to the first playing time length and the second playing time length, the playing time speed scaling ratio is determined.

[0265] In some embodiments, the candidate phonetic sequence determining module is specifically configured to:

[0266] A first time interval between a starting time stamp of the to-be-identified phonetic sequence and a starting time stamp of a candidate sub-sequence in the to-be-identified phonetic sequence is obtained;

[0267] obtaining a second time interval between the end timestamp of the to-be-recognized phonetic symbol sequence and an end timestamp of the candidate sub-sequence in the to-be-recognized phonetic symbol sequence;

[0268] determining, according to the start timestamp and the end timestamp of the candidate sub-sequence in the first phonetic symbol sequence, the first time interval, the second time interval, and the playback speed scaling ratio, a timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence;

[0269] determining, as the candidate phonetic symbol sequence, a phonetic symbol sequence in the first phonetic symbol sequence corresponding to the timestamp range.

[0270] In some embodiments, when determining the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence, the candidate phonetic symbol sequence determining module includes:

[0271] determining, according to the start timestamp of the candidate sub-sequence in the first phonetic symbol sequence, the playback speed scaling ratio, and the first time interval, a start timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence;

[0272] determining, according to the end timestamp of the candidate sub-sequence in the first phonetic symbol sequence, the playback speed scaling ratio, and the second time interval, an end timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence.

[0273] In some embodiments, the phonetic symbol alignment module is specifically configured to:

[0274] for each frame of the to-be-recognized audio, generating, by using a pre-trained phonetic symbol recognition network, a probability that each frame belongs to each international phonetic symbol;

[0275] constructing, according to the candidate phonetic symbol sequence, a decoding sequence; a node in the decoding sequence corresponds to a phonetic symbol in the candidate phonetic symbol sequence;

[0276] decoding, according to the probability that each frame belongs to each international phonetic symbol, by using the decoding sequence, to obtain a triple sequence in which phonetic symbols in the to-be-recognized audio align with the candidate phonetic symbol sequence, the triple including a phonetic symbol, a start time of the phonetic symbol in the to-be-recognized audio, and an end time of the phonetic symbol in the to-be-recognized audio.

[0277] In some embodiments, the decoding sequence adopts a serial structure and includes a plurality of sequence nodes; in a case where phonetic symbols corresponding to a plurality of continuous sequence nodes belong to a complete lyric,

[0278] a neighboring node before a start node and a neighboring node after an end node in the plurality of continuous sequence nodes are nodes representing silence; neighboring nodes of the plurality of continuous sequence nodes include a node representing silence that can be skipped.

[0279] In some embodiments, the phonetic symbol alignment module is specifically configured to:

[0280] For each triple, according to the probability of each frame in the audio to be recognized belonging to each international phonetic symbol, obtain the probability mean of each international phonetic symbol in the triple time range; the triple time range is determined according to the start time and end time of the phonetic symbol in the triple in the audio to be recognized;

[0281] According to the probability mean of the phonetic symbol in the triple and the sum of the probability mean of each international phonetic symbol in the triple time range, calculate the posterior alignment probability of the triple;

[0282] According to the posterior alignment probability of each triple, obtain the phonetic symbol alignment score corresponding to the candidate phonetic symbol sequence.

[0283] In some embodiments, the phonetic symbol sequence to be recognized obtaining module is specifically configured to:

[0284] Using a pre-trained phonetic symbol recognition network to perform phonetic symbol recognition on the audio to be recognized, to obtain the corresponding phonetic symbol sequence to be recognized and the time stamp information of each phonetic symbol in the audio to be recognized;

[0285] The apparatus further comprises:

[0286] In the case that the length of the phonetic symbol sequence to be recognized is greater than a set threshold, perform phonetic symbol alignment on the phonetic symbol sequence to be recognized and the first phonetic symbol sequence; in the case that the length of the phonetic symbol sequence to be recognized is less than a set threshold, terminate the song recognition.

[0287] FIG. 6

[0288] After introducing the method and apparatus of the exemplary embodiments of the present application, next, reference is made to Exemplary Computing Device The readable storage medium of the exemplary embodiments of the present application is described.

[0289] In the present exemplary embodiments, the above method can be realized by a program product, such as a portable compact disc read-only memory (CD-ROM) including program codes and can be run on a device, such as a personal computer. However, the program product of the present application is not limited to this, and in the present document, the readable storage medium can be any tangible medium containing or storing a program which can be used by or in conjunction with an instruction execution system, apparatus or device.

[0290] The program product can employ any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0291] The computer-readable signal medium can include a computer-readable storage medium that is propagated as a carrier wave. The computer-readable signal medium can further be any computer-readable medium that is not a storage medium. The computer-readable signal medium can be a computer-readable storage medium that is a propagated signal on a carrier wave.

[0292] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0293] The program code can be executed by one or more programmable processors, which can be implemented using one or more microprocessors, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any other devices suitable for retrieval and execution of instructions. The program code can execute entirely on a user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0294] FIG. 7

[0295] Having introduced the method, device, and medium of the exemplary embodiments of the present application, next, reference will be made to FIG. 7 A computing device of the exemplary embodiments of the present application is described.

[0296] FIG. 7The computing device 700 illustrated is merely one example and should not be taken as limiting the scope of functionality of the present embodiments.

[0297] As FIG. 7 illustrated, computing device 700 is in the form of a general-purpose computing device. The components of computing device 700 can include, but are not limited to, one or more processing units 701, one or more storage units 702, and a bus 703 that connects the various system components, including the processing units 701 and the storage units 702.

[0298] The bus 703 includes a data bus, a control bus, and an address bus.

[0299] The storage units 702 can include a readable medium that is in the form of volatile memory, such as random access memory (RAM) 7021 and / or cache memory 7022, and can further include a readable medium that is in the form of non-volatile memory, such as read-only memory (ROM) 7023.

[0300] The storage units 702 can also include a program / utility 7025 having a set of program modules 7024, including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof can include an implementation of a network environment.

[0301] The computing device 700 can also communicate with one or more external devices 704 (such as a keyboard or a pointing device) via an input / output (I / O) interface 705.

[0302] Such communication can occur via Input / Output (I / O) interface 705. Still yet, computing device 700 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet through a network adapter 706. As ​ illustrated, network adapter 706 communicates with the other system components, including the processing unit 701, via bus 703. It should be appreciated that the network adapter 706 can also be utilized to enable connection to other types of networks and remote computing systems.

[0303] It should be noted that, although in the foregoing detailed description several units / modules or sub-units / modules of the song recognition apparatus are mentioned, such a division is merely exemplary and not mandatory. Indeed, according to embodiments of the present application, the features and functionalities of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functionalities of one unit / module described above can be further divided into units / modules.

[0304] Moreover, although the operations of the method(s) of the present application are described in a particular, sequential order, this order is not meant to be a limitation and is not intended to imply that

[0305] While the spirit and principles of the present application have been described with reference to several specific embodiments, it is to be understood that the application is not limited to the precise embodiments disclosed and that various modifications and equivalents can be used without departing from the scope of the application.

Claims

1. A song recognition method characterized by comprising: The method comprises: obtaining a to-be-recognized phonetic symbol sequence corresponding to to-be-recognized audio; performing phonetic symbol comparison on the to-be-recognized phonetic symbol sequence and first phonetic symbol sequences of respective candidate songs, to determine candidate sub-sequences that are identical in phonetic symbols and meet sequence length requirements in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequences; determining, from the first phonetic symbol sequences containing the candidate sub-sequences, a candidate phonetic symbol sequence that is time-matched with the to-be-recognized phonetic symbol sequence according to positions of the candidate sub-sequences in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequences; aligning the to-be-recognized audio with phonetic symbols in the candidate phonetic symbol sequence, to obtain a phonetic symbol alignment score of the to-be-recognized audio according to pronunciation of the candidate phonetic symbol sequence; determining a song recognition result of the to-be-recognized audio according to phonetic symbol alignment scores corresponding to respective candidate phonetic symbol sequences.

2. The method of claim 1, wherein, The phonetic symbol comparison comprises: dividing the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized sub-sequences according to a set sequence length; matching each to-be-recognized sub-sequence with an index key of an inverted index; wherein the index key is a first phonetic symbol sequence, the first phonetic symbol sequence is each sub-sequence obtained by dividing a complete phonetic symbol sequence of each candidate song according to the set sequence length, and an index value corresponding to the index key is candidate song information containing the index key; taking an index key identical in phonetic symbols to the to-be-recognized sub-sequence as the candidate sub-sequence.

3. The method of claim 2, wherein, The division of the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized sub-sequences according to a set sequence length comprises: starting from a starting phonetic symbol of the to-be-recognized phonetic symbol sequence, sliding a window on the to-be-recognized phonetic symbol sequence according to a set step length and an adjacent window overlap value, to divide the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized sub-sequences; wherein a length of the window is set as the set sequence length.

4. The method of claim 1, wherein, The determination of a candidate phonetic symbol sequence that is time-matched with the to-be-recognized phonetic symbol sequence from the first phonetic symbol sequences containing the candidate sub-sequences comprises: obtaining a playing time speed scaling ratio of the candidate sub-sequences on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequences; determining the candidate phonetic symbol sequence that is time-matched with the to-be-recognized phonetic symbol sequence according to the playing time speed scaling ratio and positions of the candidate sub-sequences in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequences.

5. The method of claim 4, wherein, The obtaining of the playing time speed scaling ratio of the candidate sub-sequences on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequences comprises: determining a first playing time length according to a starting time stamp and an ending time stamp of the candidate sub-sequences in the to-be-recognized phonetic symbol sequence; determining a second playing time length according to a starting time stamp and an ending time stamp of the candidate sub-sequences in the first phonetic symbol sequence; determining the playing time speed scaling ratio according to the first playing time length and the second playing time length.

6. The method of claim 4, wherein, The determination of a candidate phonetic symbol sequence that is time-matched with the to-be-recognized phonetic symbol sequence from the first phonetic symbol sequences containing the candidate sub-sequences comprises: obtaining a first time interval between a starting time stamp of the to-be-recognized phonetic symbol sequence and a starting time stamp of the candidate sub-sequences in the to-be-recognized phonetic symbol sequence; obtaining a second time interval between an ending time stamp of the to-be-recognized phoneme sequence and an ending time stamp of a candidate sub-sequence in the to-be-recognized phoneme sequence; determining a time stamp range of the candidate phoneme sequence in the first phoneme sequence according to a starting time stamp and an ending time stamp of the candidate sub-sequence in the first phoneme sequence, a first time interval, a second time interval, and the playback speed scaling ratio; determining a phoneme sequence corresponding to the time stamp range in the first phoneme sequence as the candidate phoneme sequence.

7. The method of claim 6, wherein, The determining of the time stamp range of the candidate phoneme sequence in the first phoneme sequence comprises: determining a starting time stamp of the candidate phoneme sequence in the first phoneme sequence according to a starting time stamp of the candidate sub-sequence in the first phoneme sequence, the playback speed scaling ratio, and the first time interval; determining an ending time stamp of the candidate phoneme sequence in the first phoneme sequence according to an ending time stamp of the candidate sub-sequence in the first phoneme sequence, the playback speed scaling ratio, and the second time interval.

8. The method of claim 1, wherein, The aligning of the to-be-recognized audio with the phonemes in the candidate phoneme sequence comprises: generating, for each frame of the to-be-recognized audio, a probability that each frame belongs to each international phoneme by using a pre-trained phoneme recognition network; constructing a decoding sequence according to the candidate phoneme sequence; a node in the decoding sequence corresponds to a phoneme in the candidate phoneme sequence; decoding, according to the probability that each frame belongs to each international phoneme, by using the decoding sequence, to obtain a triple sequence in which phonemes in the to-be-recognized audio align with the candidate phoneme sequence; the triple comprises a phoneme symbol, a starting time of the phoneme symbol in the to-be-recognized audio, and an ending time of the phoneme symbol.

9. The method of claim 8, wherein, The decoding sequence adopts a serial structure and comprises a plurality of sequence nodes; in a case where phonemes corresponding to a plurality of continuous sequence nodes belong to a complete lyric, adjacent nodes before a starting node and adjacent nodes after an ending node in the plurality of continuous sequence nodes are nodes representing silence; adjacent nodes in the plurality of continuous sequence nodes comprise nodes representing silence and being skippable.

10. The method of claim 8, wherein, The obtaining of the phoneme alignment score of the to-be-recognized audio pronouncing according to the candidate phoneme sequence comprises: for each triple, obtaining a probability mean of each international phoneme in a triple time range according to the probability that each frame of the to-be-recognized audio belongs to each international phoneme; the triple time range is determined according to a starting time and an ending time of a phoneme symbol in the triple in the to-be-recognized audio; calculating a posterior alignment probability of the triple according to the probability mean of the phoneme symbol in the triple and a sum of the probability mean of each international phoneme in the triple time range; obtaining a phoneme alignment score corresponding to the candidate phoneme sequence according to the posterior alignment probability of each triple.

11. The method of claim 1, wherein, The obtaining of the to-be-recognized phoneme sequence corresponding to the to-be-recognized audio comprises: performing phoneme recognition on the to-be-recognized audio by using a pre-trained phoneme recognition network to obtain the to-be-recognized phoneme sequence and time stamp information of each phoneme in the to-be-recognized audio; The method further comprises: In a case where the length of the phonetic symbol sequence to be recognized is greater than a set threshold, performing phonetic symbol comparison between the phonetic symbol sequence to be recognized and the first phonetic symbol sequence; In a case where the length of the phonetic symbol sequence to be recognized is less than a set threshold, terminating song recognition.

12. A song identification apparatus characterized by comprising: The apparatus comprises: a phonetic symbol sequence to be recognized acquisition module configured to acquire a phonetic symbol sequence to be recognized corresponding to the audio to be recognized; a phonetic symbol comparison module configured to perform phonetic symbol comparison between the phonetic symbol sequence to be recognized and the first phonetic symbol sequence of each candidate song, and determine a candidate sub-sequence in which the phonetic symbol sequence to be recognized and the first phonetic symbol sequence have the same phonetic symbol and the sequence length meets a requirement; a candidate phonetic symbol sequence acquisition module configured to determine, from the first phonetic symbol sequence containing the candidate sub-sequence, a candidate phonetic symbol sequence that is time-matched with the phonetic symbol sequence to be recognized according to a position of the candidate sub-sequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence; a phonetic symbol alignment module configured to align the audio to be recognized with phonetic symbols in the candidate phonetic symbol sequence, and obtain a phonetic symbol alignment score of the audio to be recognized pronouncing according to the candidate phonetic symbol sequence; a song recognition result generation module configured to determine a song recognition result of the audio to be recognized according to the phonetic symbol alignment score corresponding to each candidate phonetic symbol sequence.

13. The apparatus of claim 12, wherein, The phonetic symbol comparison module is specifically configured to: a division module configured to divide the phonetic symbol sequence to be recognized into a plurality of phonetic symbol sub-sequences according to a set sequence length; an index key matching module configured to match each phonetic symbol sub-sequence with an index key of an inverted index; wherein the index key is a first phonetic symbol sequence; the first phonetic symbol sequence refers to each sub-sequence obtained by dividing a complete phonetic symbol sequence of each candidate song according to the set sequence length; and an index value corresponding to the index key is candidate song information containing the index key; a candidate sub-sequence determination module configured to take, as the candidate sub-sequence, an index key that has the same phonetic symbol as the phonetic symbol sub-sequence.

14. The apparatus of claim 13, wherein, The division module is specifically configured to: start from a starting phonetic symbol of the phonetic symbol sequence to be recognized, use a window to slide on the phonetic symbol sequence to be recognized according to a set step length and a neighboring window overlap value, and divide the phonetic symbol sequence to be recognized into a plurality of phonetic symbol sub-sequences; wherein a length of the window is set as the set sequence length.

15. The apparatus of claim 12, wherein, The candidate phonetic symbol sequence acquisition module is specifically configured to: a playback time scaling ratio determination module configured to acquire a playback time scaling ratio of a candidate sub-sequence on the phonetic symbol sequence to be recognized and the first phonetic symbol sequence; determine a candidate phonetic symbol sequence that is time-matched with the phonetic symbol sequence to be recognized according to the playback time scaling ratio and a position of the candidate sub-sequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence.

16. The apparatus of claim 15, wherein, The playback time scaling ratio determination module is specifically configured to: determine a first playback time length according to a starting time stamp and an ending time stamp of the candidate sub-sequence in the phonetic symbol sequence to be recognized; determine a second playback time length according to a starting time stamp and an ending time stamp of the candidate sub-sequence in the first phonetic symbol sequence; determine the playback time scaling ratio according to the first playback time length and the second playback time length.

17. The apparatus of claim 15, wherein, The candidate phonetic symbol sequence determination module is specifically configured to: acquire a first time interval between the start timestamp of the to-be-recognized phoneme sequence and a start timestamp of a candidate sub-sequence in the to-be-recognized phoneme sequence; acquire a second time interval between the end timestamp of the to-be-recognized phoneme sequence and an end timestamp of a candidate sub-sequence in the to-be-recognized phoneme sequence; determine, according to the start timestamp and the end timestamp of the candidate sub-sequence in the first phoneme sequence, the first time interval, the second time interval, and the playback speed scaling ratio, a timestamp range of the candidate phoneme sequence in the first phoneme sequence; determine, as the candidate phoneme sequence, a phoneme sequence in the first phoneme sequence corresponding to the timestamp range.

18. The apparatus of claim 17, wherein, The candidate phoneme sequence determination module, when determining the timestamp range of the candidate phoneme sequence in the first phoneme sequence, includes: determine, according to the start timestamp of the candidate sub-sequence in the first phoneme sequence, the playback speed scaling ratio, and the first time interval, a start timestamp of the candidate phoneme sequence in the first phoneme sequence; determine, according to the end timestamp of the candidate sub-sequence in the first phoneme sequence, the playback speed scaling ratio, and the second time interval, an end timestamp of the candidate phoneme sequence in the first phoneme sequence.

19. The apparatus of claim 12, wherein, The phoneme alignment module is specifically configured to: for each frame of the to-be-recognized audio, generate, by using a pre-trained phoneme recognition network, a probability that each frame belongs to each international phoneme; construct a decoding sequence according to the candidate phoneme sequence; a node in the decoding sequence corresponds to a phoneme in the candidate phoneme sequence; decode, by using the decoding sequence, according to the probability that each frame belongs to each international phoneme, to obtain a triple sequence in which phonemes in the to-be-recognized audio align with the candidate phoneme sequence; the triple includes a phoneme symbol, a start time of the phoneme symbol in the to-be-recognized audio, and an end time of the phoneme symbol in the to-be-recognized audio.

20. The apparatus of claim 19, wherein, The decoding sequence adopts a serial structure and includes a plurality of sequence nodes; in a case where phonemes corresponding to a plurality of continuous sequence nodes belong to a complete lyric, adjacent front nodes of a start node and adjacent rear nodes of an end node in the plurality of continuous sequence nodes are nodes representing silence; adjacent nodes in the plurality of continuous sequence nodes include skippable nodes representing silence.

21. The apparatus of claim 19, wherein, The phoneme alignment module is specifically configured to: for each triple, according to the probability that each frame of the to-be-recognized audio belongs to each international phoneme, obtain a probability mean of each international phoneme in a triple time range; the triple time range is determined according to a start time and an end time of a phoneme symbol in the triple in the to-be-recognized audio; calculate, according to the probability mean of the phoneme symbol in the triple and a sum of the probability mean of each international phoneme in the triple time range, a posterior alignment probability of the triple; obtain, according to the posterior alignment probability of each triple, a phoneme alignment score corresponding to the candidate phoneme sequence.

22. The apparatus of claim 12, wherein, The to-be-recognized phoneme sequence acquisition module is specifically configured to: perform phoneme recognition on the to-be-recognized audio by using a pre-trained phoneme recognition network to obtain a corresponding to-be-recognized phoneme sequence and timestamp information of each phoneme in the to-be-recognized audio; The device further comprises: in a case where the length of the to-be-identified phoneme sequence is greater than a set threshold, performing phoneme comparison on the to-be-identified phoneme sequence and the first phoneme sequence; and in a case where the length of the to-be-identified phoneme sequence is less than the set threshold, terminating the song identification.

23. A readable storage medium, having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the method of any one of claims 1-11.

24. A computing device, comprising: Comprising: a processor, a memory; the memory is configured to store a computer program; the processor is configured to execute the method of any one of claims 1-11 by invoking the computer program.

Images