Song identification method and device, medium and computing equipment

Abstract

The embodiment of the invention provides a song recognition method and device, a medium and computing equipment, and the method comprises the steps: converting to-be-recognized audio into to-be-recognized phonetic symbol sequences, determining candidate phonetic symbol sequences from the phonetic symbol sequences corresponding to candidate songs through phonetic symbol comparison, and sorting the candidate phonetic symbol sequences in combination with phonetic symbol alignment scores. According to song recognition based on the method, the songs are positioned through phonetic symbol matching, recognition errors caused by the near phonetic words can be effectively avoided, a large amount of mismatching is filtered out in combination with phonetic symbol alignment, and therefore the phonetic symbol retrieval accuracy and the song recognition accuracy are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of song recognition, and more specifically, embodiments of the present invention relate to a song recognition method, apparatus, medium, and computing device. Background Art

[0002] This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] With the popularization of digital music and the rapid development of Internet technology, song recognition technology is widely used in user life scenarios. At present, traditional song recognition methods mainly rely on the extraction and comparison of audio features, and identify songs by matching the melody, rhythm, lyrics and other features of audio clips. However, this method is easily affected by the accuracy of lyrics recognition when identifying audio clips containing lyrics. For example, for the recognition of near-phonetic words or dialects and songs in small languages, the error rate of lyrics recognition will increase significantly, resulting in a decrease in the accuracy of song recognition.

[0004] Therefore, how to further improve the accuracy of song recognition and reduce the cost of supporting new languages ​​has become an urgent problem to be solved in the current field of song recognition technology. Summary of the invention

[0005] In view of this, in order to solve the above problems, the embodiments of the present invention are expected to provide a song identification method, apparatus, medium and computing device.

[0006] In a first aspect of the embodiments of the present invention, a song recognition method is provided, the method comprising:

[0007] Obtain the phonetic symbol sequence to be recognized corresponding to the audio to be recognized;

[0008] Compare the to-be-recognized phonetic symbol sequence with the first phonetic symbol sequence of each candidate song to determine a candidate subsequence whose to-be-recognized phonetic symbol sequence has the same phonetic symbol as the first phonetic symbol sequence and whose sequence length meets the requirement;

[0009] According to the positions of the candidate subsequences in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence, determining, from the first phonetic symbol sequence containing the candidate subsequences, a candidate phonetic symbol sequence that time-matches the to-be-recognized phonetic symbol sequence;

[0010] Aligning the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence to obtain a phonetic symbol alignment score of the audio to be recognized according to the candidate phonetic symbol sequence;

[0011] The song recognition result of the audio to be recognized is determined according to the phonetic symbol alignment scores corresponding to each candidate phonetic symbol sequence.

[0012] Optionally, comparing the to-be-recognized phonetic symbol sequence with the first phonetic symbol sequence of each candidate song includes:

[0013] According to the set sequence length, the to-be-recognized phonetic symbol sequence is divided into a plurality of to-be-recognized subsequences;

[0014] Match each subsequence to be identified with an index key of the inverted index; wherein the index key is a first phonetic symbol sequence; the first phonetic symbol sequence refers to each subsequence obtained by dividing the complete phonetic symbol sequence of each candidate song according to the set sequence length; the index value corresponding to the index key is the candidate song information containing the index key;

[0015] The index key that is the same as the phonetic symbol of the subsequence to be identified is used as the candidate subsequence.

[0016] Optionally, dividing the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized subsequences according to the set sequence length includes:

[0017] Starting from the starting phonetic symbol of the phonetic symbol sequence to be recognized, a window is used to slide on the phonetic symbol sequence to be recognized according to a set step size and an overlap value of adjacent windows, so as to divide the phonetic symbol sequence to be recognized into a plurality of subsequences to be recognized; wherein the length of the window is set to the set sequence length.

[0018] Optionally, the determining, from the first phonetic symbol sequence including the candidate subsequences, a candidate phonetic symbol sequence that time matches the phonetic symbol sequence to be recognized comprises:

[0019] Obtaining a playback speed scaling ratio of the candidate subsequence on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence;

[0020] According to the playback speed ratio and the position of the candidate subsequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence, a candidate phonetic symbol sequence that time matches the to-be-recognized phonetic symbol sequence is determined.

[0021] Optionally, obtaining a playback speed scaling ratio of the candidate subsequence on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence includes:

[0022] Determine a first playback duration according to a start timestamp and an end timestamp of the candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0023] Determining a second playback duration according to a start timestamp and an end timestamp of the candidate subsequence in the first phonetic symbol sequence;

[0024] The playback speed scaling ratio is determined according to the first playback duration and the second playback duration.

[0025] Optionally, the determining, from the first phonetic symbol sequence including the candidate subsequences, a candidate phonetic symbol sequence that time matches the phonetic symbol sequence to be recognized comprises:

[0026] Acquire a first time interval between a start timestamp of the to-be-recognized phonetic symbol sequence and a start timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0027] Acquire a second time interval between an end timestamp of the to-be-recognized phonetic symbol sequence and an end timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0028] According to the start timestamp and the end timestamp of the candidate subsequence in the first phonetic symbol sequence, using the first time interval, the second time interval and the playback speed scaling ratio, determine the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence;

[0029] The phonetic symbol sequence corresponding to the timestamp range in the first phonetic symbol sequence is determined as a candidate phonetic symbol sequence.

[0030] Optionally, determining a timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence includes:

[0031] Determine a start timestamp of a candidate phonetic symbol sequence in the first phonetic symbol sequence according to the start timestamp of the candidate subsequence in the first phonetic symbol sequence, the playback time scaling ratio, and a first time interval;

[0032] The end timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence is determined according to the end timestamp of the candidate subsequence in the first phonetic symbol sequence, the playback time scaling ratio, and the second time interval.

[0033] Optionally, aligning the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence includes:

[0034] For each frame of the audio to be recognized, using a pre-trained phonetic symbol recognition network to generate a probability that each frame belongs to each international phonetic symbol;

[0035] According to the candidate phonetic symbol sequence, a decoding sequence is constructed; the nodes in the decoding sequence correspond to the phonetic symbols in the candidate phonetic symbol sequence;

[0036] According to the probability that each frame belongs to each international phonetic symbol, the decoding sequence is used to decode to obtain a triple sequence of the phonetic symbols in the audio to be recognized that are aligned with the candidate phonetic symbol sequence, wherein the triplet includes the phonetic symbol symbol, the start time and the end time of the phonetic symbol in the audio to be recognized.

[0037] Optionally, the decoding sequence adopts a serial structure, including multiple sequence nodes; when the phonetic symbols corresponding to the multiple consecutive sequence nodes belong to a complete lyric,

[0038] The adjacent previous node of the start node and the adjacent subsequent node of the end node in the plurality of continuous sequence nodes are nodes indicating silence; and the adjacent nodes of the plurality of continuous sequence nodes include skippable nodes indicating silence.

[0039] Optionally, obtaining a phonetic symbol alignment score of the audio to be recognized according to the candidate phonetic symbol sequence includes:

[0040] For each triplet, according to the probability that each frame in the audio to be recognized belongs to each international phonetic symbol, obtain the probability mean of each international phonetic symbol within the triplet time range; the triplet time range is determined according to the start time and end time of the phonetic symbol in the triplet in the audio to be recognized;

[0041] Calculating the posterior alignment probability of the triplet according to the probability mean of the phonetic symbols in the triplet and the sum of the probability mean of each international phonetic symbol within the triplet time range;

[0042] According to the posterior alignment probability of each triplet, a phonetic symbol alignment score corresponding to the candidate phonetic symbol sequence is obtained.

[0043] Optionally, the step of obtaining a sequence of phonetic symbols to be recognized corresponding to the audio to be recognized includes:

[0044] Performing phonetic symbol recognition on the audio to be recognized using a pre-trained phonetic symbol recognition network to obtain a corresponding phonetic symbol sequence to be recognized and timestamp information of each phonetic symbol in the audio to be recognized;

[0045] The method further comprises:

[0046] When the length of the to-be-recognized phonetic symbol sequence is greater than a set threshold, performing phonetic symbol comparison between the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence;

[0047] When the length of the to-be-recognized phonetic symbol sequence is less than a set threshold, song recognition is terminated.

[0048] In a second aspect of the embodiments of the present invention, a song recognition device is provided, the device comprising:

[0049] A module for obtaining a sequence of phonetic symbols to be recognized, used to obtain a sequence of phonetic symbols to be recognized corresponding to the audio to be recognized;

[0050] A phonetic symbol comparison module is used to compare the phonetic symbol sequence to be identified with the first phonetic symbol sequence of each candidate song, and determine a candidate subsequence whose phonetic symbols in the phonetic symbol sequence to be identified are the same as those in the first phonetic symbol sequence and whose sequence length meets the requirements;

[0051] A candidate phonetic symbol sequence acquisition module is used to determine a candidate phonetic symbol sequence that matches the phonetic symbol sequence to be identified in time from the first phonetic symbol sequence containing the candidate subsequence according to the position of the candidate subsequence in the phonetic symbol sequence to be identified and the first phonetic symbol sequence;

[0052] A phonetic symbol alignment module, used to align the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence to obtain a phonetic symbol alignment score of the audio to be recognized according to the pronunciation of the candidate phonetic symbol sequence;

[0053] The song recognition result generating module is used to determine the song recognition result of the audio to be recognized according to the phonetic symbol alignment scores corresponding to each candidate phonetic symbol sequence.

[0054] Optionally, the phonetic symbol comparison module is specifically used for:

[0055] A division module, used for dividing the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized subsequences according to a set sequence length;

[0056] An index key matching module, used for matching each subsequence to be identified with an index key of an inverted index; wherein the index key is a first phonetic symbol sequence; the first phonetic symbol sequence refers to each subsequence obtained by dividing the complete phonetic symbol sequence of each candidate song according to the set sequence length; the index value corresponding to the index key is the candidate song information containing the index key;

[0057] The candidate subsequence determination module is used to use the index key that is the same as the phonetic symbol of the subsequence to be identified as the candidate subsequence.

[0058] Optionally, the division module is specifically used for:

[0059] Starting from the starting phonetic symbol of the phonetic symbol sequence to be recognized, a window is used to slide on the phonetic symbol sequence to be recognized according to a set step size and an overlap value of adjacent windows, so as to divide the phonetic symbol sequence to be recognized into a plurality of subsequences to be recognized; wherein the length of the window is set to the set sequence length.

[0060] Optionally, the candidate phonetic symbol sequence acquisition module is specifically used to:

[0061] A scaling ratio determination module, used to obtain a scaling ratio of the playback speed of the candidate subsequence on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence;

[0062] According to the playback speed ratio and the position of the candidate subsequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence, a candidate phonetic symbol sequence that time matches the to-be-recognized phonetic symbol sequence is determined.

[0063] Optionally, the scaling ratio determination module is specifically used to:

[0064] Determine a first playback duration according to a start timestamp and an end timestamp of the candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0065] Determining a second playback duration according to a start timestamp and an end timestamp of the candidate subsequence in the first phonetic symbol sequence;

[0066] The playback speed scaling ratio is determined according to the first playback duration and the second playback duration.

[0067] Optionally, the candidate phonetic symbol sequence determination module is specifically used to:

[0068] Acquire a first time interval between a start timestamp of the to-be-recognized phonetic symbol sequence and a start timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0069] Acquire a second time interval between an end timestamp of the to-be-recognized phonetic symbol sequence and an end timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0070] According to the start timestamp and the end timestamp of the candidate subsequence in the first phonetic symbol sequence, using the first time interval, the second time interval and the playback speed scaling ratio, determine the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence;

[0071] The phonetic symbol sequence corresponding to the timestamp range in the first phonetic symbol sequence is determined as a candidate phonetic symbol sequence.

[0072] Optionally, when the candidate phonetic symbol sequence determining module is used to determine the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence, it includes:

[0073] Determine a start timestamp of a candidate phonetic symbol sequence in the first phonetic symbol sequence according to the start timestamp of the candidate subsequence in the first phonetic symbol sequence, the playback time scaling ratio, and a first time interval;

[0074] The end timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence is determined according to the end timestamp of the candidate subsequence in the first phonetic symbol sequence, the playback time scaling ratio, and the second time interval.

[0075] Optionally, the phonetic symbol alignment module is specifically used for:

[0076] For each frame of the audio to be recognized, using a pre-trained phonetic symbol recognition network to generate a probability that each frame belongs to each international phonetic symbol;

[0077] According to the candidate phonetic symbol sequence, a decoding sequence is constructed; the nodes in the decoding sequence correspond to the phonetic symbols in the candidate phonetic symbol sequence;

[0078] According to the probability that each frame belongs to each international phonetic symbol, the decoding sequence is used to decode to obtain a triple sequence of the phonetic symbols in the audio to be recognized that are aligned with the candidate phonetic symbol sequence, wherein the triplet includes the phonetic symbol symbol, the start time and the end time of the phonetic symbol in the audio to be recognized.

[0079] Optionally, the decoding sequence adopts a serial structure, including multiple sequence nodes; when the phonetic symbols corresponding to the multiple consecutive sequence nodes belong to a complete lyric,

[0080] The adjacent previous node of the start node and the adjacent subsequent node of the end node in the plurality of continuous sequence nodes are nodes indicating silence; and the adjacent nodes of the plurality of continuous sequence nodes include skippable nodes indicating silence.

[0081] Optionally, the phonetic symbol alignment module is specifically used for:

[0082] For each triplet, according to the probability that each frame in the audio to be recognized belongs to each international phonetic symbol, obtain the probability mean of each international phonetic symbol within the triplet time range; the triplet time range is determined according to the start time and end time of the phonetic symbol in the triplet in the audio to be recognized;

[0083] Calculating the posterior alignment probability of the triplet according to the mean probability of the phonetic symbols in the triplet and the sum of the mean probability of each international phonetic symbol within the triplet time range;

[0084] According to the posterior alignment probability of each triplet, a phonetic symbol alignment score corresponding to the candidate phonetic symbol sequence is obtained.

[0085] Optionally, the module for acquiring the to-be-recognized phonetic symbol sequence is specifically used for:

[0086] Performing phonetic symbol recognition on the audio to be recognized using a pre-trained phonetic symbol recognition network to obtain a corresponding phonetic symbol sequence to be recognized and timestamp information of each phonetic symbol in the audio to be recognized;

[0087] The device also includes:

[0088] When the length of the to-be-recognized phonetic symbol sequence is greater than a set threshold, the to-be-recognized phonetic symbol sequence is compared with the first phonetic symbol sequence; when the length of the to-be-recognized phonetic symbol sequence is less than the set threshold, song recognition is terminated.

[0089] In a third aspect of the embodiments of the present invention, a readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned song recognition method is implemented.

[0090] In a fourth aspect of the embodiment of the present invention, a computing device is provided, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the above-mentioned song recognition method by calling the computer program.

[0091] The song recognition method provided in the embodiment of the present invention introduces the International Phonetic Alphabet, a cross-language standardized speech representation form. It converts the audio to be recognized into a phonetic symbol sequence to be recognized, and then determines the candidate phonetic symbol sequence from the phonetic symbol sequences corresponding to each candidate song through phonetic symbol comparison, and then sorts the candidate phonetic symbol sequences in combination with the phonetic symbol alignment score to obtain the final song recognition result. Song recognition based on this method locates songs through phonetic symbol matching, which can effectively avoid recognition errors caused by near-phonetic words, and filters out a large number of mismatches in combination with phonetic symbol alignment, thereby improving the accuracy of phonetic symbol retrieval and song recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0093] Figure 1 A flowchart of the steps of a song recognition method according to an embodiment of the present invention is schematically shown;

[0094] Figure 2A A flowchart of phonetic symbol comparison based on an inverted index according to an embodiment of the present invention is schematically shown;

[0095] Figure 2B A schematic diagram showing an example of a process of dividing a to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized subsequences by means of a sliding window according to an embodiment of the present invention;

[0096] Figure 3A Schematically shows an example flowchart of determining a playback speed scaling ratio according to an embodiment of the present invention;

[0097] Figure 3BA flowchart of steps for determining a candidate phonetic symbol sequence according to an embodiment of the present invention is schematically shown;

[0098] Figure 3C An example diagram of determining a timestamp range based on a phonetic symbol sequence according to an embodiment of the present invention is schematically shown;

[0099] Figure 4A A flowchart of phonetic symbol alignment steps according to an embodiment of the present invention is schematically shown;

[0100] Figure 4B The overall training process of a phonetic symbol recognition network according to an embodiment of the present invention is schematically shown;

[0101] Figure 4C A schematic diagram of a decoding sequence according to an embodiment of the present invention is schematically shown;

[0102] Figure 4D Schematically shows another decoding sequence schematic diagram according to an embodiment of the present invention;

[0103] Figure 5 A block diagram of a song recognition device according to an embodiment of the present invention is schematically shown;

[0104] Figure 6 A schematic diagram of a medium according to an embodiment of the present invention is schematically shown;

[0105] Figure 7 A schematic diagram of a computing device according to an embodiment of the present invention is schematically shown.

[0106] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0107] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0108] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0109] According to an embodiment of the present invention, a song recognition method, apparatus, medium and computing device are proposed.

[0110] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0111] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0112] With the popularity of digital music and the rapid development of Internet technology, song recognition technology has gradually become a hot research field. Whether it is song search in music playback software, humming recognition, or background music recognition in public places, song recognition technology plays a vital role. However, although the existing song recognition technology has made significant progress in some aspects, it still faces many challenges.

[0113] Traditional song recognition methods mainly rely on the extraction and comparison of audio features, and identify songs by matching the melody, rhythm and other features of audio clips and lyrics recognition and retrieval. However, this method is easily affected by the accuracy of lyrics recognition. When encountering near-sounding words or dialects or songs in small languages, the error rate of lyrics recognition will increase significantly, resulting in a decrease in the accuracy of song recognition. For the recognition of songs in new languages, in order to improve the speech recognition ability of songs in new languages, it is usually necessary to prepare a large amount of audio data of songs in new languages ​​for model training, which not only requires a lot of time and resource investment, but also may face the difficulty of data acquisition and processing. The cost of supporting the recognition of songs in new languages ​​is relatively high.

[0114] Therefore, how to further improve the accuracy of song recognition, reduce the cost of supporting new languages, and simplify the maintenance of song libraries has become an urgent problem to be solved in the current field of song recognition technology.

[0115] In view of this, the present invention proposes a song recognition method based on phonetic symbol matching, which locates possible matching candidate song segments from candidate songs through international phonetic symbol comparison, and performs phonetic symbol alignment on the candidate song segments and the audio to be recognized to filter out mismatches, and finally obtains the song recognition result according to the phonetic symbol alignment score, which can effectively avoid recognition errors caused by near-phonetic words, thereby improving the accuracy of phonetic symbol retrieval and song recognition. The present invention bypasses language restrictions, models at the level of speech phonemes, uses international phonetic symbols as training targets, and combines pronunciation evaluation technology to perform phonetic symbol alignment scoring to improve recognition accuracy, forming a song recognition solution based on audio phonetic symbol recognition and retrieval technology, which has the characteristics of high universality and independence of song language, is suitable for recognition scenarios of songs in various languages, and can be quickly expanded to the recognition of songs in small languages ​​or dialects.

[0116] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0117] Exemplary Methods

[0118] Combine the following Figure 1-Figure 7 To describe a song recognition method according to an exemplary embodiment of the present invention,

[0119] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0120] refer to Figure 1 A flowchart of the steps of a song recognition method according to an embodiment of the present invention is schematically shown. The phonetic symbols in each phonetic symbol sequence involved in the song recognition method provided in this embodiment are all international phonetic symbols.

[0121] At least the following steps may be included:

[0122] S101, obtaining a sequence of phonetic symbols to be recognized corresponding to the audio to be recognized;

[0123] The audio to be recognized refers to the audio data whose song name or lyrics the user hopes to be able to recognize and determine, which may include but is not limited to the audio recorded by the user through a microphone or loaded from a file, and may include sounds such as singing, humming, and playing musical instruments.

[0124] The phonetic symbol sequence to be recognized is an ordered result obtained by converting the sound in the audio to be recognized into phonetic symbols, and the phonetic symbols are used to represent the pronunciation characteristics in the audio. The phonetic symbols in each phonetic symbol sequence involved in the song recognition method provided in this embodiment are all International Phonetic Alphabet (IPA), a phoneme notation designed by the International Phonetic Association for accurately recording and describing the pronunciation of human languages, including vowels, consonants, diphthongs, semi-vowels and other phoneme symbols, covering the speech characteristics of multiple languages.

[0125] With regard to obtaining the corresponding sequence of phonetic symbols to be recognized from the audio to be recognized, it can be achieved through the vocal tract model and speech knowledge, that is, comparing the frequency domain or time domain characteristics of the speech signal with the known speech primitives (i.e., the International Phonetic Alphabet) to recognize different speech. In specific implementation, the speech signal of the audio to be recognized can be segmented and labeled, and then similar phonetic symbol labels are given according to the acoustic characteristics of each segment, and then a speech primitive grid is obtained according to the sequence of phonetic symbol labels, and finally the sequence of phonetic symbols to be recognized is determined by combining the grammatical semantic information of the dictionary and the sentence.

[0126] Alternatively, the audio to be recognized may be converted into a phonetic symbol sequence by using a neural network or machine learning method, that is, a pre-trained phonetic symbol recognition network may be used to perform phonetic symbol recognition on the audio to be recognized, and obtain the corresponding phonetic symbol sequence to be recognized and the timestamp information of each phonetic symbol in the audio to be recognized. The training samples of the phonetic symbol recognition network include a large amount of audio data and corresponding phonetic symbol sequence labels, and the phonetic symbol sequence labels corresponding to each audio data may be manually annotated by a phonetics expert or a trained person to represent the accurate phonetic symbol of the pronunciation in the audio data.

[0127] S102, performing phonetic symbol comparison between the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence of each candidate song, and determining a candidate subsequence in which the to-be-recognized phonetic symbol sequence has the same phonetic symbols as the first phonetic symbol sequence and the sequence length meets the requirement;

[0128] Regarding candidate songs, in the song recognition scenario, all songs in the song library are considered candidate songs, which are pre-stored musical works with known information such as song title, singer, lyrics, etc.

[0129] The first phonetic symbol sequence represents the phonetic symbol sequence corresponding to the lyrics of the candidate song, which can be obtained by converting the lyrics of the candidate song into corresponding phonetic symbols word by word, and represents the pronunciation of the lyrics of the song. The first phonetic symbol sequence can include two forms of expression: the first phonetic symbol sequence can represent the complete phonetic symbol sequence corresponding to the candidate song, that is, the lyrics of the entire song are converted into a continuous phonetic symbol sequence; or, the first phonetic symbol sequence can also include multiple first subsequences obtained by dividing the complete phonetic symbol sequence according to a set sequence length, each of which belongs to a segment of the complete phonetic symbol sequence, and together constitute the phonetic symbol representation of the entire candidate song.

[0130] Phonetic symbol comparison refers to comparing the phonetic symbols contained in each of the two phonetic symbol sequences one by one to find the same phonetic symbol parts between the two. In this embodiment, the phonetic symbol comparison is used to find candidate subsequences that exist in both the first phonetic symbol sequence and the phonetic symbol sequence to be identified and whose sequence length meets the requirements. Among them, the sequence length can be used to represent the number of phonetic symbols contained in the candidate subsequence, or, based on the situation that there is a phonetic symbol including multiple characters, the sequence length can also be used to represent the number of single characters contained in the candidate subsequence, and this application does not limit this. The sequence length meets the requirements, that is, the sequence length is greater than or equal to the preset length threshold, that is, the number of international phonetic symbols / single characters included in the candidate subsequence is greater than or equal to the set length threshold. For each candidate song, if the candidate subsequence exists in the candidate song, the candidate song is used as a candidate recognition result, such as being added to the candidate recognition result set.

[0131] Regarding obtaining candidate subsequences through phonetic symbol comparison, it can be implemented by using any applicable method such as a dynamic programming algorithm such as the longest common subsequence algorithm, a string matching algorithm such as a brute force matching or a KMP algorithm, a hash method, and a machine learning algorithm. In order to improve the efficiency of phonetic symbol comparison, the complete phonetic symbol sequence corresponding to each candidate song can be divided according to the set sequence length to obtain multiple first subsequences. Based on the multiple first subsequences corresponding to each candidate song, an inverted index is constructed, with the first subsequence as the index key and the candidate song containing the first subsequence as the index value, so as to facilitate rapid search for candidate subsequences. For the specific implementation process, please refer to the relevant embodiment description below.

[0132] For example, suppose the sequence of phonetic symbols to be recognized is The first phonetic symbol sequence of a candidate song A is Then, after comparing the phonetic symbols, the subsequence that exists in both the phonetic symbol sequence to be identified and the first phonetic symbol sequence of A is determined to be Taking the example that the sequence length represents the number of phonetic symbols contained in a subsequence, the sequence length of the subsequence is 16, which meets the length requirement, so the subsequence is taken as a candidate subsequence.

[0133] S103, determining a candidate phonetic symbol sequence that time-matches the phonetic symbol sequence to be recognized from the first phonetic symbol sequence containing the candidate subsequence according to the position of the candidate subsequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence;

[0134] Time matching is used to indicate that the candidate phonetic symbol sequence selected from the first phonetic symbol sequence is synchronous or nearly synchronous with the phonetic symbol sequence to be recognized in terms of pronunciation time. After determining the candidate subsequence, this embodiment can compare the time features of the candidate subsequence in the first International Phonetic Symbol sequence and the phonetic symbol sequence to be recognized, for example, by calculating the duration, pronunciation rate or other relevant time parameters of the candidate subsequence in two different sequences.

[0135] According to the time feature, the timestamp or frame number of the candidate subsequence in the to-be-recognized phonetic symbol sequence can be aligned with the time information of the candidate subsequence in the first phonetic symbol sequence, and then according to the position of the candidate subsequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence, the position of the candidate subsequence in the first phonetic symbol sequence is used as the starting point to expand forward and backward to determine the candidate phonetic symbol sequence that matches the playback duration of the to-be-recognized phonetic symbol sequence.

[0136] S104, aligning the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence to obtain a phonetic symbol alignment score of the audio to be recognized when pronounced according to the candidate phonetic symbol sequence;

[0137] Phonetic symbol alignment refers to the process of matching the speech signal in the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence. It involves phonetic symbol recognition technology, such as dynamic time warping DTW, hidden Markov model or deep learning models such as recurrent neural network RNN, long short-term memory network LSTM, Transformer, etc., to find the best match between the audio signal and the phonetic symbol sequence.

[0138] The phonetic symbol alignment score is a quantitative indicator used to measure the quality of alignment between the audio to be recognized and the candidate phonetic symbol sequence. The phonetic symbol alignment score is calculated based on the matching degree, error rate or other related indicators in the alignment process, reflecting the degree of pronunciation matching between the audio signal and the candidate phonetic symbol sequence. The phonetic symbol alignment score is positively correlated with the quality of alignment, that is, the higher the phonetic symbol alignment score, the higher the alignment quality, that is, the higher the matching degree between the audio to be recognized and the candidate phonetic symbol sequence.

[0139] Regarding phonetic symbol alignment, the audio frame of the audio to be recognized can be matched with the phonetic symbols in the candidate phonetic symbol sequence through alignment algorithms such as dynamic time warping DTW, hidden Markov model HMM, deep learning model, etc. The goal of phonetic symbol alignment is to find an optimal matching method so that the overall difference between the audio frame of the audio to be recognized and the candidate phonetic symbol sequence is minimized. During the alignment process, a time range is determined for each phonetic symbol, indicating the start and end time of the phonetic symbol in the audio to be recognized. The time range is obtained based on the optimal matching result calculated by the alignment algorithm.

[0140] S105, determining a song recognition result of the audio to be recognized according to the phonetic symbol alignment scores corresponding to each candidate phonetic symbol sequence.

[0141] The song recognition result is the most likely song name or identifier determined based on the comparison between the audio to be recognized and the candidate phonetic symbol sequence. This result is the final output of song recognition and is used to tell the user which song the audio to be recognized belongs to.

[0142] In this step, the candidate phonetic symbol sequences can be sorted according to the phonetic symbol alignment scores, that is, all candidate phonetic symbol sequences are sorted from high to low according to their phonetic symbol alignment scores. The higher the score of the candidate phonetic symbol sequence, the higher the degree of match with the audio to be recognized. The candidate phonetic symbol sequence with the highest score or the top N candidate phonetic symbol sequences in the score ranking are selected, and the song information to which the selected candidate phonetic symbol sequence belongs is obtained to generate the song recognition result. For example, the song title and the start timestamp and end timestamp of the candidate phonetic symbol sequence in the candidate song to which it belongs are obtained, and the song recognition result interface is generated to display the lyrics corresponding to the candidate phonetic symbol sequence.

[0143] Alternatively, in order to ensure the accuracy of song recognition, after sorting the candidate phonetic symbol sequences according to the phonetic symbol alignment scores, one or more candidate phonetic symbol sequences with the top N scores and greater than the score threshold can be selected, and the corresponding candidate song information can be obtained to generate a song recognition result.

[0144] In the disclosed embodiment, the International Phonetic Alphabet, a standardized speech representation form across languages, is used to achieve accurate recognition of song audio. First, the phonetic symbol sequence to be recognized of the audio to be recognized is obtained, and then it is compared with the first phonetic symbol sequence of the candidate song to determine the matching candidate subsequence, and then the candidate phonetic symbol sequence with time matching is determined from the first phonetic symbol sequence where the candidate subsequence is located. In combination with phonetic symbol alignment, a large number of mismatches are further filtered out, thereby improving the accuracy of phonetic symbol retrieval and final recognition. The candidate phonetic symbol sequences are sorted by phonetic symbol alignment scores to obtain the final song recognition result. The song recognition process does not directly rely on the language model for lyrics recognition, but locates the song through phonetic symbol matching, so it can effectively avoid recognition errors caused by near-sounding words, and significantly improve the accuracy of song recognition. In addition, the method is modeled based on the International Phonetic Alphabet. When expanding songs in a new language, it only needs to establish the corresponding relationship between the new language and the International Phonetic Alphabet, and there is no need to prepare a large amount of audio data of the new language for retraining, so that the method can quickly adapt to and support the recognition of songs in the new language, reducing the cost and time of new language support.

[0145] In some embodiments, in order to further optimize the accuracy and efficiency of audio recognition song matching, especially when processing large-scale music libraries, the traditional linear scanning comparison method is relatively inefficient. Therefore, this embodiment proposes a phonetic symbol comparison method based on inverted index, which can speed up the comparison process while ensuring the accuracy of the comparison results, thereby improving the comparison efficiency. Figure 2A The flowchart of phonetic symbol comparison based on inverted index is shown as an example. This method can realize the phonetic symbol comparison between the phonetic symbol sequence to be recognized and the first phonetic symbol sequence of each candidate song through the following steps:

[0146] S201, dividing the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized subsequences according to a set sequence length;

[0147] The set sequence length can be used to indicate the number of phonetic symbols contained in each subsequence to be identified, or, based on the situation that a phonetic symbol includes multiple characters, the sequence length can also be used to indicate the number of single characters contained in each subsequence to be identified. The value and meaning of the set sequence length can be determined according to the needs of the actual application and the expected comparison accuracy. Based on the present invention, song recognition is performed at the phoneme level. Therefore, the relevant description about the set sequence length is explained by taking the number of phonetic symbols contained in the sequence as an example. It can be understood that the relevant description is only for illustrative purposes and does not constitute a limitation on the scope of the present application.

[0148] Regarding dividing the sequence of phonetic symbols to be recognized into multiple subsequences to be recognized, it can be achieved by means of a sliding window. Among them, the sliding window is a virtual frame or tool used to move step by step on the sequence of phonetic symbols to be recognized, so as to extract multiple subsequences. The starting position of the window starts from the starting phonetic symbol of the sequence of phonetic symbols to be recognized, and then moves backward step by step according to the set sliding step. The sliding step determines the distance the window moves each time. When the sliding step is equal to the set sequence length, each subsequence to be recognized will not overlap; when the sliding step is less than the set sequence length, adjacent subsequences to be recognized will partially overlap.

[0149] In this embodiment, starting from the starting phonetic symbol of the phonetic symbol sequence to be recognized, a sliding window can be used to slide on the phonetic symbol sequence to be recognized according to a set step size and an adjacent window overlap value to divide the phonetic symbol sequence to be recognized into multiple subsequences to be recognized; wherein the length of the sliding window is set to the set sequence length.

[0150] The adjacent window overlap value is an optional parameter that specifies the number of overlapping phonetic symbols between adjacent subsequences. When the sliding step size is less than the set sequence length, the adjacent window overlap value determines the degree of phonetic symbol overlap.

[0151] See also Figure 2B An exemplary diagram showing a process of dividing a to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized subsequences by means of a sliding window, wherein the to-be-recognized phonetic symbol sequence For example, it contains 19 phonetic symbols. According to the set sequence length, it contains 15 phonetic symbols, that is, the window length is 15 phonetic symbols and the sliding step is 1. It can be divided into 5 subsequences to be identified, namely, [1-15], [2-16], [3-17], [4-18], and [5-19] of phonetic symbol positions.

[0152] S202, matching each subsequence to be identified with an index key of the inverted index; wherein the index key is each subsequence contained in the first phonetic symbol sequence, which is obtained by dividing the complete phonetic symbol sequence of each candidate song according to the set sequence length; the index value corresponding to the index key is the candidate song information containing the index key;

[0153] An inverted index is an index structure used for full-text search, which records the position of a word (or phrase) in a document and is suitable for quickly finding all documents containing a specific word or phrase. Unlike the traditional forward index (which records which words a document contains), the inverted index starts from a word and records all documents containing the word and their position information. In this embodiment, the inverted index is used to record all candidate songs containing a phonetic symbol subsequence and the position information of the phonetic symbol subsequence in each candidate song.

[0154] In an inverted index, an index key refers to a word or phrase used for retrieval. In this embodiment, for all candidate songs, after the lyrics of each candidate song are converted into international phonetic symbols to obtain a complete phonetic symbol sequence, the complete phonetic symbol sequence is divided according to the set sequence length described in step S201 to obtain multiple subsequences, and the multiple subsequences corresponding to all candidate songs are deduplicated to form index keys respectively. The manner of dividing the complete phonetic symbol sequence of the candidate song into each subsequence is the same as the manner of dividing the to-be-recognized phonetic symbol sequence into multiple to-be-recognized subsequences in the aforementioned step S201.

[0155] For example, candidate song A1 is divided into M1 subsequences according to the set sequence length, which contains 15 phonetic symbols. Candidate song A2 is divided into M2 subsequences... Candidate song An is divided into Mn subsequences. Then, after removing duplicates from (M1+M2+...+Mn) subsequences, each subsequence forms an index key. Assuming that there are (M1+M2+...+Mn-100) subsequences after removing duplicates, there are corresponding (M1+M2+...+Mn-100) index keys.

[0156] The index value corresponding to the index key refers to a list or set of documents containing the index key, and in the present embodiment, the index value corresponding to the index key refers to the candidate song information of the subsequence represented by the index key, such as the song ID, and the position information of the subsequence in the candidate song, such as represented by the start timestamp and the end timestamp.

[0157] Continuing with the aforementioned candidate song A1 as an example, the M1 subsequences obtained by dividing the candidate song A1 form M1 index keys, and the index value corresponding to each index key includes the candidate song A1 and the position of the subsequence represented by the index key in the candidate song A1. For example, the first subsequence obtained by dividing the candidate song A1 forms index key 1, and the value corresponding to index key 1 is "candidate song A1: [00:07, 00:33]", and the second subsequence forms index key 2, and the value corresponding to index key 2 is "candidate song A1: [00:40, 01:01]". If the subsequence is successfully deduplicated, it means that there are multiple candidate songs that contain the subsequence, so the index value corresponding to the index key formed by the subsequence includes the position information of multiple candidate songs and the subsequences. For example, the value corresponding to a certain index key n1 is "candidate song A3: [01:30, 01:55], candidate song A9: [02:15, 02:30]".

[0158] In the process of matching each subsequence to be identified with the index key of the inverted index, since the phonetic symbol sequence to be identified is divided into multiple independent subsequences for matching, in order to improve the matching efficiency, the matching of the subsequences to be identified with the index keys can be processed in parallel.

[0159] Regarding the inverted index, a phonetic symbol song library including the data structure of the inverted index can be established in advance, that is, the complete phonetic symbol sequence corresponding to each candidate song can be divided into multiple first subsequences according to the set sequence length, and the first subsequences can be deduplicated;

[0160] The first subsequence is used as the key of the inverted index, and the song information containing the first subsequence is used as the value corresponding to the key to construct a phonetic song library.

[0161] S203: Use the index key that is the same as the phonetic symbol of the subsequence to be identified as the candidate subsequence.

[0162] This embodiment is used to compare the phonetic symbol sequence to be identified with the first phonetic symbol sequence of each candidate song, and determine the candidate subsequence whose phonetic symbol sequence to be identified is the same as the phonetic symbol in the first phonetic symbol sequence and whose sequence length meets the requirements, by dividing the phonetic symbol sequence to be identified into multiple subsequences to be identified, and comparing them with each index key in the inverted index. If any subsequence to be identified is the same as the phonetic symbol of any index key, then the subsequence represented by the index key is the candidate subsequence whose phonetic symbol sequence to be identified is the same as the phonetic symbol in the first phonetic symbol sequence and whose sequence length meets the requirements.

[0163] In the disclosed embodiment, by dividing the to-be-recognized phonetic symbol sequence into multiple shorter subsequences and matching them with the index keys in the inverted index, the local matching feature is utilized, which can significantly reduce the number of phonetic symbols that need to be compared, making the matching process more efficient, thereby quickly determining the candidate subsequence and multiple candidate songs containing the candidate subsequence as candidate recognition results, thereby improving the efficiency of song recognition.

[0164] In some embodiments, after determining that the phonetic symbol sequence to be recognized is the same as the phonetic symbol in the first phonetic symbol sequence and the candidate subsequence whose sequence length meets the requirements, based on the fact that the candidate subsequence belongs to a segment in the phonetic symbol sequence to be recognized, based on local segment matching, through the local to global approach, a candidate phonetic symbol sequence that matches the entire phonetic symbol sequence to be recognized in time can be located from the first phonetic symbol sequence containing the candidate subsequence, which can be achieved through the following steps a1-a2:

[0165] a1, obtaining the playback speed scaling ratio of the candidate subsequence on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence;

[0166] The playback speed scaling ratio is used to reflect the difference in singing speed of the candidate subsequence in two different sequences. The singing speed of the candidate subsequences with the same phonetic symbols under different audio sources is different. For example, the timestamp information of the candidate subsequence in the phonetic symbol sequence to be recognized is [01:45-02:00], that is, it is sung for 15 seconds, while the timestamp information in the first phonetic symbol sequence corresponding to the candidate song is [01:27-01:57], that is, it is sung for 30 seconds. In the case of local segment matching, in order to ensure the accuracy and reliability of audio recognition, the playback speed scaling ratio of the candidate subsequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence can be obtained based on the local segment, that is, the timestamp information of the candidate subsequence in the two sequences, so as to achieve that the final candidate phonetic symbol sequence can match the audio to be recognized.

[0167] The calculation of the playback speed scaling ratio is determined based on the playback time of the candidate subsequence in the phonetic symbol sequence to be identified and the first phonetic symbol sequence containing the candidate subsequence. That is, the first playback time can be determined based on the start timestamp and end timestamp of the candidate subsequence in the phonetic symbol sequence to be identified, and the second playback time can be determined based on the start timestamp and end timestamp of the candidate subsequence in the first phonetic symbol sequence; the playback speed scaling ratio is determined based on the first playback time and the second playback time.

[0168] For example, see Figure 3AAn example diagram for determining the scaling ratio of the playback speed is shown exemplarily. The sequence of phonetic symbols to be recognized corresponding to the audio to be recognized contains m phonetic symbols, which are represented as R1...Rm, and the pronunciation timestamp of each corresponding phonetic symbol in the audio to be recognized is represented as T1...Tm. When a candidate subsequence with the same phonetic symbol sequence in the candidate song S is matched according to the subsequence to be recognized R_h1...R_hn, it is known that the timestamp of the candidate subsequence R_h1...R_hn in the audio to be recognized is represented as T_r1...T_rn, and the pronunciation timestamp in the candidate song S is represented as T_s1...T_sn, then the playback speed scaling ratio can be expressed as: scale = (playing time in the candidate song S) / (playing time in the audio to be recognized) = (T_sn-T_s1) / (T_rn-T_r1).

[0169] Assuming that the candidate subsequence R_h1...R_hn is played for 30 seconds in the first phonetic symbol sequence and for 15 seconds in the audio to be recognized, the playback speed scaling ratio can be calculated to be 2. The playback speed scaling ratio is used to correspond the entire audio sequence to be recognized to the matching sequence range in the first phonetic symbol sequence, and the matching sequence time range is doubled to expand the total duration of the audio sequence to be recognized. That is, if the playback duration of the complete phonetic symbol sequence to be recognized is 25 seconds, the playback duration of the candidate phonetic symbol sequence corresponding to the phonetic symbol sequence to be recognized in the first phonetic symbol sequence is 50 seconds.

[0170] a2, determining a candidate phonetic symbol sequence that time-matches the phonetic symbol sequence to be recognized according to the playback speed scaling ratio and the position of the candidate subsequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence.

[0171] In this step, the playback time of the phonetic symbol sequence to be identified in the first phonetic symbol sequence can be determined based on the playback speed scaling ratio. Based on the position of the candidate subsequence in the phonetic symbol sequence to be identified and the first phonetic symbol sequence, the candidate subsequence is used as the starting point for expanding the candidate phonetic symbol sequence. Starting from the candidate subsequence, the matching range is expanded to both sides. Through the relative position relationship combined with the playback time, the candidate phonetic symbol sequence that matches the time of the entire phonetic symbol sequence to be identified can be accurately located.

[0172] In this way, the influence of playback speed differences on audio recognition can be effectively overcome, and the accuracy and robustness of recognition can be improved. At the same time, the local-to-global matching approach enables more flexible processing of audio sequences of different lengths and complexities, providing a wider range of possibilities for the application of song recognition technology.

[0173] For the aforementioned step a2, the step of determining a candidate phonetic symbol sequence that time-matches the phonetic symbol sequence to be recognized from the first phonetic symbol sequence containing the candidate subsequences, see Figure 3BThe exemplary flow chart can be implemented by the following steps:

[0174] S301, obtaining a first time interval between a start timestamp of the to-be-recognized phonetic symbol sequence and a start timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0175] The starting timestamp of the phonetic symbol sequence to be recognized is used to indicate the pronunciation time of the first phonetic symbol in the phonetic symbol sequence to be recognized in the audio to be recognized. Figure 3C An example diagram schematically shows a timestamp range determination based on a phonetic symbol sequence, where the phonetic symbol sequence to be recognized corresponding to the audio to be recognized contains m phonetic symbols, represented as R1...Rm, and the pronunciation timestamp of each corresponding phonetic symbol in the audio to be recognized is represented as T1...Tm, and the starting timestamp of the phonetic symbol sequence to be recognized is T1.

[0176] The starting timestamp of the candidate subsequence in the to-be-recognized phonetic symbol sequence is used to indicate the pronunciation time of the first phonetic symbol in the candidate subsequence in the to-be-recognized audio. Figure 3C As shown, the timestamps of the candidate subsequences R_h1 ... R_hn in the audio to be recognized are represented as T_r1 ... T_rn, and the starting timestamp of the candidate subsequence is T_r1.

[0177] The first time interval refers to the difference between the starting timestamp of the phonetic symbol sequence to be recognized and the starting timestamp of the candidate subsequence, which represents the time offset of the candidate subsequence within the phonetic symbol sequence to be recognized relative to the starting position of the entire phonetic symbol sequence to be recognized. Figure 3C For example, the first time interval is the illustrated time interval L1 = |T_r1-T1|.

[0178] S302, obtaining a second time interval between an end timestamp of the to-be-recognized phonetic symbol sequence and an end timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0179] The end timestamp of the phonetic symbol sequence to be recognized is used to indicate the pronunciation time of the last phonetic symbol in the phonetic symbol sequence to be recognized in the audio to be recognized. Figure 3C As shown, the pronunciation timestamp of each phonetic symbol corresponding to the phonetic symbol sequence to be recognized in the audio to be recognized is represented as T1...Tm, and the end timestamp of the phonetic symbol sequence to be recognized is Tm.

[0180] The end timestamp of the candidate subsequence in the to-be-recognized phonetic symbol sequence is used to indicate the pronunciation time of the last phonetic symbol in the candidate subsequence in the to-be-recognized audio. Figure 3C As shown, the timestamps of the candidate subsequences R_h1 ... R_hn in the audio to be recognized are represented as T_r1 ... T_rn, and the end timestamp of the candidate subsequence is T_rn.

[0181] The second time interval refers to the difference between the end timestamp of the phonetic symbol sequence to be recognized and the end timestamp of the candidate subsequence. The difference represents the time offset of the candidate subsequence within the phonetic symbol sequence to be recognized relative to the end position of the entire sequence. It can also be understood as the time distance between the end point of the candidate subsequence and the end point of the phonetic symbol sequence to be recognized. Figure 3C For example, the first time interval is the illustrated time interval L2 = |Tn-T_rn|.

[0182] S303, determining a timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence according to the start timestamp and the end timestamp of the candidate subsequence in the first phonetic symbol sequence, using the first time interval, the second time interval, and the playback speed scaling ratio;

[0183] The starting timestamp and the ending timestamp of the candidate subsequence in the first phonetic symbol sequence identify the starting position and the ending position of the candidate subsequence in the first phonetic symbol sequence.

[0184] In this embodiment, the start timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence can be determined according to the start timestamp of the candidate subsequence in the first phonetic symbol sequence, the playback time scaling ratio and the first time interval; similarly, the end timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence can be determined according to the end timestamp of the candidate subsequence in the first phonetic symbol sequence, the playback time scaling ratio and the second time interval.

[0185] Among them, based on the first time interval reflecting the time offset of the candidate subsequence within the to-be-recognized phonetic symbol sequence relative to the start position of the entire to-be-recognized phonetic symbol sequence, the time offset of the candidate subsequence within the first phonetic symbol sequence relative to the start position of the candidate phonetic symbol sequence can be calculated according to the playback speed scaling ratio and the first time interval; according to the calculated time offset combined with the start timestamp of the candidate subsequence in the first phonetic symbol sequence, the expected start timestamp of the candidate phonetic symbol sequence can be calculated. For example, still with Figure 3C For example, for the time interval L1 on the phonetic symbol sequence to be identified, to determine its corresponding time interval L1' on the first phonetic symbol sequence, it can be determined by the playback time scaling ratio. When the playback time scaling ratio represents (playing time in the candidate song S) / (playing time in the audio to be identified), the time interval L1' can be expressed as L1'=L1*scale, and the revelation timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence can be expressed as Seg_start=T_s1-L1'.

[0186] Based on a similar principle, for the time interval L2 on the phonetic symbol sequence to be identified, to determine its corresponding time interval L2' on the first phonetic symbol sequence, it can be determined by the playback time scaling ratio. When the playback time scaling ratio represents (playing time in the candidate song S) / (playing time in the audio to be identified), the time interval L2' can be expressed as L2'=L2*scale, and the revelation timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence can be expressed as Seg_end=T_s+L2'.

[0187] According to the expected start timestamp and the expected end timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence determined in this step, the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence can be determined, such as Figure 3C [Seg_start, Seg_end] as shown.

[0188] S304: Determine the phonetic symbol sequence corresponding to the timestamp range in the first phonetic symbol sequence as a candidate phonetic symbol sequence.

[0189] See also Figure 3C According to the starting position Seg_start and the ending position Seg_end of the segment in the candidate song S, the phonetic symbol sequence within the time range is found as the candidate phonetic symbol sequence corresponding to the candidate song. The candidate phonetic symbol sequence satisfies the time matching with the phonetic symbol sequence to be recognized corresponding to the audio to be recognized.

[0190] In the disclosed embodiment, based on the phonetic symbol sequence to be recognized and the corresponding timestamps, as well as the candidate subsequences and timestamps that match them in the candidate song set, the time scaling ratio can be calculated, and based on this, the candidate phonetic symbol sequence that temporally matches the entire phonetic symbol sequence to be recognized, as well as the starting position and ending position in the candidate songs, is determined, thereby achieving precise temporal alignment and providing a data basis for subsequent phonetic symbol alignment.

[0191] In some embodiments, for the aforementioned step S104 of aligning the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence, see Figure 4A The schematic diagram of the phonetic symbol alignment steps can be implemented in the following way:

[0192] S401, for each frame of the audio to be recognized, using a pre-trained phonetic symbol recognition network to generate a probability that each frame belongs to each international phonetic symbol;

[0193] Each frame of the audio to be recognized represents a series of small time periods into which the audio signal is segmented. Each frame usually contains a fixed number of audio samples for subsequent processing and analysis. The length of each frame can be adjusted according to actual conditions, but should be short enough to capture subtle changes in speech.

[0194] The pre-trained phonetic symbol recognition network refers to a neural network model that has been trained with a large amount of data and is used to recognize phonetic symbols in audio signals. It can be constructed using deep learning technology and can automatically extract features from audio signals and map them to corresponding phonetic symbols. The phonetic symbol recognition network and the phonetic symbol sequence to be recognized obtained through the phonetic symbol recognition network in the aforementioned embodiment belong to the same network.

[0195] The phonetic symbol recognition network is used to convert audio into an international phonetic symbol sequence. The training purpose is to associate the speech features in the audio with the corresponding international phonetic symbols and learn the audio representation of each phonetic symbol. In the present invention, 108 phonetic symbols can be used as training targets, including: 107 international phonetic symbols, plus one other pronunciation or silence symbol sil. The phonetic symbol recognition network can be modeled using a traditional hidden Markov model or an end-to-end deep learning model. The model structure of the phonetic symbol recognition network can adopt a neural network such as a convolutional neural network or a recurrent neural network, etc., which can learn the deep features of the phonetic symbols from the audio and improve the robustness to non-speech interference.

[0196] The overall training process of the phonetic symbol recognition network can be found in Figure 4B As shown, for the song audio with the actual phonetic symbol sequence label of the song in the training sample, the song audio can be Fourier transformed frame by frame, such as a frame length of 25ms and a frame shift of 10ms to obtain the original spectrum features; next, the original spectrum features are converted into Mel spectrum features or MFCC features, and the features are input into the phonetic symbol recognition network for inference. The target label of the training is the actual phonetic symbol sequence of the song, and the loss function such as the CTC function is calculated as the error, that is, the maximum probability when the output is the target phonetic symbol sequence under the premise of given input features. The gradient descent is used for error back propagation to continuously update the neural network model until the error calculated by the loss function no longer decreases or the specified training condition is reached to terminate the training.

[0197] Before inputting the audio into the phonetic symbol recognition network, the audio to be recognized can be preprocessed, such as noise reduction, echo removal, volume normalization, etc., to improve the accuracy of phonetic symbol recognition.

[0198] For each frame, the audio features are extracted through the phonetic symbol recognition network. The audio features can be Mel-frequency cepstral coefficients, spectrum envelopes, fundamental frequencies, etc., which are used to describe the spectrum and time domain characteristics of the audio signal. The extracted features are automatically processed by the phonetic symbol recognition network, and the probability of each frame belonging to each international phonetic symbol is output. The probability is used to indicate the possibility that the pronunciation in the frame belongs to the international phonetic symbol. For a certain frame in the audio, after network calculation, the probability of the phonetic symbol [p] is 0.1, the probability of the phonetic symbol [b] is 0.05, etc. If 108 international phonetic symbols are used, a total of 108 probability values ​​corresponding to the international phonetic symbol will be output for each frame.

[0199] S402, constructing a decoding sequence according to the candidate phonetic symbol sequence; the nodes in the decoding sequence correspond to the phonetic symbols in the candidate phonetic symbol sequence;

[0200] The decoding sequence is used to represent the state transition relationship of the phonetic symbols in the candidate phonetic symbol sequence at different time points, and is used to decode according to the probability that each frame belongs to each international phonetic symbol, so as to determine the exact position (start time and end time) of the phonetic symbol in the audio.

[0201] See also Figure 4C A schematic diagram of a decoding sequence is shown schematically, with a candidate phonetic symbol sequence For example, the phonetic symbols are [m], [th], [j], [ε], [n], [i], [x], There are 10 phonetic symbols in total, and the constructed decoding sequence can be expressed as follows Figure 4B The state transition order indicated by the arrows is consistent with the order of the included phonetic symbols, and 10 phonetic symbols form at least 10 nodes.

[0202] Based on the characteristics that most songs may pause between sentences but not within a sentence, the decoding sequence can adopt a serial structure, including multiple sequence nodes; when the phonetic symbols corresponding to multiple continuous sequence nodes belong to a complete lyric, the adjacent front node of the start node and the adjacent rear node of the end node in the multiple continuous sequence nodes are nodes indicating silence; the adjacent nodes of the multiple continuous sequence nodes include skippable nodes indicating silence. Through this decoding sequence setting, there must be sil nodes before and after each lyric, and the phonetic symbols within the sentence are skippable sil nodes, thereby improving the accuracy of phonetic symbol alignment.

[0203] like Figure 4D Another schematic diagram of a decoding sequence is shown schematically, with a candidate phonetic symbol sequence For example, the candidate phonetic symbol sequence belongs to a complete lyric, and there are sequence nodes representing silence before and after it. For each phonetic symbol in the lyric, a sequence node representing silence that can be skipped is set.

[0204] S403, according to the probability that each frame belongs to each international phonetic symbol, use the decoding sequence to decode, and obtain a triple sequence of the phonetic symbols in the audio to be recognized that are aligned with the candidate phonetic symbol sequence, wherein the triplet includes the phonetic symbol symbol, the start time and the end time of the phonetic symbol in the audio to be recognized.

[0205] The triple sequence is the output of the decoding process. Each triple includes a phonetic symbol, the start time of the phonetic symbol in the audio to be recognized, and the end time. The triple sequence provides complete information about the exact position and duration of each phonetic symbol in the candidate phonetic sequence in the audio to be recognized.

[0206] Regarding decoding using the decoding sequence, the phonetic symbol probability of each frame can be used for Viterbi decoding to obtain the alignment result of the final candidate phonetic symbol sequence with the input audio to be recognized. Viterbi decoding can find the most likely phonetic symbol alignment result based on the phonetic symbol probability and the structure of the decoding sequence, and calculate the corresponding decoding path score.

[0207] To better understand the decoding process, this embodiment takes Viterbi decoding and the decoding sequence of the example in Figure D as an example to exemplarily describe the decoding process:

[0208] Assume that the probability of phonetic symbols for each frame of the audio to be recognized is as follows:

[0209] frame 1: [0.01,0.02,...,0.9,...]

[0210] frame 2: [0.01,0.02,...,0.8,...]

[0211] frame 3: [0.01,0.02,...,0.7,...]

[0212] frame 4: [0.01,0.02,...,0.6,...]

[0213] frame 5: [0.01,0.02,...,0.6,...]

[0214] frame N: [0.01,0.02,...,0.5,...]

[0215] based on Figure 4D The example decoding sequence performs Viterbi decoding (f:1 represents the first frame, f:1-2 represents the 1st-2nd frame, and the same applies to the others):

[0216] Possible path for frame 1: sil(f:1)

[0217] Possible paths for frame 2: sil(f:1-2), sil(f:1)->m(f:2)

[0218] Possible paths for frame 3: sil(f:1-3), sil(f:1-2)->m(f:3), sil(f:1)->m(f:2-3), sil(f:1)->m(f:2)->I(f:3)

[0219] …

[0220] And so on, the possible path of the Nth frame is: ..., sil(f:1)->m(f:2)->I(f:3)-> ->…-> ->sil(f:N).

[0221] The triplet series obtained by phonetic symbol alignment is to select the path with the highest probability of the last node being the network end node sil from all decoding paths, assuming it is sil(f:1-2)->m(f:3)->I(f:4)-> ->…-> ->sil(f:N), the path to the last node must pass through all phonetic symbols.

[0222] The resulting triple sequence consists of a sequence of phonetic symbols, the start time of the phonetic symbol in the audio to be recognized, and the end time, which represents the final result of phonetic symbol alignment and clarifies the exact time range of each phonetic symbol in the audio to be recognized. Figure 4D The decoding sequence shown, its corresponding triple sequence can be expressed as (sil,0,10), (m,10,12), (I,12,13), (t,15,16),(j,16,17)…….

[0223] In the disclosed embodiment, by utilizing a pre-trained phonetic symbol recognition network to generate the probability of belonging to each international phonetic symbol for each frame of the audio to be recognized, the pronunciation details in the audio can be captured more accurately, and a decoding sequence is constructed based on the candidate phonetic symbol sequence that time-matches the phonetic symbol sequence to be recognized in the candidate song. By using this sequence for decoding, pronunciation variations and connected reading phenomena in the audio can be handled more flexibly, thereby achieving phonetic symbol alignment between the audio to be recognized and the candidate phonetic symbol sequence, which can reduce the possibility of mismatches and thus improve the accuracy of song recognition.

[0224] In some embodiments, based on the triplet obtained by the aforementioned phonetic symbol alignment, for the phonetic symbol alignment score of the audio to be recognized obtained according to the candidate phonetic symbol sequence in the aforementioned step S104, the phonetic symbol alignment score can be calculated by the posterior alignment probability, which can be specifically implemented by the following steps b1-b3:

[0225] b1, for each triplet in the decoded triplet sequence, first obtain the probability mean of each international phonetic symbol within the triplet time range according to the probability that each frame in the audio to be recognized belongs to each international phonetic symbol; the triplet time range is determined according to the start time and end time of the phonetic symbol in the triplet in the audio to be recognized;

[0226] For any international phonetic symbol, the probability mean is used to represent the average value of the probability that each frame of the audio to be recognized within the time range represented by the triplet is recognized as belonging to the international phonetic symbol, reflecting the stability and consistency of the audio content being recognized as a specific phonetic symbol within the time range.

[0227] Take the triple (j, 16, 17) as an example. In the triple, j represents the international phonetic symbol, 16 and 17 are in seconds, representing the start time and end time of the phonetic symbol in the audio to be recognized. The probability of all frames from 16s to 17s being recognized as each international phonetic symbol is obtained, for example, Frame 16: [0.01, 0.02, 0.05, ..., 0.90, ...], Frame 17: [0.02, 0.03, 0.06, ..., 0.85, ...]. The correspondence between the index in the score array and the phonetic symbol has been determined. Assuming that the scores corresponding to the phonetic symbol j are 0.90 and 0.85 respectively, the mean probability of j in this triple time range is the sum of the probabilities of the phonetic symbol j in the two frames, and then divided by the number of frames 2, that is, the mean probability of the phonetic symbol j = (0.90 + 0.85) / 2 = 0.875. Based on the same principle, the calculation method for each other international phonetic symbol is the same as that for this phonetic symbol j.

[0228] b2, calculating the posterior alignment probability of the triplet according to the mean probability of the phonetic symbols in the triplet and the sum of the mean probability of each international phonetic symbol within the time range of the triplet;

[0229] That is, after obtaining the probability mean of each international phonetic symbol within the triple time range, first calculate the sum of the probability means of the international phonetic symbol Where M represents the total number of international phonetic symbols (such as 108), represents the mean probability.

[0230] The posterior alignment probability of this triple can be expressed as Where char represents the probability mean of the phonetic symbol in the triple within the triple time range. For example, taking the triple (j, 16, 17) as an example, its posterior alignment probability represents the probability mean of the phonetic symbol j / the sum of the probability means of all international phonetic symbols in the 16th-17th second.

[0231] The value range of the posterior alignment probability is [0, 1]. The larger the value of the posterior alignment probability is, the greater the probability that the pronunciation of the time period is the phonetic symbol in the triplet.

[0232] b3, obtaining the phonetic symbol alignment score corresponding to the candidate phonetic symbol sequence according to the posterior alignment probability of each triplet.

[0233] After obtaining the posterior alignment probability of each triple, the posterior alignment probabilities of all triples can be cumulatively summed, and the cumulative sum result can be used as the phonetic symbol alignment score. Alternatively, a weight can be assigned to each triple according to factors such as the importance of the phonetic symbol in the song and the difficulty of pronunciation, and then the posterior alignment probability of each triple is multiplied by the corresponding weight, and then the cumulative sum is performed to obtain the phonetic symbol alignment score to more accurately reflect the degree of matching between the audio and the candidate phonetic symbol sequence. Alternatively, any other applicable phonetic symbol alignment score calculation method can also be used, and the present invention does not limit this.

[0234] In the disclosed embodiment, the posterior alignment probability reflects the degree of match between a certain phonetic symbol and the audio content within a given audio frame range. By calculating the posterior alignment probability of each triplet, the degree of match between the candidate phonetic symbol sequence and the audio to be recognized can be more accurately evaluated, and then the phonetic symbol alignment score can be obtained based on the posterior alignment probability, which can more objectively reflect the similarity between the candidate phonetic symbol sequence and the audio to be recognized, and provide strong support for the subsequent sorting or selection of song recognition results.

[0235] In some embodiments, song recognition based on phonetic symbols relies on phonetic symbol matching. In order to effectively match the phonetic symbol information of the audio to be recognized with the information in the database and optimize the efficiency and accuracy of the recognition process, after obtaining the phonetic symbol sequence to be recognized, it is possible to detect whether the amount of phonetic symbol information contained in the phonetic symbol sequence to be recognized is rich enough to avoid the problem that there is too little phonetic symbol information to make an effective match. That is, for the obtained phonetic symbol sequence to be recognized, the length of the phonetic symbol sequence to be recognized can be obtained, and it is detected whether the length of the phonetic symbol sequence to be recognized is greater than a set threshold; if the length of the phonetic symbol sequence to be recognized is greater than the set threshold, the phonetic symbol sequence to be recognized is compared with the first phonetic symbol sequence; if the length of the phonetic symbol sequence to be recognized is less than the set threshold, the song recognition is terminated.

[0236] The length of the phonetic symbol sequence to be identified is used to indicate the number of international phonetic symbols contained in the phonetic symbol sequence to be identified. The longer the length of the phonetic symbol sequence to be identified, the more pronunciation features are contained in the audio to be identified, thereby providing more information for comparison and identification. The length of the phonetic symbol sequence to be identified can be obtained by traversing the phonetic symbol sequence to be identified and counting each phonetic symbol, or by using a string processing function or other statistical tools.

[0237] The threshold is used to represent the minimum number of phonetic symbols required for song recognition, and the value can be determined by the developer based on experience or through multiple trials.

[0238] The length of the phonetic symbol sequence to be recognized is less than the set threshold, which indicates that the audio segment to be recognized is too short and contains limited phonetic symbol information, which is insufficient to support effective matching with the complete song sequence. It is difficult to obtain reliable recognition results by phonetic symbol matching. Therefore, in order to save computing resources and avoid unnecessary processing, this embodiment terminates the song recognition process when the length of the phonetic symbol sequence to be recognized is less than the set threshold, thereby ensuring the accuracy of the recognition results while improving the overall operating efficiency.

[0239] When song recognition is terminated, a prompt message such as "The audio is too short, please re-enter" can be displayed to the user through the song recognition interactive interface to prompt the user the specific reason for the current recognition failure and guide the user to perform corresponding operations, helping the user to quickly understand the problem, so that the user can complete the song recognition process more smoothly and improve the user experience.

[0240] In the disclosed embodiment, by detecting the length of the phonetic symbol sequence to be recognized and directly terminating the song recognition when the length of the phonetic symbol sequence to be recognized is insufficient, unnecessary comparison processes and users waiting for inaccurate or failed results for a long time are avoided, the user experience is optimized, the waste of computing resources is reduced, and the efficiency of the entire recognition process is improved. At the same time, this method allows sequences containing sufficiently rich phonetic symbol information to participate in the comparison, which can effectively avoid misidentification due to insufficient information. Rich phonetic symbol information can more accurately reflect the characteristics of the audio content, thereby improving the accuracy of song recognition.

[0241] In some embodiments, based on the above song recognition method, when there are songs in a new language that need to be incorporated into the database so that they can be retrieved for subsequent song recognition, the songs in the new language can be converted into an international phonetic symbol sequence and stored. In the method of performing phonetic symbol comparison with an inverted index, the song in the new language can be converted into an international phonetic symbol sequence and divided into multiple subsequences according to a set sequence length to form a newly added index key, which is added to the data structure of the existing inverted index. The index value corresponding to the newly added index key is the song in the new language, and the time position of each index key in the song in the new language.

[0242] In the disclosed embodiment, when there are songs in a new language, it is only necessary to establish a phoneme correspondence between the new language and the International Phonetic Alphabet, so as to quickly support the recognition of songs in the new language. There is no need to prepare a large amount of audio data in the new language to retrain the model, which makes it more scalable and reduces the cost of supporting new languages.

[0243] Exemplary Devices

[0244] After introducing the method of the exemplary embodiment of the present invention, next, refer to Figure 5 A device according to an exemplary embodiment of the present invention is described.

[0245] The implementation process of the functions and effects of each module in the following device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here. For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment.

[0246] Figure 5 A song recognition device according to an embodiment of the present invention is schematically shown, the device comprising:

[0247] The to-be-recognized phonetic symbol sequence acquisition module 501 is used to acquire the to-be-recognized phonetic symbol sequence corresponding to the to-be-recognized audio;

[0248] The phonetic symbol comparison module 502 is used to compare the phonetic symbol sequence to be identified with the first phonetic symbol sequence of each candidate song, and determine the candidate subsequence whose phonetic symbols in the phonetic symbol sequence to be identified are the same as those in the first phonetic symbol sequence and whose sequence length meets the requirements;

[0249] The candidate phonetic symbol sequence acquisition module 503 is used to determine a candidate phonetic symbol sequence that matches the phonetic symbol sequence to be recognized in time from the first phonetic symbol sequence containing the candidate subsequence according to the position of the candidate subsequence in the phonetic symbol sequence to be recognized and the first phonetic symbol sequence;

[0250] A phonetic symbol alignment module 504 is used to align the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence to obtain a phonetic symbol alignment score of the audio to be recognized according to the candidate phonetic symbol sequence;

[0251] The song recognition result generating module 505 is used to determine the song recognition result of the audio to be recognized according to the phonetic symbol alignment scores corresponding to each candidate phonetic symbol sequence.

[0252] In some embodiments, the phonetic symbol comparison module is specifically used to:

[0253] A division module, used for dividing the to-be-recognized phonetic symbol sequence into a plurality of to-be-recognized subsequences according to a set sequence length;

[0254] An index key matching module, used for matching each subsequence to be identified with an index key of an inverted index; wherein the index key is a first phonetic symbol sequence; the first phonetic symbol sequence refers to each subsequence obtained by dividing the complete phonetic symbol sequence of each candidate song according to the set sequence length; the index value corresponding to the index key is the candidate song information containing the index key;

[0255] The candidate subsequence determination module is used to use the index key that is the same as the phonetic symbol of the subsequence to be identified as the candidate subsequence.

[0256] In some embodiments, the partitioning module is specifically used to:

[0257] Starting from the starting phonetic symbol of the phonetic symbol sequence to be recognized, a window is used to slide on the phonetic symbol sequence to be recognized according to a set step size and an overlap value of adjacent windows, so as to divide the phonetic symbol sequence to be recognized into a plurality of subsequences to be recognized; wherein the length of the window is set to the set sequence length.

[0258] In some embodiments, the candidate phonetic symbol sequence acquisition module is specifically used to:

[0259] A scaling ratio determination module, used to obtain a scaling ratio of the playback speed of the candidate subsequence on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence;

[0260] According to the playback speed ratio and the position of the candidate subsequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence, a candidate phonetic symbol sequence that time matches the to-be-recognized phonetic symbol sequence is determined.

[0261] In some embodiments, the scaling ratio determination module is specifically used to:

[0262] Determine a first playback duration according to a start timestamp and an end timestamp of the candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0263] Determining a second playback duration according to a start timestamp and an end timestamp of the candidate subsequence in the first phonetic symbol sequence;

[0264] The playback speed scaling ratio is determined according to the first playback duration and the second playback duration.

[0265] In some embodiments, the candidate phonetic symbol sequence determination module is specifically used to:

[0266] Acquire a first time interval between a start timestamp of the to-be-recognized phonetic symbol sequence and a start timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0267] Acquire a second time interval between an end timestamp of the to-be-recognized phonetic symbol sequence and an end timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence;

[0268] According to the start timestamp and the end timestamp of the candidate subsequence in the first phonetic symbol sequence, using the first time interval, the second time interval and the playback speed scaling ratio, determine the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence;

[0269] The phonetic symbol sequence corresponding to the timestamp range in the first phonetic symbol sequence is determined as a candidate phonetic symbol sequence.

[0270] In some embodiments, the candidate phonetic symbol sequence determination module, when used to determine the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence, includes:

[0271] Determine a start timestamp of a candidate phonetic symbol sequence in the first phonetic symbol sequence according to the start timestamp of the candidate subsequence in the first phonetic symbol sequence, the playback time scaling ratio, and a first time interval;

[0272] The end timestamp of the candidate phonetic symbol sequence in the first phonetic symbol sequence is determined according to the end timestamp of the candidate subsequence in the first phonetic symbol sequence, the playback time scaling ratio, and the second time interval.

[0273] In some embodiments, the phonetic symbol alignment module is specifically used to:

[0274] For each frame of the audio to be recognized, using a pre-trained phonetic symbol recognition network to generate a probability that each frame belongs to each international phonetic symbol;

[0275] According to the candidate phonetic symbol sequence, a decoding sequence is constructed; the nodes in the decoding sequence correspond to the phonetic symbols in the candidate phonetic symbol sequence;

[0276] According to the probability that each frame belongs to each international phonetic symbol, the decoding sequence is used to decode to obtain a triple sequence of the phonetic symbols in the audio to be recognized that are aligned with the candidate phonetic symbol sequence, wherein the triplet includes the phonetic symbol symbol, the start time and the end time of the phonetic symbol in the audio to be recognized.

[0277] In some embodiments, the decoding sequence adopts a serial structure, including multiple sequence nodes; when the phonetic symbols corresponding to multiple consecutive sequence nodes belong to a complete lyric,

[0278] The adjacent previous node of the start node and the adjacent subsequent node of the end node in the plurality of continuous sequence nodes are nodes indicating silence; and the adjacent nodes of the plurality of continuous sequence nodes include skippable nodes indicating silence.

[0279] In some embodiments, the phonetic symbol alignment module is specifically used to:

[0280] For each triplet, according to the probability that each frame in the audio to be recognized belongs to each international phonetic symbol, obtain the probability mean of each international phonetic symbol within the triplet time range; the triplet time range is determined according to the start time and end time of the phonetic symbol in the triplet in the audio to be recognized;

[0281] Calculating the posterior alignment probability of the triplet according to the mean probability of the phonetic symbols in the triplet and the sum of the mean probability of each international phonetic symbol within the triplet time range;

[0282] According to the posterior alignment probability of each triplet, a phonetic symbol alignment score corresponding to the candidate phonetic symbol sequence is obtained.

[0283] In some embodiments, the module for acquiring the to-be-recognized phonetic symbol sequence is specifically used to:

[0284] Performing phonetic symbol recognition on the audio to be recognized using a pre-trained phonetic symbol recognition network to obtain a corresponding phonetic symbol sequence to be recognized and timestamp information of each phonetic symbol in the audio to be recognized;

[0285] The device also includes:

[0286] When the length of the to-be-recognized phonetic symbol sequence is greater than a set threshold, the to-be-recognized phonetic symbol sequence is compared with the first phonetic symbol sequence; when the length of the to-be-recognized phonetic symbol sequence is less than the set threshold, song recognition is terminated.

[0287] Exemplary Media

[0288] After introducing the method and apparatus of the exemplary embodiment of the present invention, next, reference is made to Figure 6 A readable storage medium according to an exemplary embodiment of the present invention is described.

[0289] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present invention is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device.

[0290] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0291] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0292] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.

[0293] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0294] Exemplary Computing Devices

[0295] After introducing the method, apparatus and medium of the exemplary embodiments of the present invention, next, reference is made to Figure 7 A computing device according to an exemplary embodiment of the present invention is described.

[0296] Figure 7The computing device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0297] like Figure 7 As shown, the computing device 700 is in the form of a general computing device. The components of the computing device 700 may include but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including the processing unit 701 and the storage unit 702).

[0298] The bus 703 includes a data bus, a control bus, and an address bus.

[0299] The storage unit 702 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 7021 and / or a cache memory 7022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 7023 .

[0300] The storage unit 702 may also include a program / utility 7025 having a set (at least one) of program modules 7024, such program modules 7024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0301] Computing device 700 may also communicate with one or more external devices 704 (eg, keyboards, pointing devices, etc.).

[0302] Such communication may be performed via input / output (I / O) interface 705. In addition, computing device 700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via network adapter 706. Figure 7 As shown, network adapter 706 communicates with other modules of computing device 700 via bus 703. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with computing device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0303] It should be noted that although several units / modules or sub-units / modules of the song recognition device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0304] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0305] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A song recognition method, characterized in that: The method comprises: Obtain the phonetic symbol sequence to be recognized corresponding to the audio to be recognized; Compare the to-be-recognized phonetic symbol sequence with the first phonetic symbol sequence of each candidate song to determine a candidate subsequence whose to-be-recognized phonetic symbol sequence has the same phonetic symbol as the first phonetic symbol sequence and whose sequence length meets the requirement; According to the positions of the candidate subsequences in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence, determining, from the first phonetic symbol sequence containing the candidate subsequences, a candidate phonetic symbol sequence that time-matches the to-be-recognized phonetic symbol sequence; Aligning the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence to obtain a phonetic symbol alignment score of the audio to be recognized according to the candidate phonetic symbol sequence; The song recognition result of the audio to be recognized is determined according to the phonetic symbol alignment scores corresponding to each candidate phonetic symbol sequence.

2. The method according to claim 1, characterized in that The step of comparing the to-be-recognized phonetic symbol sequence with the first phonetic symbol sequence of each candidate song includes: According to the set sequence length, the to-be-recognized phonetic symbol sequence is divided into a plurality of to-be-recognized subsequences; Match each subsequence to be identified with an index key of the inverted index; wherein the index key is a first phonetic symbol sequence; the first phonetic symbol sequence refers to each subsequence obtained by dividing the complete phonetic symbol sequence of each candidate song according to the set sequence length; the index value corresponding to the index key is the candidate song information containing the index key; The index key that is the same as the phonetic symbol of the subsequence to be identified is used as the candidate subsequence.

3. The method according to claim 1, characterized in that: The step of determining, from the first phonetic symbol sequence including the candidate subsequences, a candidate phonetic symbol sequence that time matches the phonetic symbol sequence to be recognized comprises: Obtaining a playback speed scaling ratio of the candidate subsequence on the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence; According to the playback speed ratio and the position of the candidate subsequence in the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence, a candidate phonetic symbol sequence that time matches the to-be-recognized phonetic symbol sequence is determined.

4. The method according to claim 3, characterized in that The step of determining, from the first phonetic symbol sequence including the candidate subsequences, a candidate phonetic symbol sequence that time matches the phonetic symbol sequence to be recognized comprises: Acquire a first time interval between a start timestamp of the to-be-recognized phonetic symbol sequence and a start timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence; Acquire a second time interval between an end timestamp of the to-be-recognized phonetic symbol sequence and an end timestamp of a candidate subsequence in the to-be-recognized phonetic symbol sequence; According to the start timestamp and the end timestamp of the candidate subsequence in the first phonetic symbol sequence, using the first time interval, the second time interval and the playback speed scaling ratio, determine the timestamp range of the candidate phonetic symbol sequence in the first phonetic symbol sequence; The phonetic symbol sequence corresponding to the timestamp range in the first phonetic symbol sequence is determined as a candidate phonetic symbol sequence.

5. The method according to claim 1, characterized in that Aligning the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence, comprising: For each frame of the audio to be recognized, using a pre-trained phonetic symbol recognition network to generate a probability that each frame belongs to each international phonetic symbol; According to the candidate phonetic symbol sequence, a decoding sequence is constructed; the nodes in the decoding sequence correspond to the phonetic symbols in the candidate phonetic symbol sequence; According to the probability that each frame belongs to each international phonetic symbol, the decoding sequence is used to decode to obtain a triple sequence of the phonetic symbols in the audio to be recognized that are aligned with the candidate phonetic symbol sequence, wherein the triplet includes the phonetic symbol symbol, the start time and the end time of the phonetic symbol in the audio to be recognized.

6. The method according to claim 5, characterized in that The step of obtaining the phonetic symbol alignment score of the audio to be recognized according to the candidate phonetic symbol sequence includes: For each triplet, according to the probability that each frame in the audio to be recognized belongs to each international phonetic symbol, obtain the probability mean of each international phonetic symbol within the triplet time range; the triplet time range is determined according to the start time and end time of the phonetic symbol in the triplet in the audio to be recognized; Calculating the posterior alignment probability of the triplet according to the probability mean of the phonetic symbols in the triplet and the sum of the probability mean of each international phonetic symbol within the triplet time range; According to the posterior alignment probability of each triplet, a phonetic symbol alignment score corresponding to the candidate phonetic symbol sequence is obtained.

7. The method according to claim 1, characterized in that The step of obtaining a sequence of phonetic symbols to be recognized corresponding to the audio to be recognized includes: Performing phonetic symbol recognition on the audio to be recognized using a pre-trained phonetic symbol recognition network to obtain a corresponding phonetic symbol sequence to be recognized and timestamp information of each phonetic symbol in the audio to be recognized; The method further comprises: When the length of the to-be-recognized phonetic symbol sequence is greater than a set threshold, performing phonetic symbol comparison between the to-be-recognized phonetic symbol sequence and the first phonetic symbol sequence; When the length of the to-be-recognized phonetic symbol sequence is less than a set threshold, song recognition is terminated.

8. A song recognition device, characterized in that: The device comprises: A module for obtaining a sequence of phonetic symbols to be recognized, used to obtain a sequence of phonetic symbols to be recognized corresponding to the audio to be recognized; A phonetic symbol comparison module is used to compare the phonetic symbol sequence to be identified with the first phonetic symbol sequence of each candidate song, and determine a candidate subsequence whose phonetic symbols in the phonetic symbol sequence to be identified are the same as those in the first phonetic symbol sequence and whose sequence length meets the requirements; A candidate phonetic symbol sequence acquisition module is used to determine a candidate phonetic symbol sequence that matches the phonetic symbol sequence to be identified in time from the first phonetic symbol sequence containing the candidate subsequence according to the position of the candidate subsequence in the phonetic symbol sequence to be identified and the first phonetic symbol sequence; A phonetic symbol alignment module, used to align the audio to be recognized with the phonetic symbols in the candidate phonetic symbol sequence to obtain a phonetic symbol alignment score of the audio to be recognized according to the pronunciation of the candidate phonetic symbol sequence; The song recognition result generating module is used to determine the song recognition result of the audio to be recognized according to the phonetic symbol alignment scores corresponding to each candidate phonetic symbol sequence.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computing device, characterized in that include: Processor, memory; The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 7 by calling the computer program.

Images