Humming audio recognition method and device, electronic equipment and storage medium
By acquiring and correcting the keyword sequence and tune characteristics in the humming audio data, the problem of user identification is solved by unclear speech, and the recognition accuracy of humming audio and the universality of the system are improved.
Patent Information
- Application Number
- CN202411323907.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-27
AI Technical Summary
Existing humming recognition solutions are difficult to accurately identify people with unclear speech, especially humming audio data from children's users, resulting in low recognition accuracy.
By obtaining the user's original audio data, building the original keyword sequence and correcting the pronunciation data, extracting tune characteristics, and retrieving the corresponding target songs in combination with the target keyword sequence, correcting keyword deviations caused by inaccurate pronunciation, and improving the search accuracy.
It improves the accuracy of users' humming audio recognition for unclear speech, can handle the humming styles and characteristics of different users, and increases the versatility and adaptability of the system.
Smart Images
Figure CN119207469A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of audio recognition, and in particular to a method, device, electronic equipment and storage medium for recognizing humming audio. Background Art
[0002] The current humming recognition solution means that users can use fuzzy recognition to fuzzily match the music library by humming without playing the original song, thereby helping users find the corresponding original song. However, most of the current mainstream mature solutions are designed for adult users. For some special groups, such as people with unclear articulation, it is impossible to accurately identify them based on the audio data of their humming. For example, children users cannot control their vocal organs as accurately as adults because their vocal organs are still in the development stage, and some children have inaccurate pronunciation. These factors greatly affect the accuracy of humming recognition. Summary of the invention
[0003] In view of this, the embodiments of the present invention provide a humming audio recognition method, device, electronic device and storage medium to solve the problem that some special groups, such as people with unclear articulation, cannot be accurately recognized based on their humming audio data.
[0004] In a first aspect, an embodiment of the present invention provides a method for recognizing humming audio, the method comprising:
[0005] Obtaining the original audio data of the user during humming;
[0006] Constructing an original keyword sequence based on the original keywords contained in the original audio data, and obtaining pronunciation data corresponding to each original keyword from the original audio data;
[0007] Modifying the original keyword sequence based on the pronunciation data to obtain a target keyword sequence;
[0008] The melody features of the original audio data are extracted, and the corresponding target song is retrieved using the melody features and the target keyword sequence.
[0009] Furthermore, constructing an original keyword sequence based on the original keywords contained in the original audio data includes:
[0010] Recognize the original audio data to obtain a humming sentence corresponding to the original audio data;
[0011] Detecting the words contained in the humming sentence, and counting the frequency of occurrence of each word in the humming sentence;
[0012] The words with an appearance frequency lower than a preset frequency are used as the original keywords, and the original keyword sequence is constructed based on the appearance order of the original keywords in the original audio data.
[0013] Furthermore, the original keyword sequence is modified based on the pronunciation data to obtain a target keyword sequence, including:
[0014] Acquire a target standard pronunciation library, wherein the target standard pronunciation library includes a plurality of standard pronunciation data;
[0015] Comparing the pronunciation data with the standard pronunciation data to obtain abnormal data with confused pronunciation in the pronunciation data;
[0016] Abnormal keywords corresponding to abnormal data are obtained from the original keyword sequence, and the abnormal keywords in the original keyword sequence are corrected to obtain the target keyword sequence.
[0017] Furthermore, the step of obtaining a target standard pronunciation library includes:
[0018] Identify the pronunciation data and obtain the language type corresponding to the pronunciation data;
[0019] Based on the mapping relationship between the preset language type and the standard pronunciation library, a target standard pronunciation library corresponding to the language type is determined.
[0020] Furthermore, the comparing the pronunciation data with the standard pronunciation data to obtain abnormal data with pronunciation confusion in the pronunciation data includes:
[0021] Comparing the pronunciation data with the standard pronunciation data, and taking the pronunciation data that is different from the standard pronunciation data as candidate pronunciation data;
[0022] matching the candidate pronunciation data with a plurality of pronunciation confusion types;
[0023] If there is any candidate pronunciation data that matches the pronunciation confusion type, the candidate pronunciation data is used as the abnormal data.
[0024] Furthermore, the correcting abnormal keywords in the original keyword sequence to obtain the target keyword sequence includes:
[0025] Performing semantic recognition on the original audio data to obtain semantic information;
[0026] Obtaining a pronunciation confusion type corresponding to the abnormal keyword;
[0027] The abnormal keywords in the original keyword sequence are corrected based on the semantic information and the pronunciation confusion type to obtain the target keyword sequence.
[0028] Further, the method of retrieving a corresponding target song by using the melody feature and the target keyword sequence includes:
[0029] Generate a target simplified musical notation curve corresponding to the melody feature, wherein the abscissa of the target simplified musical notation curve is time, and the ordinate of the target simplified musical notation curve is the melody feature;
[0030] Matching the target simplified musical notation curve with the simplified musical notation curves of each song in the music library to obtain at least one first candidate song that matches the target simplified musical notation curve;
[0031] Using the target keyword sequence to match the lyrics of each song in the music library, obtaining at least one second candidate song that matches the target keyword sequence;
[0032] The target song is obtained from the first candidate songs and the second candidate songs.
[0033] In a second aspect, an embodiment of the present invention provides a device for identifying humming audio, the device comprising:
[0034] An acquisition module is used to acquire the original audio data of the user during the humming process;
[0035] A recognition module, configured to construct an original keyword sequence based on the original keywords contained in the original audio data, and obtain pronunciation data corresponding to each original keyword from the original audio data;
[0036] A correction module, used for correcting the original keyword sequence based on the pronunciation data to obtain a target keyword sequence;
[0037] The retrieval module is used to extract the melody features of the original audio data and retrieve the corresponding target song using the melody features and the target keyword sequence.
[0038] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0039] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.
[0040] The embodiment of the present application is for people with unclear articulation, such as children's humming, where there is inaccurate pronunciation and unclear articulation. By obtaining the pronunciation data corresponding to each original keyword and correcting the original keyword sequence based on these pronunciation data, the keyword deviation caused by inaccurate pronunciation can be corrected. Thereby improving the accuracy of subsequent retrieval of target songs. Then, by combining the target keyword sequence after the correction of the original keyword sequence and the melody features for retrieval, the user's humming intention can be understood more comprehensively and accurately, thereby improving the accuracy of finding the target song. And it can handle the humming styles and characteristics of different users, including situations such as non-standard pronunciation and unclear keywords, which increases the versatility of the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0042] Figure 1 is a flow chart of a method for recognizing humming audio according to some embodiments of the present invention;
[0043] Figure 2 is a flow chart of another method for recognizing humming audio according to some embodiments of the present invention;
[0044] Figure 3 is a structural block diagram of a device for recognizing humming audio according to an embodiment of the present invention;
[0045] Figure 4 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0047] According to an embodiment of the present invention, a method, device, electronic device and storage medium for recognizing humming audio are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0048] In this embodiment, a method for recognizing humming audio is provided. Figure 1 is a flow chart of a method for recognizing humming audio according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:
[0049] Step S101, obtaining original audio data of the user during humming.
[0050] The method provided in the embodiment of the present application is applied to a mobile application or desktop software that collects user humming audio. The user can open the application, select the humming function, and then start humming a song or melody. The application will record the original audio data of the user's humming and further process it, such as pitch analysis, melody recognition, or matching with other music data.
[0051] Specifically, a user interface is provided in the application, and the user interface includes a "start humming" button and a "stop humming" button. When the application starts, the audio acquisition device is initialized, and parameters such as the sampling rate, the number of channels, and the buffer size are set. When the user clicks the "start humming" button, the audio acquisition process is started. The original audio data is read from the audio device and stored in a memory buffer or a temporary file. During the user's humming process, data is continuously read from the audio device and added to the collected data set. In addition, the application can set an appropriate time interval or stop the acquisition according to the user's operation (such as clicking the "stop humming" button). When the user clicks the "stop humming" button, the audio acquisition process is stopped. Then some basic processing is performed on the collected original audio data, such as noise removal, gain adjustment, etc., to obtain the original audio data of the user during the humming process.
[0052] Step S102: construct an original keyword sequence based on the original keywords contained in the original audio data, and obtain pronunciation data corresponding to each original keyword from the original audio data.
[0053] In the embodiment of the present application, the audio data is converted into text using speech recognition technology. For example, it can be done by an existing speech recognition engine or tool. The result of speech recognition is a text sequence that contains the content of the audio. The original keywords are extracted from the text obtained by speech recognition by word frequency statistics, semantic analysis or other keyword extraction methods, and they are organized into an original keyword sequence.
[0054] The original audio data is segmented according to the position of the keyword in the text. Specifically, the start and end time of the audio segment corresponding to each keyword can be determined using a timestamp or other related audio processing technology. According to the segmented audio segments, the pronunciation data corresponding to each original keyword is extracted. Specifically, this can be achieved by cutting out the audio segments from the original audio data. The cut audio segments are the pronunciation data of each original keyword.
[0055] For example: there is a piece of original audio data, which contains some keywords, such as "apple", "banana", and "orange". First, perform speech recognition processing on the original audio data to identify the keywords contained therein. Arrange these keywords in the order in which they appear in the audio to construct an original keyword sequence, namely: "apple", "banana", and "orange". Next, obtain the pronunciation data corresponding to each original keyword from the original audio data. Through audio processing technology, the position where each keyword appears in the original audio data can be located. Then, an audio segment of a certain length is extracted from this position as the pronunciation data of the keyword. For the keyword "apple", find the time period where the word "apple" appears in the original audio data, and extract a segment of audio from this time period as the pronunciation data of "apple". Similarly, perform similar operations on "banana" and "orange" to obtain their pronunciation data respectively.
[0056] Step S103, modifying the original keyword sequence based on the pronunciation data to obtain a target keyword sequence.
[0057] In an embodiment of the present application, the collected pronunciation data is compared with the standard pronunciation data. The comparison may involve multiple aspects of speech, such as phonemes, intonations, speech speed, etc. According to the comparison results between the pronunciation data and the standard pronunciation data, problems or differences in the original keyword sequence are found. For example, if a certain sound in the pronunciation data is inconsistent with the standard pronunciation, the corresponding keyword may need to be corrected. After correction, a more accurate keyword sequence, i.e., the target keyword sequence, is obtained.
[0058] Step S104, extracting the melody features of the original audio data, and using the melody features and the target keyword sequence to retrieve the corresponding target song.
[0059] In an embodiment of the present application, first, the original audio data is preprocessed, including operations such as sampling, quantization and encoding of the audio, and the audio signal is converted into a digital form for subsequent analysis. Secondly, an audio feature extraction algorithm, such as pitch detection, melody contour analysis, rhythm analysis, etc., is used to extract the melody features in the audio data. These features may include the frequency, duration, intensity, direction of the melody, the pattern of the rhythm, etc. of the note. The extracted melody features are then described in a suitable mathematical representation, such as using vectors, matrices or other data structures to represent the numerical information of the melody features. In addition, the target keyword sequence is encoded or vectorized so as to be combined and matched with the melody features.
[0060] During the retrieval process, the extracted melody features and the encoded target keyword sequence are compared with the songs in the music library. Songs in the music library with similarity exceeding the threshold are retrieved and returned as similar target songs. The retrieved similar target songs are sorted according to similarity so that users can more easily find the most relevant songs.
[0061] The embodiment of the present application is for people with unclear articulation, such as children's humming, where there is inaccurate pronunciation and unclear articulation. By obtaining the pronunciation data corresponding to each original keyword and correcting the original keyword sequence based on these pronunciation data, the keyword deviation caused by inaccurate pronunciation can be corrected. Thereby improving the accuracy of subsequent retrieval of target songs. Then, by combining the target keyword sequence after the correction of the original keyword sequence and the melody features for retrieval, the user's humming intention can be understood more comprehensively and accurately, thereby improving the accuracy of finding the target song. And it can handle the humming styles and characteristics of different users, including situations such as non-standard pronunciation and unclear keywords, which increases the versatility of the embodiment of the present application.
[0062] Figure 2 is a flow chart of a method for recognizing humming audio according to an embodiment of the present invention, such as Figure 2 As shown, the process includes the following steps:
[0063] Step S201, obtaining original audio data of the user during humming.
[0064] In the embodiment of the present application, the humming process of the user can be collected through the vehicle's in-vehicle entertainment system to obtain original audio data. Specifically, in the operating system of the in-vehicle entertainment system, the user can turn on the audio recording function. For example, a button or icon for starting humming recording is designed on the interface of the in-vehicle entertainment system so that the user can easily start humming and recording. At the same time, some prompt information can also be provided, such as humming requirements, precautions, etc., to help the user obtain a better recording effect.
[0065] When the user clicks the button to start humming recording, the in-vehicle entertainment system starts to collect the original audio data of the user's humming through the microphone. The system stores the audio data in the form of digital signals, and can choose to temporarily store it in the memory of the in-vehicle system or in a local storage device. According to actual needs, the collected original audio data can be transmitted to the cloud server through the vehicle's communication module (such as Bluetooth, Wi-Fi or in-vehicle network) for further processing and analysis. For example, recognize the vocabulary, melody, pitch, rhythm and other information of the humming audio.
[0066] Step S202: construct an original keyword sequence based on the original keywords contained in the original audio data, and obtain pronunciation data corresponding to each original keyword from the original audio data.
[0067] In the embodiment of the present application, constructing an original keyword sequence based on the original keywords contained in the original audio data includes the following steps A1-A3:
[0068] Step A1, identifying the original audio data to obtain the humming sentence corresponding to the original audio data.
[0069] Specifically, the collected raw audio data is preprocessed, including noise reduction, background noise removal, audio segmentation and other operations to improve the accuracy of subsequent processing. Features such as Mel Frequency Cepstral Coefficients (MFCC) and Perceptual Linear Prediction (PLP) are extracted from the preprocessed raw audio data. These features can capture the acoustic characteristics of the audio. The raw audio data to be recognized is input into the trained model, and the model will output the prediction result of the humming sentence. For example, the user hums a simple melody "Do Re Mi Fa So La Xi". After the above steps and model recognition, the final recognition result is the humming sentence "Do Re Mi Fa So La Xi".
[0070] The model training process is as follows: a large amount of humming audio data and corresponding annotated text (humming sentences) are used to train a speech recognition model. The model can be a neural network model based on deep learning, such as a recurrent neural network (RNN), a long short-term memory network (LSTM), or a combination of a convolutional neural network (CNN) and a recurrent neural network.
[0071] Step A2, detecting the words contained in the humming sentence, and counting the occurrence frequency of each word in the humming sentence.
[0072] Specifically, speech recognition technology is used to convert the hummed sentences into text. The converted text is preprocessed, including removing special characters and punctuation marks, converting the text to lowercase, etc., so as to perform vocabulary analysis more accurately. The text is segmented into words using a lexical analyzer or a word segmentation tool to split the text into individual words. Then a data structure (such as a dictionary) is created to record the number of occurrences of each word. The segmented vocabulary list is traversed, and for each word, if it is already in the dictionary, its corresponding value is increased by 1; if it is not in the dictionary, it is added to the dictionary as a new key and its value is initialized to 1. Finally, the statistical results are output in an appropriate manner, such as printing out each word and its frequency of occurrence.
[0073] Step A3: taking words with an appearance frequency lower than a preset frequency as original keywords, and constructing an original keyword sequence based on the appearance order of the original keywords in the original audio data.
[0074] Specifically, the frequency of each word in the text is counted, and a frequency threshold is set as a preset frequency. Words with a frequency lower than the preset frequency are selected as original keywords. The original keywords are arranged in order according to the order in which they appear in the text converted from the original audio data to construct an original keyword sequence.
[0075] Step S203: modify the original keyword sequence based on the pronunciation data to obtain a target keyword sequence.
[0076] In the embodiment of the present application, the original keyword sequence is modified based on the pronunciation data to obtain the target keyword sequence, including the following steps B1-B3:
[0077] Step B1, obtaining a target standard pronunciation library, wherein the target standard pronunciation library includes a plurality of standard pronunciation data.
[0078] Specifically, obtaining the target standard pronunciation library includes: recognizing pronunciation data to obtain the language type corresponding to the pronunciation data; and determining the target standard pronunciation library corresponding to the language type based on a mapping relationship between a preset language type and the standard pronunciation library.
[0079] First, the pronunciation data is identified and analyzed to determine the language type corresponding to the pronunciation data. This may involve the use of speech recognition technology and related algorithms to determine the characteristics and patterns of the pronunciation, so as to determine which language it belongs to. Then, according to the mapping relationship between the language type and the standard pronunciation library set in advance, the target standard pronunciation library corresponding to the language type of the pronunciation data is found. In other words, each language type has a corresponding standard pronunciation library. Through this mapping relationship, the appropriate standard pronunciation library can be accurately found for the pronunciation data of a specific language type, so as to carry out subsequent comparison, evaluation or other related processing. For example, when it is identified that the pronunciation data belongs to English, according to the preset mapping relationship, it can be determined that the corresponding target standard pronunciation library is the standard pronunciation library of English. This ensures that the correct standard of the language is used when evaluating or correcting the pronunciation.
[0080] Step B2, comparing the pronunciation data with the standard pronunciation data, and obtaining abnormal data with confused pronunciation in the pronunciation data.
[0081] Specifically, the pronunciation data is compared with the standard pronunciation data to obtain abnormal data with pronunciation confusion in the pronunciation data, including: comparing the pronunciation data with the standard pronunciation data, and taking the pronunciation data that is different from the standard pronunciation data as candidate pronunciation data; matching the candidate pronunciation data with multiple pronunciation confusion types; if there is candidate pronunciation data that hits the pronunciation confusion type, then taking the candidate pronunciation data as abnormal data.
[0082] First, using speech recognition technology or related algorithms, the pronunciation data is compared with the standard pronunciation data phoneme or syllable by phoneme. For each compared phoneme or syllable, if there is a difference between it and the standard pronunciation data, such as differences in pitch, length, timbre, etc., it is marked as candidate pronunciation data. Determine the common pronunciation confusion types defined in advance, such as "nl confusion" ("south" and "blue"), "rl confusion" ("hot" and "music"), etc. in Chinese, and similar confusion types in other languages. For each candidate pronunciation data, its features are matched with the features of multiple defined pronunciation confusion types. If a candidate pronunciation data is consistent with the features of a certain pronunciation confusion type, that is, the pronunciation confusion type is hit, then this candidate pronunciation data is marked as abnormal data. For example, the standard pronunciation data is "south sky", and the pronunciation data is "blue sky". In the comparison process, it is found that there is a difference in the pronunciation of "south" and "blue", and "south" is marked as candidate pronunciation data. Then, "south" is matched with the pre-defined "nl confusion" type, and a hit is found, so the candidate pronunciation data of "south" is determined as abnormal data.
[0083] For example: the standard pronunciation data is set to "lai". After comparison, it is found that part of the pronunciation data is "nai", which is different from the standard pronunciation data "lai", so "nai" is used as candidate pronunciation data. Then, the candidate pronunciation data "nai" is matched with multiple pronunciation confusion types. For example, the set pronunciation confusion types are "nl confusion type" (such as "nai" and "lai"), "rl confusion type", etc. Since "nai" and "lai" meet the "nl confusion type", "nai" hits the pronunciation confusion type, so the "nai" part of the candidate pronunciation data is determined to be abnormal data. For another example, in another humming audio, the standard pronunciation data is "haore", and part of the pronunciation data is "haole". After comparison, "le" becomes the candidate pronunciation data. When matching with multiple pronunciation confusion types, the "rl confusion type" is hit, so "le" is also identified as abnormal data.
[0084] By comparing the pronunciation data with the standard pronunciation data, the embodiment of the present application can accurately find out the parts with differences, and provide a clear target for subsequent analysis and processing. Matching the candidate pronunciation data with multiple pronunciation confusion types helps to systematically classify and summarize pronunciation errors, and facilitates in-depth understanding of the patterns and laws of pronunciation problems. Once the abnormal data is determined, it can be corrected in a targeted manner to improve the accuracy of pronunciation.
[0085] Step B3, obtaining abnormal keywords corresponding to the abnormal data from the original keyword sequence, and correcting the abnormal keywords in the original keyword sequence to obtain a target keyword sequence.
[0086] Specifically, the original keyword sequence is analyzed to find data with pronunciation confusion, and these data are marked as abnormal data, and their corresponding abnormal keywords are determined. Then the abnormal keywords in the original keyword sequence are corrected to obtain the target keyword sequence. For example, some phonemes may lead to similar pronunciations in different words, such as "f" and "v", "s" and "z", "b" and "p", etc. When words with possible pronunciation confusion are found, they are marked as abnormal data and the corresponding abnormal keywords are determined. These abnormal keywords may be words that are misrecognized or recorded due to similar pronunciations.
[0087] Specifically, correcting abnormal keywords in an original keyword sequence to obtain a target keyword sequence includes: performing semantic recognition on original audio data to obtain semantic information; obtaining pronunciation confusion types corresponding to the abnormal keywords; and correcting the abnormal keywords in the original keyword sequence based on the semantic information and the pronunciation confusion types to obtain a target keyword sequence.
[0088] First, use natural language processing technology to convert and analyze the original audio data to extract semantic information. For example: convert the audio into text, and then use a text analysis algorithm to understand the meaning of the text. By analyzing semantic information, you can better understand the theme, context, and intent of the audio content, providing a basis for subsequent keyword corrections. Secondly, obtain the pronunciation confusion type corresponding to the abnormal keywords. Then, based on the semantic information and the pronunciation confusion type, correct the abnormal keywords in the original keyword sequence. For example, understand the context of the abnormal keywords based on semantic information, combine the pronunciation confusion type, infer the correct keywords, and replace the abnormal keywords with the corrected keywords to obtain the target keyword sequence. After determining the pronunciation confusion type of the abnormal keywords and the semantic information of the audio, you can start the correction work.
[0089] As an example, the context in which the abnormal keyword is located is understood based on semantic information. This can help determine the role and meaning of the keyword in the sentence. Then, combined with the pronunciation confusion type, the possible correct keyword is inferred. For example, if the pronunciation confusion type is "f" and "v", and the abnormal keyword is "fife", but it should be "vife" according to the context, then "fife" can be corrected to "vife". Finally, the abnormal keyword is replaced with the corrected keyword to obtain the target keyword sequence. This target keyword sequence can more accurately reflect the content of the original audio data.
[0090] The embodiment of the present application corrects abnormal keywords by combining semantic information and pronunciation confusion types, which can more accurately understand the true intention expressed by the user's humming, reduce misunderstandings caused by pronunciation errors, and thus greatly improve the accuracy of keyword recognition. For users who have difficulty in pronunciation, such as children, people with heavy accents or language barriers, their needs can be better met, more accurate song retrieval results can be provided, and the user experience can be improved. In addition, in the face of various non-standard pronunciation situations, the embodiment of the present application can make the system have stronger adaptability and fault tolerance through multi-dimensional analysis and correction, and is not easily disturbed by abnormal pronunciation.
[0091] Step S204, extracting the melody features of the original audio data, and using the melody features and the target keyword sequence to retrieve the corresponding target song.
[0092] In the embodiment of the present application, the corresponding target song is retrieved using the melody feature and the target keyword sequence, including the following steps C1-C4:
[0093] Step C1, generating a target simplified musical notation curve corresponding to the melody feature, wherein the abscissa of the target simplified musical notation curve is time, and the ordinate of the target simplified musical notation curve is the melody feature.
[0094] Specifically, audio analysis algorithms such as fast Fourier transform (FFT) and Mel frequency cepstral coefficients (MFCC) are used to extract melody features such as pitch, duration, and rhythm in the audio. According to the duration of the audio and the required accuracy, the time axis is evenly divided into several small segments. For example, if the audio duration is 10 seconds and the desired accuracy is 0.1 second, the time axis is divided into 100 small segments. Then the melody feature values extracted in each time segment are mapped to the corresponding ordinate position. For the pitch feature, it can be converted into the corresponding musical notation note and its position on the ordinate is determined. For the length and rhythm features, they can be reflected in the shape and slope of the curve through appropriate calculation or conversion methods. Finally, a drawing library (such as matplotlib, etc.) is used, with time as the horizontal axis and the melody feature value as the vertical axis, to draw a continuous curve point by point to form the target musical notation curve.
[0095] For example, in a certain time period, the extracted pitch feature corresponds to the "do" in the musical notation, which has a long duration and a steady rhythm. Then, at the horizontal axis position corresponding to the time point, the vertical axis will be marked as the corresponding height of "do", and the features of adjacent time points will be connected by a smooth curve to show the changing trend of the melody.
[0096] Step C2, using the target simplified musical notation curve to match the simplified musical notation curves of each song in the music library, to obtain at least one first candidate song that matches the target simplified musical notation curve.
[0097] Specifically, the target musical notation curve and the musical notation curves of songs in the music library are standardized, such as unifying the sampling frequency and removing noise, to ensure the quality and consistency of the data. Key features are extracted from the musical notation curve, such as the frequency, amplitude, and rhythm pattern of pitch changes. A suitable similarity measurement method is selected, such as the dynamic time warping (DTW) algorithm. The DTW algorithm can effectively compare the degree of distortion of two time series on the time axis, thereby measuring their similarity.
[0098] For each song in the music library, the similarity score between the simple musical notation curve and the target simple musical notation curve is calculated. If the similarity score exceeds a preset threshold, the song is marked as the first candidate song. The first candidate songs are sorted in descending order according to the similarity score. A certain number of songs ranked high are selected as the final first candidate songs.
[0099] Step C3, using the target keyword sequence to match the lyrics of each song in the music library, to obtain at least one second candidate song that matches the target keyword sequence.
[0100] Specifically, first, obtain the target keyword sequence. For example, the target keyword sequence is ["dream", "flying", "faraway"]. Then, traverse the lyrics of each song in the music library. When reading the lyrics of each song, perform some preprocessing, such as removing special characters, unifying uppercase and lowercase, etc., to improve the accuracy of matching. Next, use string matching algorithms, such as exact matching, fuzzy matching, or regular expression matching, to match the lyrics and the target keyword sequence. If the three keywords "dream", "flying" and "faraway" are found in the lyrics of a song at the same time, then this song will be marked as the second candidate song.
[0101] For example, the lyrics of song A are "We hold dreams in our hearts, spread our wings and fly far away", because it completely contains the target keyword sequence, so song A is determined as the second candidate song. For another example, the lyrics of song B are "Dreams are in our hearts, we long to go far away, and fly bravely", which also contains the target keyword sequence and is also identified as the second candidate song. In this way, after traversing all the songs in the music library, at least one second candidate song matching the target keyword sequence can be obtained.
[0102] Step C4, obtaining a target song from the first candidate song and the second candidate song.
[0103] Specifically, weights are set for melody feature matching and lyrics keyword matching respectively, for example, the melody feature matching weight is w1, the lyrics keyword matching weight is w2, and w1+w2=1. For each first candidate song, obtain its melody feature matching similarity score1 with the target simple score curve. If the song is also in the second candidate songs, obtain its lyrics keyword matching score2. The comprehensive score calculation formula is: total_score=w1×score1+w2×score2. For each second candidate song, if it is not in the first candidate songs, the default melody feature matching similarity score1 is 0. Obtain its lyrics keyword matching score2.
[0104] Calculate total_score according to the comprehensive score calculation formula. Compare the total_scores of all first candidate songs and second candidate songs. Select the song with the highest total_score as the target song that is closest to the original audio data.
[0105] As an example, first, the original audio data is analyzed to extract the melody features, such as pitch, length, rhythm, etc. Then, with time as the horizontal coordinate and the extracted melody feature value as the vertical coordinate, a continuous curve is drawn to generate a target musical notation curve. The musical notation curve data of each song in the music library is read in sequence. For each song, the similarity between its musical notation curve and the target musical notation curve is calculated. A variety of algorithms can be used to calculate the similarity, such as similarity measurement based on curve shape, feature point matching, etc. The songs with similarity higher than the set threshold are marked as the first candidate songs. For example, the musical notation curve of song A is highly similar to the target musical notation curve in key feature points and overall shape. After calculating the similarity, if 80% (assuming the threshold is 70%) is achieved, song A becomes the first candidate song.
[0106] Then obtain the target keyword sequence, such as "missing", "hometown", "moonlight". Then, read the lyrics text of each song in the music library. Perform preprocessing on the lyrics text, such as word segmentation and stop word removal. Through keyword matching, determine whether the lyrics of each song contain all the keywords in the target keyword sequence. If included, the song is marked as the second candidate song. For example, the lyrics of song B contain "deep longing, distant hometown, and shining moonlight", which contains all the target keywords, so song B becomes the second candidate song.
[0107] Comprehensively consider the first candidate song and the second candidate song. You can set weights for melody feature matching and keyword matching respectively, for example, the melody feature matching weight is 0.6 and the keyword matching weight is 0.4. For songs that belong to both the first candidate song and the second candidate song, calculate their comprehensive scores. Comprehensive score = melody feature matching similarity × 0.6 + keyword matching × 0.4. Select the song with the highest comprehensive score as the target song that is closest to the original audio data.
[0108] Assuming that song C is both the first candidate song with a melody feature matching similarity of 75% and the second candidate song with a keyword matching degree of 80%, its comprehensive score = 75% × 0.6 + 80% × 0.4 = 77%. If song C has the highest comprehensive score among all candidate songs, then song C is the final target song.
[0109] The method provided in the embodiment of the present application can analyze and match the original audio data from multiple dimensions by considering the melody features and lyrics keywords at the same time, so as to find the target song closest to the original audio data more comprehensively and accurately. It does not rely solely on a single factor, reducing the possibility of misjudgment and omission. For example, there may be two songs with similar lyrics but different tunes, or similar tunes but different lyrics. Comprehensive consideration can avoid inaccurate results caused by matching only on one side. The search process is divided into two parts: melody feature matching and lyrics keyword matching. First, the first candidate song and the second candidate song are obtained respectively, and then screened from the two, which can narrow the final search range and improve the efficiency of obtaining the target song.
[0110] In addition, the user's memory and description of music may focus on the melody or lyrics. The embodiment of the present application takes both situations into account, which can better meet the needs of different users and improve the user experience. For example, some users hum a melody but cannot remember the lyrics. They can find related songs through melody feature matching; while some users only remember a few key words in the lyrics, and they can also find possible songs through keyword matching.
[0111] In the present embodiment, a humming audio recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" may implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.
[0112] This embodiment provides a humming audio recognition device, such as Figure 3 As shown, including:
[0113] An acquisition module 301 is used to acquire original audio data of the user during humming;
[0114] The recognition module 302 is used to recognize the original keyword sequence contained in the original audio data and obtain the pronunciation data corresponding to the original keyword sequence from the original audio data;
[0115] A correction module 303, used to correct the original keyword sequence based on the pronunciation data to obtain a target keyword sequence;
[0116] The retrieval module 304 is used to extract the melody features of the original audio data and retrieve the corresponding target song using the melody features and the target keyword sequence.
[0117] In the embodiment of the present application, the recognition module 302 is used to recognize the original audio data to obtain the humming sentence corresponding to the original audio data; detect the words contained in the humming sentence, and count the frequency of occurrence of each word in the humming sentence; use the words with an occurrence frequency lower than the preset frequency as the original keywords, and construct the original keyword sequence based on the appearance order of the original keywords in the original audio data.
[0118] In the embodiment of the present application, the correction module 303 includes:
[0119] An acquisition submodule is used to acquire a target standard pronunciation library, wherein the target standard pronunciation library includes a plurality of standard pronunciation data;
[0120] A comparison submodule is used to compare the pronunciation data with the standard pronunciation data to obtain abnormal data with pronunciation confusion in the pronunciation data;
[0121] The processing submodule is used to obtain abnormal keywords corresponding to abnormal data from the original keyword sequence, and correct the abnormal keywords in the original keyword sequence to obtain a target keyword sequence.
[0122] In the embodiment of the present application, the acquisition submodule is used to identify the pronunciation data and obtain the language type corresponding to the pronunciation data; based on the mapping relationship between the preset language type and the standard pronunciation library, the target standard pronunciation library corresponding to the language type is determined.
[0123] In an embodiment of the present application, a comparison submodule is used to compare pronunciation data with standard pronunciation data, and to use pronunciation data that is different from the standard pronunciation data as candidate pronunciation data; to match the candidate pronunciation data with multiple pronunciation confusion types; and if there is candidate pronunciation data that hits a pronunciation confusion type, the candidate pronunciation data is used as abnormal data.
[0124] In an embodiment of the present application, a processing submodule is used to perform semantic recognition on original audio data to obtain semantic information; obtain pronunciation confusion types corresponding to abnormal keywords; and correct abnormal keywords in the original keyword sequence based on the semantic information and pronunciation confusion types to obtain a target keyword sequence.
[0125] In an embodiment of the present application, the retrieval module 304 is used to generate a target simplified musical notation curve corresponding to the melody feature, wherein the horizontal coordinate of the target simplified musical notation curve is time, and the vertical coordinate of the target simplified musical notation curve is the melody feature; the target simplified musical notation curve is matched with the simplified musical notation curves of each song in the music library to obtain at least one first candidate song that matches the target simplified musical notation curve; the target keyword sequence is matched with the lyrics of each song in the music library to obtain at least one second candidate song that matches the target keyword sequence; and the target song is obtained from the first candidate song and the second candidate song.
[0126] See also Figure 4 , Figure 4 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.
[0127] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0128] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0129] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of an electronic device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0130] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory. The electronic device also includes a communication interface 30 for the electronic device to communicate with other devices or a communication network.
[0131] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0132] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A method for recognizing humming audio, characterized in that: The method comprises: Obtaining the original audio data of the user during humming; Constructing an original keyword sequence based on the original keywords contained in the original audio data, and obtaining pronunciation data corresponding to each of the original keywords from the original audio data; Modifying the original keyword sequence based on the pronunciation data to obtain a target keyword sequence; The melody features of the original audio data are extracted, and the corresponding target song is retrieved using the melody features and the target keyword sequence.
2. The method according to claim 1, characterized in that The step of constructing an original keyword sequence based on the original keywords contained in the original audio data includes: Recognize the original audio data to obtain a humming sentence corresponding to the original audio data; Detecting the words contained in the humming sentence, and counting the frequency of occurrence of each word in the humming sentence; The words with an appearance frequency lower than a preset frequency are used as the original keywords, and the original keyword sequence is constructed based on the appearance order of the original keywords in the original audio data.
3. The method according to claim 1, characterized in that: The step of modifying the original keyword sequence based on the pronunciation data to obtain a target keyword sequence includes: Acquire a target standard pronunciation library, wherein the target standard pronunciation library includes a plurality of standard pronunciation data; Comparing the pronunciation data with the standard pronunciation data to obtain abnormal data with confused pronunciation in the pronunciation data; Abnormal keywords corresponding to abnormal data are obtained from the original keyword sequence, and the abnormal keywords in the original keyword sequence are corrected to obtain the target keyword sequence.
4. The method according to claim 3, characterized in that The step of obtaining a target standard pronunciation library comprises: Identify the pronunciation data and obtain the language type corresponding to the pronunciation data; Based on the mapping relationship between the preset language type and the standard pronunciation library, a target standard pronunciation library corresponding to the language type is determined.
5. The method according to claim 3, characterized in that: The comparing the pronunciation data with the standard pronunciation data to obtain abnormal data with pronunciation confusion in the pronunciation data includes: Comparing the pronunciation data with the standard pronunciation data, and taking the pronunciation data that is different from the standard pronunciation data as candidate pronunciation data; matching the candidate pronunciation data with a plurality of pronunciation confusion types; If there is any candidate pronunciation data that matches the pronunciation confusion type, the candidate pronunciation data is used as the abnormal data.
6. The method according to claim 3, characterized in that The correcting abnormal keywords in the original keyword sequence to obtain the target keyword sequence includes: Performing semantic recognition on the original audio data to obtain semantic information; Obtaining a pronunciation confusion type corresponding to the abnormal keyword; The abnormal keywords in the original keyword sequence are corrected based on the semantic information and the pronunciation confusion type to obtain the target keyword sequence.
7. The method according to claim 1, characterized in that The method of retrieving a corresponding target song by using the melody feature and the target keyword sequence includes: Generate a target simplified musical notation curve corresponding to the melody feature, wherein the abscissa of the target simplified musical notation curve is time, and the ordinate of the target simplified musical notation curve is the melody feature; Matching the target simplified musical notation curve with the simplified musical notation curves of each song in the music library to obtain at least one first candidate song that matches the target simplified musical notation curve; Using the target keyword sequence to match the lyrics of each song in the music library, obtaining at least one second candidate song that matches the target keyword sequence; The target song is obtained from the first candidate songs and the second candidate songs.
8. A humming audio recognition device, characterized in that: The device comprises: An acquisition module is used to acquire the original audio data of the user during the humming process; A recognition module, configured to construct an original keyword sequence based on the original keywords contained in the original audio data, and obtain pronunciation data corresponding to each original keyword from the original audio data; A correction module, used for correcting the original keyword sequence based on the pronunciation data to obtain a target keyword sequence; The retrieval module is used to extract the melody features of the original audio data and retrieve the corresponding target song using the melody features and the target keyword sequence.
9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.