A method for converting human vocal humming into simplified musical notation and guzheng melody based on the YIN algorithm

Abstract

This invention relates to a method for converting human humming into simplified musical notation and guzheng melodies based on the YIN algorithm, belonging to the field of audio processing and intelligent music composition strategies. The method includes the following steps: S1: Obtaining audio feature value information and rhythm information of the humming audio using the YIN method; S2: Converting existing piano spectrum information into a piano spectrum; S3: Establishing a mapping relationship between the piano spectrum and guzheng simplified musical notation, converting the piano spectrum into the corresponding guzheng simplified musical notation; S4: Filtering the guzheng simplified musical notation to obtain the filtered guzheng simplified musical notation, and calculating the start position information and duration information of each note; S5: Constructing the melody corresponding to the guzheng instrument, and adjusting the melody according to adjacent notes; S6: Optimizing the melody corresponding to the guzheng instrument, and outputting the final guzheng audio. This invention can automatically convert human humming into guzheng instrument melodies, enriching the melodic content and ensuring auditory comfort.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for converting vocal humming into a simplified score and a zither melody based on YIN algorithm, belonging to the field of audio processing and intelligent arrangement strategy, and is especially suitable for converting vocal humming into a simplified score and a zither melody based on YIN algorithm. BACKGROUND

[0002] As a traditional Chinese musical instrument, the zither is known as the king of Chinese folk music and is an important part of Chinese traditional music, with a long history, rich artistic connotations and profound cultural heritage. However, due to its unique 21-string structure and five-tone score characteristics, its international promotion needs to be improved. According to the vocal humming audio, the corresponding simplified score and zither performance music are automatically generated, and the pitch running problem in the vocal humming is corrected according to the characteristics of the zither. This method can record the randomly generated musical inspiration, greatly enrich people's life content and improve people's life quality. At the same time, it promotes the spread and influence of Chinese traditional zither culture in the world.

[0003] Compared with traditional vocal humming score conversion, the current score conversion is mostly converted into a five-line score, which is only suitable for professionals with certain music theory. However, the present method directly converts into a simplified score, which is more suitable for the general public. At the same time, the vocal humming is corrected according to the special characteristics of the zither glissando, shaking sound, playing sound and piper sound to perfect the melody. In addition, the traditional score conversion method needs to find the melody beat and divide it before processing. This method does not need strict beat division, and the corresponding zither music can be constructed.

[0004] Invention patent CN103824565A proposes a "humming score recognition method based on note and time value modeling", which uses statistical methods to realize accurate humming score recognition through noise-resistant pitch feature extraction, note pitch model set and note time value model set parameter training, and music theory information decoding recognition. With the development of artificial intelligence, big data analysis and processing technology, and automatic arrangement, the cross-disciplinary intelligent arrangement based on artificial intelligence and music composition gradually enters the public view. Alain proposed YIN algorithm for pitch estimation in the article "YIN, a fundamental frequency estimator for speech and music", which realizes the extraction of pitch features related to fundamental frequency. The pitch feature is also the pitch, that is, the tone. However, due to the characteristics of the human auditory nerve that cannot distinguish sounds within 0.1 seconds, and the next sound within 0.1 seconds is the reinforced sound of the previous sound, the audio information obtained by this algorithm has low recognition and does not conform to human auditory perception, and it cannot realize the conversion of simplified score and zither melody. SUMMARY

[0005] Therefore, the present application provides a method for converting a vocal humming score into a guzheng melody based on a YIN algorithm, which aims to filter and screen each frame of audio data information in the audio information extracted by the YIN method, calculate monophonic information, then call corresponding guzheng audio sources, and construct a corresponding guzheng instrument melody of the vocal humming according to the characteristics and laws of the guzheng.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions.

[0007] A method for converting a vocal humming score into a guzheng melody based on a YIN algorithm, characterized in that it comprises the following steps:

[0008] S1: processing the vocal humming audio by using the YIN method to obtain the audio characteristic value information of the humming audio, and simultaneously obtaining the beat information of the humming audio;

[0009] S2: converting the extracted audio characteristic value information into a piano frequency spectrum according to the corresponding existing piano frequency spectrum information;

[0010] S3: establishing a mapping relationship between the piano frequency spectrum and the guzheng score, and converting the piano frequency spectrum into the corresponding guzheng score;

[0011] S4: filtering the information in the guzheng score that does not meet the requirements according to the characteristics of the human auditory nervous system that cannot distinguish sounds within 0.1 seconds, obtaining the filtered guzheng score, and calculating the monophonic start position information and monophonic duration information;

[0012] S5: constructing the corresponding melody of the guzheng instrument according to the filtered guzheng score and the corresponding relationship between the guzheng score and the guzheng source data information in the database, and adjusting the corresponding melody of the guzheng instrument according to the adjacent tones;

[0013] S6: optimizing the corresponding melody of the guzheng instrument according to the obtained beat information, and outputting the final guzheng audio.

[0014] Further, the YIN method is to estimate the period of the single frame pitch, set an absolute threshold, search for the optimal value of the period in the vicinity of the period value, and further obtain the optimal frequency of the single frame pitch, i.e. the audio characteristic value.

[0015] Further, the step S1 is specifically:

[0016] S101: calling the YIN method in the librosa library to frame the vocal humming audio with equal time length, and pre-processing the vocal humming audio frame by frame to obtain the audio characteristic value information librosa.yin of the humming audio and store it in the pitch array frame by frame;

[0017] S102: Call the librosa.beat.beat_track and librosa.frames_to_time functions in the librosa library to obtain the beat position information of the humming audio and store it in the beats__position array.

[0018] Preferably, the sampling duration of the equal-duration frame segmentation described in S101 is generally, but not limited to, about 0.023 seconds per frame, and the specific duration is related to the sampling device; the higher the sampling rate, the clearer the sound quality.

[0019] Furthermore, step S2 specifically includes:

[0020] Based on the audio feature values ​​in the pitch array, the map_pitch_to_piano_notes function in the music21 library is called to match the existing piano spectrum information, convert it into a piano spectrum, and store it in the corresponding pitch array.

[0021] Among them, music21 is a powerful Python library for computational musicology analysis developed by MIT.

[0022] Furthermore, the mapping relationship between the piano spectrum and the guzheng notation described in step S3 is shown in Table 1. Due to the limitation that the highest pitch is only double the octave, the pitches in the pitch array of the piano spectrum obtained in step S2 that exceed the pitch range are processed by lowering the pitch by one octave. At the same time, since there are no semitone keys in the guzheng like in the piano, semitones that are close to the guzheng pitch are converted into similar guzheng notation pitches, and the melody is enriched later by ascending and descending glissando.

[0023] Table 1. Mapping Relationship between Piano Spectrum and Guzheng Simplified Musical Notation

[0024]

[0025]

[0026] Furthermore, step S4 specifically includes:

[0027] S401: Using the dimension m of the pitch array and the duration t of the human humming audio, calculate the duration of each frame of audio information: duration = t / m;

[0028] S402: Calculate the smallest positive integer k such that k·duration ≥ 0.1 seconds;

[0029] S403: Starting from i=1, for the guzheng sheet music in the pitch array of frame i, if there is a guzheng sheet music in any frame from frame i+1 to frame i+k that is the same as or very close to the guzheng sheet music in frame i, then set the guzheng sheet music in frames i+1 to i+k in the pitch array to the guzheng sheet music in frame i. At the same time, move the pointer to frame i+k and record the guzheng sheet music information in the array simpleGZ. The starting position of the note is stored in gz_position, and the duration of the note is stored in the array time_duration. Otherwise, set the guzheng sheet music in frame i in the pitch array to a rest, and record the rest-related information in the array simpleGZ, the starting position in the array gz_position, and the duration in the array time_duration. At the same time, move the pointer to frame i+1.

[0030] S404: Execute step S403 until the pointer moves to nk. At this time, if there is a guzheng score in any frame from frame n-k+1 to frame n that is the same as or very close to the score of frame i, then set all the guzheng scores in the pitch array from frame n-k+1 to frame n to the guzheng score in the pitch array of frame nk, and execute step S405; otherwise, set all the guzheng scores in the pitch array from frame n-k+1 to frame n to rests and record the rest-related information in the array simpleGZ, the start position in the data gz_position, and the duration in time_duration, and execute step S405.

[0031] S405: Iterate through the guzheng sheet music information in the simpleGZ array, merge c consecutive identical audio notes with a duration of less than 0.1 seconds in between into a single note, calculate the start position and duration of the single note, and obtain the filtered guzheng sheet music. The updated sheet music information is stored in the array new_sample_jp.

[0032] Among them, the extremely close guzheng sheet music refers to the guzheng sheet music corresponding to pitches with a pitch difference of half a pitch difference or less.

[0033] Furthermore, step S5 specifically involves: based on the correspondence between the guzheng simplified score and the guzheng sound source data information in the database, converting the single notes of the filtered guzheng simplified score according to the corresponding guzheng sound source to construct the melody corresponding to the guzheng instrument; then, according to Table 2, traversing the filtered guzheng simplified score note by note, adjusting the adjacent single notes that meet the conditions. Here, Pitch[i] is the frequency of the i-th single note, min_duration is the shortest duration of a single note in the entire vocal humming audio, and max_duration is the longest duration of a single note in the entire vocal humming audio.

[0034] Table 2 shows the correspondence between melodies of guzheng instruments adjusted by adjacent monotones.

[0035]

[0036] Due to the seven-tone characteristics of the guzheng—Gong, Shang, Jiao, Zhi, Yu, Bian Gong, Bian Zhi—the up / down glissando adjustment in the table does not include the glissando fa (4) gliding up to the Zhi (5) and the glissando si (7) gliding up to the Gong (1) in a higher register. It also does not include the glissando suo (5) gliding down to the glissando fa (4) and the Gong (1) gliding down to the glissando si (7) in a lower register.

[0037] Furthermore, step S6 specifically involves: determining the starting position of a measure based on the acquired rhythm information, adjusting the first note of each measure to be the accent, optimizing the melody corresponding to the guzheng instrument, and outputting the final guzheng audio.

[0038] The beneficial effects of this invention are as follows: This invention provides a method for converting human vocal humming into simplified musical notation and guzheng melodies based on the YIN algorithm. It considers the unique characteristics of the human auditory nervous system in recognizing sound, as well as the uniqueness of the guzheng instrument's musical composition based on the five notes of the pentatonic scale (Gong, Shang, Jiao, Zhi, Yu, Bian Gong, Bian Zhi). First, it utilizes the audio feature value data extracted using the YIN method. Then, it filters out audio that does not conform to human auditory perception. Finally, based on the conversion relationship and the characteristics of the guzheng, it obtains the corresponding guzheng instrument melody for the human vocal humming. This invention can automatically convert human vocal humming into guzheng instrument melodies, enriching the melodic content and ensuring auditory comfort. Attached Figure Description

[0039] To illustrate the objectives and technical solutions of this invention, the following figures are provided:

[0040] Figure 1 This is a flowchart of a method for converting human vocal humming into simplified musical notation and guzheng melody based on the YIN algorithm.

[0041] Figure 2This is a diagram of the guzheng simplified musical score obtained in step three of this embodiment of the invention; where the horizontal axis represents time in seconds and the vertical axis represents the simplified musical score.

[0042] Figure 3 This is a graph showing the filtered Guzheng simplified musical score obtained in step four of this embodiment of the invention; where the horizontal axis represents time in seconds; and the vertical axis represents the simplified musical score.

[0043] Figure 4 The image shows the optimized musical score obtained in step six of this embodiment of the invention; where the horizontal axis represents time in seconds and the vertical axis represents simplified musical notation. Detailed Implementation

[0044] The following will be combined with the appendix Figure 1 The preferred embodiments of the present invention will be described in detail below.

[0045] Example: Taking a clip from "A Chinese Ghost Story" hummed in a park as an example, the audio is stored as a WAV file at path D: / 2023 / AIArrangement of Music / test songs / qnyhLY.wav. This audio file needs to be processed to generate the corresponding guzheng melody. Since hummed melodies can be inaccurate, this invention provides a "method for converting hummed melodies into simplified musical notation and guzheng melodies based on the YIN algorithm," combined with... Figure 1 It includes the following steps:

[0046] Step 1: Using a Python development environment, call the humming audio file audio_file = 'D: / 2023 / AIArrangement of Music / test songs / qnyhLY.wav', call the librosa library function YIN to divide the humming audio into equal-length frames, and preprocess the humming audio frame by frame to extract audio feature value information to obtain the array pitch = librosa.yin(audio_data, fmin = 50, fmax = 300, sr = sample_rate), and obtain the beat position beat_positions = get_beat_positions(audio_file), and the audio length information audio_length = get_audio_length(audio_file).

[0047] Step 2: Based on the extracted audio feature value information and the corresponding existing piano spectrum information, convert it into a piano spectrum.

[0048] Specifically, based on the audio feature value information in the pitch array, the map_pitch_to_piano_notes function in the music21 library is called to correspond with the existing piano spectrum information, convert it into a piano spectrum, and store it in the corresponding pitch array.

[0049] Step 3: Establish a mapping relationship between piano spectrum and guzheng sheet music, and convert the piano spectrum into the corresponding guzheng sheet music.

[0050] Specifically, the mapping relationship between the piano spectrum and the guzheng notation is shown in Table 1. Due to the limitation that the highest pitch is only double the duo, the spectrum in the pitch array of the piano spectrum obtained in step two that exceeds the pitch range is processed by lowering the pitch by one octave. At the same time, since the guzheng does not have the semitone keys of the piano, semitones that are close to the guzheng pitch are converted into similar guzheng notation pitches, and the melody is enriched later by ascending and descending glissando.

[0051] Step 4: Based on the characteristic that the human auditory nerve system cannot distinguish intervals within 0.1 seconds, filter out information in the guzheng simplified score that does not meet the requirements to obtain the filtered guzheng simplified score, and calculate the start position information and duration information of the single note.

[0052] Based on the algorithm in Table 3, the specific steps are as follows:

[0053] S401: Using the dimension m of the pitch array and the duration t of the human humming audio, calculate the duration of each frame of audio information: duration = t / m;

[0054] S402: Calculate the smallest positive integer k such that k·duration ≥ 0.1 seconds;

[0055] S403: Starting from i=1, for the guzheng sheet music in the pitch array of frame i, if there is a guzheng sheet music in any frame from frame i+1 to frame i+k that is the same as or very close to the guzheng sheet music in frame i, then set the guzheng sheet music in frames i+1 to i+k in the pitch array to the guzheng sheet music in frame i. At the same time, move the pointer to frame i+k and record the guzheng sheet music information in the array simpleGZ. The starting position of the note is stored in gz_position, and the duration of the note is stored in the array time_duration. Otherwise, set the guzheng sheet music in frame i in the pitch array to a rest, and record the rest-related information in the array simpleGZ, the starting position in the array gz_position, and the duration in the array time_duration. At the same time, move the pointer to frame i+1.

[0056] S404: Execute step S403 until the pointer moves to nk. At this time, if there is a guzheng score in any frame from frame n-k+1 to frame n that is the same as or very close to the score of frame i, then set all the guzheng scores in the pitch array from frame n-k+1 to frame n to the guzheng score in the pitch array of frame nk, and execute step S405; otherwise, set all the guzheng scores in the pitch array from frame n-k+1 to frame n to rests and record the rest-related information in the array simpleGZ, the start position in the data gz_position, and the duration in time_duration, and execute step S405.

[0057] S405: Iterate through the guzheng sheet music information in the simpleGZ array, merge c consecutive identical audio notes with a rest duration shorter than 0.1 seconds into a single note, calculate the start position and duration of the single note, and obtain the filtered guzheng sheet music. The updated sheet music information is stored in the array new_sample_jp. For the note from simpleGZ[i] to simpleGZ[j], we have: gz_position[i] is the start position of the single note, updated and stored in the array new_time_position, and gz_position[j] - gz_position[i] + time_duration[j] is the duration of the single note, updated and stored in the array new_time.duration.

[0058] Table 3. Partial algorithm illustrations for steps S403 to S405.

[0059]

[0060] Among them, the extremely close guzheng sheet music refers to the guzheng sheet music corresponding to pitches with a pitch difference of half a pitch difference or less.

[0061] Step 5: Based on the correspondence between the guzheng simplified score and the guzheng sound source data in the database, construct the melody corresponding to the guzheng instrument from the filtered guzheng simplified score, and adjust the melody corresponding to the guzheng instrument according to the adjacent notes.

[0062] Specifically, based on the correspondence between the simplified guzheng score and the guzheng sound source data in the database, the single notes of the filtered simplified guzheng score are converted according to the corresponding guzheng sound source to construct the melody corresponding to the guzheng instrument. Then, as shown in Table 2, the filtered simplified guzheng score is traversed one by one, and the adjacent single notes that meet the conditions are adjusted.

[0063] Step Six: Based on the obtained rhythm information, optimize the melody corresponding to the guzheng instrument and output the final guzheng audio.

[0064] Specifically, based on the acquired rhythm information, the starting position of each measure is determined, and the first note of each measure is adjusted to be the accented note. The melody corresponding to the guzheng instrument is optimized, and the final guzheng audio is output.

[0065] To better demonstrate the method of this invention, this embodiment shows the guzheng simplified musical score obtained in step three, the filtered guzheng simplified musical score obtained in step four, and the optimized melody score for the guzheng instrument obtained in step six, as shown below. Figure 1 , Figure 2 and Figure 3 As shown, the method of the present invention optimizes the melody while preserving it, making the melody smoother and more comfortable for the human ear.

[0066] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A method for converting a human voice humming score and a guzheng melody based on a YIN algorithm, characterized in that, The method comprises the following steps: S1: processing the human voice humming audio by using the YIN method to obtain the audio feature value information of the humming audio, and simultaneously obtaining the beat information of the humming audio; S2: converting the audio feature value information extracted and the corresponding existing piano spectrum information into a piano spectrum; S3: establishing a mapping relationship between the piano spectrum and the guzheng tablature, and converting the piano spectrum into the corresponding guzheng tablature; S4: filtering the information in the guzheng tablature that does not meet the requirements according to the characteristic that the human auditory nervous system cannot distinguish the interval sound within 0.1 seconds, obtaining the filtered guzheng tablature, and calculating the single sound starting position information and the single sound duration information; S5: constructing the guzheng instrument corresponding melody according to the filtered guzheng tablature and the corresponding relationship between the guzheng tablature and the guzheng sound source data information in the database, and adjusting the guzheng instrument corresponding melody according to the adjacent tone; S6: optimizing the guzheng instrument corresponding melody according to the obtained beat information, and outputting the final guzheng audio; The step S4 is specifically: S401: calculating the duration of each frame of audio information by using the dimension m of the pitch array and the time t of the human voice humming audio, duration=t / m; S402: calculating the smallest positive integer k, such that k·duration≥0.1 seconds; S403: starting the pointer from i=1, if there is any frame of guzheng tablature corresponding to the i+1th frame to the i+kth frame that is the same or extremely close to the i frame of guzheng tablature in the pitch array, then setting the guzheng tablature of the i+1th frame to the i+kth frame in the pitch array as the guzheng tablature of the i frame in the pitch array, moving the pointer to the i+kth frame and recording the guzheng tablature information in the array simpleGZ, the starting position of the single sound is saved in the array gz_position, and the duration corresponding to the single sound is saved in the array time_duration; otherwise, setting the guzheng tablature of the i frame in the pitch array as a rest sound, recording the rest sound related information in the array simpleGZ, the starting position in the data gz_position, and the duration in the time_duration, and moving the pointer to the i+1th frame; S404: executing step S403 until the pointer moves to n-k, at this time, if there is any frame of guzheng tablature corresponding to the n-k+1th frame to the nth frame that is the same or extremely close to the i frame of guzheng tablature, then setting the guzheng tablature of the n-k+1th frame to the nth frame in the pitch array as the guzheng tablature of the n-kth frame in the pitch array, and executing step S405; otherwise, setting the guzheng tablature of the n-k+1th frame to the nth frame in the pitch array as a rest sound and recording the rest sound related information in the array simpleGZ, the starting position in the data gz_position, and the duration in the time_duration, and executing step S405; S405: Traverse the guzheng tablature information in the simpleGZ array, recombine c audio with the same continuous and intermediate rest duration shorter than 0.1 seconds into a single tone, calculate the starting position and duration of the single tone, and obtain the filtered guzheng tablature updated tablature information. The updated tablature information is saved in the array new_sample_jp.

2. The method for converting the Yin algorithm-based humming melody of a human voice into a simplified score and a guzheng melody according to claim 1, characterized in that, The step S1 is specifically: S101: Call the YIN method in the librosa library to perform equal-length framing on the vocal humming audio, and pre-process the vocal humming audio frame by frame to obtain the audio feature value information librosa.yin of the humming audio and store it in the pitch array frame by frame; S102: Call the librosa.beat.beat_track function and librosa.frames_to_time function in the librosa library to obtain the beat information of the humming audio and store it in the beats_position array.

3. The method for converting the Yin algorithm-based humming melody into a simplified score and a guzheng melody according to claim 1, characterized in that, Step S2 is specifically: According to the audio feature value information in the pitch array, the map_pitch_to_piano_notes function in the music21 library is called to correspond to the existing piano frequency spectrum information, convert it into a piano frequency spectrum, and store it in the corresponding pitch array.

4. The method for converting the Yin algorithm-based humming melody into a simplified score and a guzheng melody according to claim 1, characterized in that, The mapping relationship between the piano frequency spectrum and the guzheng tablature in step S3 is to process the frequency spectrum in the pitch array of the piano frequency spectrum obtained in step S2 that exceeds the pitch range by lowering one octave; at the same time, because there is no semitone key in the guzheng, the semitone close to the guzheng sound is converted into the close guzheng tablature sound, and the melody is enriched through up and down glissando in the later stage.

5. The method for converting the Yin algorithm-based humming melody into a simplified score and a guzheng melody according to claim 1, characterized in that, The extremely close guzheng tablature is the guzheng tablature corresponding to the pitch difference within half a pitch difference.

6. The method for converting the Yin algorithm-based humming melody into a simplified score and a guzheng melody according to claim 1, characterized in that, According to the corresponding relationship between the guzheng tablature and the guzheng sound source data information in the database, the single tone of the filtered guzheng tablature is converted according to the corresponding guzheng sound source to construct the corresponding melody of the guzheng instrument, and then the filtered guzheng tablature is traversed single tone by single tone. Adjust the adjacent single tones that meet the conditions.

7. The method for converting the Yin algorithm-based humming melody into a simplified score and a guzheng melody according to claim 1, characterized in that, The step S6 is specifically: According to the obtained beat information, determine the start position of the measure, and adjust the first tone of each measure to be the accented tone, optimize the corresponding melody of the guzheng instrument, and output the final guzheng audio.

Images