Music generation method and device and storage medium

Abstract

The invention discloses a music generation method and device and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring melody audio, and extracting music information of the melody audio; sequentially traversing each note in the melody audio, and screening out a first sound source matched with the pitch of the current note from a sound source database of the target type of musical instrument according to the note information of the current note; screening out a second sound source from the first sound source according to a playing technique rule corresponding to the target type of musical instrument based on the note information and the note context information of the current note; the playing technique rule comprises a corresponding relation between playing fingering and notes; determining a matched sound source corresponding to the current note according to the second sound source; and according to the matched sound source corresponding to each note in the melody audio and the melody information, generating music of a target type of musical instrument corresponding to the melody audio. The musical instrument music is generated by combining fingering factors, so that the finally synthesized audio is more natural and accords with the actual playing characteristics of the musical instrument.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a music generating method, device and storage medium. Background Art

[0002] In order to enrich the content of music, especially for music of a specific instrument, music played by the instrument is often automatically generated through music generation. In the prior art, instrument timbre synthesis is performed by digital signal processing or machine learning, specifically by obtaining the fundamental frequency sequence of the melody to be synthesized, then modeling the timbre of the instrument, extracting the timbre features, and finally synthesizing the obtained fundamental frequency sequence and timbre features to obtain the audio file of the instrument performance. However, the synthesized music is stiff and unnatural, and parameter-based timbre modeling requires a lot of time and model training costs. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a music generation method, device and storage medium, which can make the final synthesized music more natural and in line with the actual performance characteristics of musical instruments. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a music generation method, comprising:

[0005] Acquire melody audio, and extract music information of the melody audio; the music information includes note information of different notes in the melody audio, note context information, and melody information;

[0006] Traversing each note in the melody audio in turn, and selecting a first sound source that matches the pitch of the current note from a sound source database of a target type of musical instrument according to the note information of the current note;

[0007] Based on the note information and note context information of the current note, and in accordance with the playing technique rules corresponding to the target type of musical instrument, a second sound source is selected from the first sound source; the playing technique rules include the corresponding relationship between playing fingerings and notes;

[0008] Determine a matching sound source corresponding to the current note according to the second sound source;

[0009] According to the matching sound source corresponding to each note in the melody audio and the melody information, music of a target type instrument corresponding to the melody audio is generated.

[0010] Optionally, the playing technique rules include a first technique rule representing the correspondence between the target position of a note in a phrase or a beat and the fingering, and a second technique rule representing the correspondence between the dynamics and the string number and the fingering;

[0011] The selecting the second sound source from the first sound source based on the note information and the note context information of the current note and according to the playing technique rules corresponding to the target type of musical instrument includes:

[0012] According to the note context information of the current note and the first technique rule, selecting a target first sound source that matches the fingering of the current note from the first sound source;

[0013] According to the note information of the current note and in accordance with the second technique rule, a second sound source matching the fingering of the current note is selected from the target first sound source.

[0014] Optionally, the target position includes the first note of a phrase, the last note of a phrase, and a note located on a beat;

[0015] Before selecting a target first sound source matching the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule, the method further includes:

[0016] Determining position information of the current note according to the note context information of the current note;

[0017] Determining whether the current note belongs to the target position according to the position information, and if it does, performing the step of selecting a target first sound source matching the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule;

[0018] If it does not belong to the target position, then according to the note information of the current note and the second technique rule, a second sound source matching the fingering of the current note is screened out from the first sound source.

[0019] Optionally, determining the position information of the current note according to the note context information of the current note includes:

[0020] Determining, according to the note context information of the current note, a first time difference between the current note and a previous note, and a second time difference between the current note and a subsequent note;

[0021] If the first time difference is greater than the time of one beat, determining that the current note is the first note of the phrase;

[0022] If the second time difference is greater than the time of one beat, it is determined that the current note is the last note of the phrase.

[0023] Optionally, determining the position information of the current note according to the note context information of the current note includes:

[0024] If the time difference between the current note and the starting point of the beat in the current time signature is less than a preset duration threshold, it is determined that the current note is on the beat;

[0025] The starting point time of the rebeat within the time signature is an element in the rebeat starting point time sequence; the rebeat starting point time sequence is generated according to the time signature, the music speed and a linear growth series.

[0026] Optionally, the target type of musical instrument is guqin;

[0027] Before traversing each note in the melody audio in sequence, it also includes:

[0028] Determine whether the range of the melody information is within the range of the guqin; if not, convert the range of the melody audio into the range of the guqin by raising or lowering the octave.

[0029] Optionally, the process of generating the sound source database of the target type of musical instrument includes:

[0030] Acquire the sound source data of the target type of musical instrument;

[0031] Performing sound source cleaning on the sound source data; the sound source cleaning includes checking the pitch value and the sound source starting point;

[0032] The cleaned sound source data is expanded to obtain a sound source database.

[0033] Optionally, after generating music of a target type instrument corresponding to the melody audio, the method further includes:

[0034] Query whether there are two consecutive notes of the target type instrument corresponding to the melody audio on the same string and the interval between the notes is less than an interval threshold;

[0035] If so, the first note of the two consecutive notes is faded out.

[0036] In a second aspect, the present application discloses an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement the aforementioned music generation method.

[0039] In a third aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the aforementioned music generation method when executed by a processor.

[0040] In a fourth aspect, the present application discloses a computer program product, including a computer program, which implements the aforementioned music generation method when executed by a processor.

[0041] In the present application, a melody audio is obtained, and music information of the melody audio is extracted; the music information includes note information, note context information and melody information of different notes in the melody audio; each note in the melody audio is traversed in turn, and according to the note information of the current note, a first sound source that matches the pitch of the current note is screened out from a sound source database of a target type of musical instrument; based on the note information and note context information of the current note, a second sound source is screened out from the first sound source in accordance with the playing technique rules corresponding to the target type of musical instrument; the playing technique rules include the correspondence between playing fingerings and notes; a matching sound source corresponding to the current note is determined according to the second sound source; and music of the target type of musical instrument corresponding to the melody audio is generated according to the matching sound source corresponding to each note in the melody audio and the melody information. By first selecting a part of the sound sources from the sound source database according to the pitch of the notes, and then based on the note information and the note context information, selecting the second sound source from the first sound source according to the correspondence between the playing fingering and the notes; by combining the note information and the note context information, and according to the correspondence between the notes and the playing fingering, determining the sound source with matching fingering for the notes, the final synthesized music is more natural and in line with the actual playing characteristics of the instrument; and the user only needs to provide the melody he wants to synthesize to quickly generate an audio file of the instrument playing the melody. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flow chart of a music generation method provided for this application;

[0044] Figure 2 A schematic diagram of a system framework applicable to the music generation solution provided in this application;

[0045] Figure 3 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the prior art, the timbre of musical instruments is synthesized by means of digital signal processing or machine learning, specifically by obtaining the fundamental frequency sequence of the melody to be synthesized, then modeling the timbre of the musical instrument, extracting the timbre features, and finally synthesizing the obtained fundamental frequency sequence and timbre features to obtain an audio file of the instrument performance. However, the synthesized music is stiff and unnatural, and parameter-based timbre modeling requires a lot of time and model training costs. In order to overcome the above technical problems, the present application proposes a music generation method that can make the final synthesized music more natural, in line with the actual performance characteristics of the instrument, and does not require a lot of training costs.

[0048] The present application embodiment discloses a music generation method, see Figure 1 As shown, the method may include the following steps:

[0049] Step S11: Acquire melody audio and extract music information of the melody audio; the music information includes note information of different notes in the melody audio, note context information and melody information.

[0050] First, a melody audio is obtained so as to generate music played by a target type of instrument according to the melody. Music information is extracted from the melody audio, specifically including but not limited to note information of different notes, context information between notes, and melody information.

[0051] That is, by preprocessing the input melody audio, the preprocessing includes: extracting the note information of the melody audio, storing each note in the form of a token; traversing the token sequence, recording the context information of each token; extracting the tonality, range, music speed (BPM, Beat Per Minute), and time signature of the note information; adjusting the range of the input melody. Among them, the note information includes but is not limited to pitch, starting point, duration and intensity; the melody information includes but is not limited to tonality, range, music speed and time signature; the target type instrument can be a guqin or guzheng, etc.

[0052] Specifically, for note information, the melody audio is input in the form of a MIDI (Musical Instrument Digital Interface) file; the MIDI file is parsed, all notes are traversed in turn, and the pitch (pitch), starting point (onset), duration (duration) and velocity (velocity) of each note are recorded; it can be understood that the melody audio is segmented according to the notes to obtain multiple audio segments, and the starting point represents the time point in the audio segment when the note starts to be played. The four attributes obtained above are stored in a token, namely [pitch, onset, duration, velocity]. And so on. Traverse the note token sequence and record the context information of each token. Specifically, for each note, it is necessary to record its previous note, the next note, and its own token; the context information can be used to determine some special techniques.

[0053] For melody information, the tonality, range, music speed (BPM) and time signature of the input melody are extracted. Among them, the tonality is obtained by calculating the Pearson product-moment correlation coefficient between the cumulative total duration of notes of different pitches in the melody audio and different tonality feature weights; the tonality corresponding to the Pearson product-moment correlation coefficient with the maximum absolute value is used as the tonality of the melody audio. For example, the tonality of the input melody is extracted using the Krumhansl-Schmuckler algorithm; the core idea is to calculate the Pearson product-moment correlation coefficient between the cumulative total duration of notes of 12 different pitches in the melody audio and 24 different tonality feature weights, and the corresponding tonality with the maximum absolute value of the 24 values ​​is the tonality of the melody. The range of the input melody is the maximum and minimum intervals of the note pitch in the melody audio. The music speed and time signature can be directly parsed from the MIDI file.

[0054] If the target instrument is a guqin, before traversing each note in the melody audio in turn, it may also include: judging whether the range in the melody information is within the range of the guqin; if not, converting the range of the melody audio to the range of the guqin by raising or lowering the octave. It is understandable that since the range of the guqin is only in the C1-D5 range, and in order to ensure that the final generated guqin music is more natural, it is necessary to convert the range of the input melody into the range of the guqin. Converting to the range of the guqin by raising or lowering the octave specifically includes: judging whether the range of the input melody is within the range of the guqin, if so, no processing is performed; otherwise, the MIDI pitch value of all note tokens is reduced by 12 (i.e., one octave), and then judging after processing until the conditions are met.

[0055] Step S12: traverse each note in the melody audio in turn, and according to the note information of the current note, select the first sound source that matches the pitch of the current note from the sound source database of the target type of musical instrument.

[0056] In this embodiment, the notes extracted from the melody audio are matched with the cleaned sound source database in sequence. First, the sound source with the same pitch as the note is selected as the first batch of candidate sound sources to obtain the first sound source, that is, the sound source data of the same string is selected.

[0057] In some embodiments, the process of generating the sound source database of the target type of musical instrument may include: obtaining the sound source data of the target type of musical instrument; performing sound source cleaning on the sound source data; the sound source cleaning includes checking the pitch value and checking the sound source starting point; expanding the cleaned sound source data to obtain a sound source database. It is understandable that the acquired sound source data may be inaccurate, and direct use will result in the effect of synthesizing music, so it is first cleaned; in addition, the acquired sound source data may not be comprehensive, so the richness of the sound source data is increased by expansion to meet the needs of music synthesis and to adapt to melody input of different tones.

[0058] Among them, the sound source cleaning of the sound source data may include: using a fundamental frequency detection method to detect the fundamental frequency value of the sound source data; by comparing the fundamental frequency value with the pitch value in the annotation information corresponding to the current sound source data, judging whether the pitch of the current sound source data is standard, and updating the pitch value according to the fundamental frequency value for non-standard sound source data. It can be understood that the sound source database cleaning is to check whether the annotated pitch value of each sound source is accurate.

[0059] Specifically, the fundamental frequency value predicted by the fundamental frequency detection algorithm is compared with the pitch value marked by the audio data. If the interval difference between the extracted fundamental frequency value and the marked value is less than a minor second, the pitch value marked by the sound source is considered accurate, otherwise it is considered inaccurate. Among them, the fundamental frequency detection algorithm can use the PYIN algorithm, autocorrelation or average amplitude difference to extract the fundamental frequency; secondly, the fundamental frequency value and the pitch value need to be converted to MIDI pitch in the same direction for comparison, that is, the two are unified to the same unit for comparison. The formula for converting fundamental frequency to MIDI pitch is:

[0060] ;

[0061] The fundamental frequency value extracted by the fundamental frequency detection algorithm is in Hz. The process of converting the pitch value to MIDI pitch is a simple mapping, as shown in Table 1 below:

[0062] Table 1

[0063] Pitch value MIDI pitch C4 48 C#4 / Db4 49 D4 50 D#4 / Eb4 51

[0064] If the difference between the extracted fundamental frequency value and the marked pitch value is less than a minor second, that is, the absolute value of the MIDI pitch converted from the fundamental frequency value minus the MIDI pitch converted from the marked value is less than 1, the pitch is considered standard, otherwise it is not standard; for non-standard sound source data, the pitch marking needs to be removed or changed.

[0065] The above-mentioned sound source cleaning of the sound source data may also include: determining the time point of the first local peak of the sound source data by peak extraction according to the energy of the sound source data in each frame; aligning the time point of the first local peak with the preset time point. That is, the sound source database cleaning includes checking whether the starting point of each sound source is aligned. The starting point of each sound source is detected, specifically by shifting the audio horizontally on the time axis, and aligning the detected sound source starting point with 20ms (or other time points), that is, all audio data starts to have sound after 20ms. The starting point detection process of the sound source includes: first calculating the energy envelope of the audio, framing the audio, and then calculating the energy and timestamp of each frame x(n); wherein the energy is calculated as: ); Then, the peak picking algorithm is used to find all the local peaks of the energy envelope. The time point of the first local peak is the starting point of the sound source. Starting point alignment can ensure the uniformity of the effect when all sound source data is played, and avoid notes not being on the beat.

[0066] Among them, the sound source data after cleaning is expanded to obtain a sound source database, including: obtaining sound source information of each sound source data; the sound source information is pre-labeled, and the sound source information includes sound source ID, playing hand, fingering, string number, pitch, strength, and file path; determining the target sound source to be obtained by traversing the sound source information, and determining the most similar original sound source data corresponding to the target sound source according to the sound source information; performing Fourier transform on the original sound source data to obtain corresponding frequency domain data; performing frequency expansion on the frequency domain data using a frequency expansion method to obtain expanded data, and performing inverse Fourier transform on the expanded data to obtain the target audio.

[0067] First, extract the information by parsing the file name of each sound source, mainly including: sound source ID, which hand is playing (left hand (L) or right hand (R)), playing technique, string number (fret position), pitch, intensity, and file path. For example, if a sound source name is "L_MO_NEI_Str_5_E4_T0_f_TK1.wav", the extracted information is: [0001, L, wipe, fifth string, 52, f, "data / L_HARM_NEI_Str_5_E4_T0_f_TK1.wav"], where MO is the wipe technique, 52 is the MIDI pitch of E4, and f is the abbreviation of "strong" intensity, which means that the volume of the note needs to be larger.

[0068] In this embodiment, it is considered that the general guqin sound source database will only collect sounds in the pentatonic scale under a certain fixed tonality. In order to achieve the ability to generate guqin music of different tonality, the sound source library is expanded with those sounds that have not been collected. The process of sound source expansion includes: traversing the note information, and filtering out the sounds that have not been collected according to the guqin's range C1-D5. For example, if the sound of playing F4 with p force using the wiping technique on the five strings is not collected, it is recorded as the target sound source, and then the sound source with the same string, the same technique, the same force and the smallest pitch difference with F4 is found from the sound source library, recorded as the original sound source, and the target sound source is generated on the basis of the original sound source through the frequency expansion algorithm. The frequency extension algorithm not only moves the fundamental frequency, but also moves all harmonics in the same proportion, thereby maintaining the characteristics of the original timbre. Specifically, it includes: first, using Fourier transform (FFT, fast Fourier transform) to convert the original sound source signal from the time domain to the frequency domain; then, in the frequency domain, all frequency components are moved in the same proportion. For example, to move E4 (approximately 329.63 Hz) to F4 (approximately 349.23 Hz), all frequency components need to be multiplied by 349.23 / 329.63, that is, the proportion of the movement; finally, using inverse Fourier transform (IFFT, Inverse Fast Fourier Transform) to convert the modified spectrum from the frequency domain back to the time domain to obtain the target sound source.

[0069] Step S13: Based on the note information and note context information of the current note, and in accordance with the playing technique rules corresponding to the target type of musical instrument, a second sound source is selected from the first sound source; the playing technique rules include the correspondence between playing fingerings and notes.

[0070] That is, the fingering of a note is also related to the note information and the context in which the note is located. Therefore, after filtering based on pitch, the first sound source is further filtered based on the note information and note context information and in accordance with the playing technique rules of the target type of instrument to filter out the second sound source.

[0071] That is, for each note in the melody audio, a corresponding matching sound source is selected from the sound source database, and the matching includes pitch matching, loudness matching, and appropriate fingering, etc. The above-mentioned sound source database contains a large amount of sound source data of the target type of musical instrument, and the sound source data distinguishes fingering, that is, the technique used for playing. Taking the guqin as an example, the fingering includes but is not limited to: wiping, picking, hooking, picking, hitting, picking, breaking, supporting, stroking, bumping, harmonics, etc. The sound source database contains audio data of different strings using different fingerings, and audio data of different strings with different loudness. Using the sound source database constructed in this way to generate music, it is possible to form guqin music with rich playing techniques.

[0072] In this embodiment, the playing technique rules include a first technique rule that characterizes the correspondence between the target position of a note in a phrase or beat and the fingering, and a second technique rule that characterizes the correspondence between the dynamics and the string number and the fingering. The second sound source is selected from the first sound source based on the note information and note context information of the current note and in accordance with the playing technique rules corresponding to the target type of instrument. This may include: selecting the target first sound source that matches the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule; selecting the second sound source that matches the fingering of the current note from the target first sound source according to the note information of the current note and the second technique rule. That is, after obtaining the first sound source, the target first sound source is selected from the first sound source according to the context information of the note using the first technique rule.

[0073] It is understandable that, taking the guqin as an example, the guqin is generally played by pressing the strings with the left hand and plucking the strings with the right hand. In terms of the position of pressing the strings, there is a difference in timbre between pressing with the flesh and pressing with the flesh. As for plucking the strings with the right hand, there is also a difference between fingers and the difference between plucking with the back of the fingernails and plucking with the flesh of the finger surface. The amplitude of the change in strength is very different. Therefore, the model is built through the context information of the notes. The context information of the notes is divided into three situations: 1. The current note is the first note of the phrase; 2. The current note is the last note of the phrase; 3. The current note appears on the beat (such as the first beat of the 4 / 4 beat). For example, when a note is the first note in a phrase, you can randomly use fingerings such as "release and close", "lift as one", "pick as one" or "pull"; when a note is the last note in a phrase, you can randomly use fingerings such as "single hit", "harmonic (inner)", "harmonic (outer)", "harmonic (support)"; when a note appears on the beat (the beat is the first beat of each measure), you can randomly use fingerings such as "release and close", "lift as one", "pick as one" or "pull". In addition, it should be noted that when the technique of the note is determined to be "harmonic", it is necessary to match a sound source in the sound source library that is an octave higher than the pitch of the note.

[0074] The target position includes the first note of a phrase, the last note of a phrase, and a note on a beat. Before selecting the target first sound source that matches the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule, the method further includes: determining the position information of the current note according to the note context information of the current note; judging whether the current note belongs to the target position according to the position information; if it belongs to the target position, selecting the target first sound source that matches the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule; if it does not belong to the target position, selecting the second sound source that matches the fingering of the current note from the first sound source according to the note information of the current note and the second technique rule. That is, first determine whether it is the target position. If it is, use the fingering corresponding to the position as a screening condition to further screen the first sound source to obtain the target first sound source, and then screen the second sound source based on the target first sound source according to the second technical rule; if it does not belong to the target position, directly screen the second sound source based on the first sound source according to the second technical rule.

[0075] Specifically, the above-mentioned determination of the position information of the current note based on the note context information of the current note includes: determining the first time difference between the current note and the previous note, and the second time difference between the current note and the next note based on the note context information of the current note; if the first time difference is greater than the time of one beat, the current note is determined to be the first note of the phrase; if the second time difference is greater than the time of one beat, the current note is determined to be the last note of the phrase. For example, using the note context information, the time difference between the current note and the previous note and the next note is calculated respectively. If the time difference between the starting point of the current note and the previous note is greater than or equal to the time taken for one beat, the current note is the first note in the phrase; if the time difference between the starting point of the current note and the next note is greater than or equal to the time taken for one beat, the current note is the last note in the phrase; the time taken for one beat is denoted as t, and the calculation formula is:

[0076] .

[0077] Specifically, the above-mentioned determining the position information of the current note based on the note context information includes: if the time difference between the current note and the starting point time of the beat in the current time signature is less than a preset duration threshold, then the current note is located on the beat; the starting point time of the beat in the time signature is an element in the beat starting point time sequence; the beat starting point time sequence is generated based on the time signature, the music speed and a linear growth series.

[0078] For example, if the time difference between the starting point of the current note and the starting point of the downbeat of the current time signature is less than the set threshold, the current note is considered to be on the downbeat; the threshold can be set to 20ms, because this value is the minimum time difference that humans can distinguish between two sounds. The process of determining whether it is a downbeat is as follows: first generate the downbeat starting point time series, recorded as downbeat (n), and the calculation formula is:

[0079] ;

[0080] in is a linearly growing sequence representing the number of measures, [0, 1, 2, 3, 4…]; is the number above the time signature. Subtract the starting point of the current note from the elements in the downbeat (n) sequence, and record it as , which is the time difference between the starting point of the current note and the starting point of the current beat. The formula is as follows:

[0081] ;in, It can be seen that through the above three methods, it is possible to determine whether the note is the note at the target position, and then according to the fingering commonly used by the instrument for the note at the target position, the fluency and authenticity of the finally generated music can be improved.

[0082] Furthermore, different strings have different fingerings at different volumes, and the same string may have different fingerings at different volumes. For example, the light notes played on the first or second string are played with "da" or "pick", and the heavy notes are played with "hook" or "tick". The light notes played on the third, fourth, and fifth strings are played with "wipe" or "hook", and the heavy notes are played with "hook" or "tick". Therefore, the second sound source is selected from the target first sound source according to the strength of the note.

[0083] It can be seen that according to the correspondence between the note position and the fingering, the target first sound source that matches the fingering is screened out from the first sound source, and then, according to the correspondence between the velocity and the string number and the fingering, the second sound source that matches the fingering is screened out from the target first sound source; it can be seen that considering that different note positions, velocities and string numbers will use different fingerings, the sound source database contains sound source data of different fingerings, and generates instrumental music by combining fingering factors, so that the final synthesized audio is more natural and in line with the actual playing characteristics of the instrument; the user only needs to provide the melody he wants to synthesize, and an audio file of the instrument playing the melody can be quickly generated.

[0084] Step S14: determining a matching sound source corresponding to the current note according to the second sound source.

[0085] That is, if there are multiple second sound sources, one is randomly selected as the matching sound source of the current note; and so on, until all notes are matched. If there is only one second sound source for the current note after screening, the second sound source is directly used as the matching sound source of the current note.

[0086] Step S15: Generate music of a target type of instrument corresponding to the melody audio according to the matching sound source corresponding to each note in the melody audio and the melody information.

[0087] Finally, based on the matching sound sources corresponding to each note, music of the target type of instrument corresponding to the melody audio is generated according to the melody information of the melody audio. Thus, by first extracting each note information in the melody audio input by the user, matching the corresponding sound source in the sound source database according to the pitch, note information and note context information, and combining the performance technique rules, and then superimposing the matched sound sources in the time domain. Finally, after post-processing, including but not limited to reverberation, delay, compression, etc., an audio file of the instrument's performance is generated, realizing automatic music synthesis based on performance technique rules; making the final synthesized audio more natural and more guqin-flavored.

[0088] In some embodiments, after generating the music of the target type instrument corresponding to the melody audio, it may also include: querying whether there are two consecutive notes of the music of the target type instrument corresponding to the melody audio that are of the same string and the note interval is less than an interval threshold; if so, fading the first note of the two consecutive notes to achieve an update of the music of the target type instrument corresponding to the melody audio.

[0089] It is understandable that in order to generate audio that conforms to the physical sounding principle of musical instruments and to make each note sound naturally, it is necessary to process the situation where two notes that are too close in time play the same string at the same time. The processing logic is to fade the signal before the second note starts to sound, and then let the second note start to sound. The specific attenuation operation can be: multiply the signal 20ms before the second note by a nonlinear amplitude envelope (denoted as fadeout (t)), the formula is as follows:

[0090] ;

[0091] Where T is time. After reverberation, delay, compression and other post-processing, the final music is obtained, and the effect is real.

[0092] As can be seen from the above, in this embodiment, melody audio is obtained, and music information of the melody audio is extracted; the music information includes note information, note context information and melody information of different notes in the melody audio; each note in the melody audio is traversed in turn, and according to the note information of the current note, a first sound source matching the pitch of the current note is screened out from the sound source database of the target type of musical instrument; based on the note information and note context information of the current note, a second sound source is screened out from the first sound source according to the playing technique rules corresponding to the target type of musical instrument; the playing technique rules include the correspondence between playing fingerings and notes; the matching sound source corresponding to the current note is determined according to the second sound source; according to the matching sound source corresponding to each note in the melody audio and the melody information, music of the target type of musical instrument corresponding to the melody audio is generated. According to the matching sound source corresponding to each note in the melody audio and the melody information, music of the target type of musical instrument corresponding to the melody audio is generated. By first selecting a part of the sound sources from the sound source database according to the pitch of the notes, and then based on the note information and the note context information, selecting the second sound source from the first sound source according to the correspondence between the playing fingering and the notes; by combining the note information and the note context information, and according to the correspondence between the notes and the playing fingering, determining the sound source with matching fingering for the notes, the final synthesized music is more natural and in line with the actual playing characteristics of the instrument; and the user only needs to provide the melody he wants to synthesize to quickly generate an audio file of the instrument playing the melody.

[0093] The system framework used in the music generation solution of this application can be found in Figure 2 As shown, it may specifically include: a backend server and a number of user terminals that establish communication connections with the backend server. The user terminals include but are not limited to tablet computers, laptop computers, smart phones, and personal computers (PCs), which are not limited here.

[0094] In the present application, the user end is used to send melody audio to the background server, and the background server executes the steps of the music generation method, including obtaining melody audio, extracting music information of the melody audio; the music information includes note information, note context information and melody information of different notes in the melody audio; traversing each note in the melody audio in turn, and filtering out a first sound source that matches the pitch of the current note from the sound source database of the target type instrument according to the note information of the current note; based on the note information and note context information of the current note, according to the performance technique rules corresponding to the target type instrument, filtering out a second sound source from the first sound source; the performance technique rules include the correspondence between performance fingerings and notes; determining the matching sound source corresponding to the current note according to the second sound source; generating music of the target type instrument corresponding to the melody audio according to the matching sound source corresponding to each note in the melody audio and the melody information. The background server pushes the generated music to the user end.

[0095] Below, a certain music APP is taken as an example to illustrate the technical solution of this application. Assuming that the user has installed this music APP on the user side, when he wants to listen to music with a specific melody played by instrument A, he enters the APP and uploads the melody audio and instrument type to the server. Among them, the melody audio can be a song or pure music, etc.

[0096] After the server obtains the melody audio, it extracts the music information of the melody audio; the music information includes the note information of different notes in the melody audio, the note context information and the melody information; the note information includes the pitch, starting point, duration and intensity; the melody information includes the tonality, range, music speed and time signature.

[0097] The server traverses each note in the melody audio in turn, and selects 10 sound sources with the same pitch as the current note from the sound source database of instrument A according to the note information to obtain the first sound source; according to the note context information and the first technical rule corresponding to the target type of instrument, the first technical rule is the correspondence between the target position of the note in the phrase or beat and the fingering, and selects 5 sound sources that match the fingering of the current note from the first sound source to obtain the target first sound source; according to the note information and the second technical rule corresponding to the target type of instrument, selects 2 sound sources that match the fingering of the current note from the target first sound source to obtain the second sound source, and selects one sound source from the second sound source as the matching sound source corresponding to the current note; the second technical rule is the correspondence between the velocity and the string number and the fingering; finally, according to the matching sound source corresponding to each note in the melody audio, generate music of instrument A corresponding to the melody audio.

[0098] Accordingly, the embodiment of the present application further discloses a music generating device, which includes:

[0099] The audio acquisition module 11 is used to acquire melody audio and extract music information of the melody audio; the music information includes note information of different notes in the melody audio, note context information and melody information;

[0100] A first sound source determination module 12 is used to sequentially traverse each note in the melody audio, and select a first sound source that matches the pitch of the current note from a sound source database of a target type of musical instrument according to the note information of the current note;

[0101] A second sound source determination module 13 is used to filter out a second sound source from the first sound source based on the note information and note context information of the current note and according to the playing technique rules corresponding to the target type of musical instrument; the playing technique rules include the correspondence between playing fingerings and notes;

[0102] A matching sound source determination module 14, configured to determine a matching sound source corresponding to the current note according to the second sound source;

[0103] The music generation module 15 is used to generate music of a target type of instrument corresponding to the melody audio according to the matching sound source corresponding to each note in the melody audio and the melody information.

[0104] As can be seen from the above, in this embodiment, melody audio is obtained and music information of the melody audio is extracted; the music information includes note information, note context information and melody information of different notes in the melody audio; each note in the melody audio is traversed in turn, and according to the note information of the current note, a first sound source matching the pitch of the current note is screened out from the sound source database of the target type of musical instrument; based on the note information and note context information of the current note, a second sound source is screened out from the first sound source in accordance with the playing technique rules corresponding to the target type of musical instrument; the playing technique rules include the correspondence between playing fingerings and notes; the matching sound source corresponding to the current note is determined according to the second sound source; and music of the target type of musical instrument corresponding to the melody audio is generated according to the matching sound source corresponding to each note in the melody audio and the melody information. By first selecting a part of the sound sources from the sound source database according to the pitch of the notes, and then based on the note information and the note context information, selecting the second sound source from the first sound source according to the correspondence between the playing fingering and the notes; by combining the note information and the note context information, and according to the correspondence between the notes and the playing fingering, determining the sound source with matching fingering for the notes, the final synthesized music is more natural and in line with the actual playing characteristics of the instrument; and the user only needs to provide the melody he wants to synthesize to quickly generate an audio file of the instrument playing the melody.

[0105] In some specific embodiments, the playing technique rule includes a first technique rule characterizing the correspondence between the target position of a note in a phrase or a beat and the fingering, and a second technique rule characterizing the correspondence between the intensity and the string number and the fingering; the second sound source determination module 13 may specifically include:

[0106] a target first sound source screening unit, configured to screen out a target first sound source matching the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule;

[0107] The second sound source screening unit is used to screen out a second sound source that matches the fingering of the current note from the target first sound source according to the note information of the current note and the second technique rule.

[0108] In some specific embodiments, the target position may specifically include the first note of a phrase, the last note of a phrase, and a note located on a beat;

[0109] The music generating device may further include:

[0110] a position determination unit, configured to determine the position information of the current note according to the note context information of the current note before selecting a target first sound source matching the fingering of the current note from the first sound source according to the first technique rule;

[0111] A first execution unit is used to determine whether the current note belongs to the target position according to the position information, and if it belongs to the target position, to perform the step of selecting a target first sound source matching the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule;

[0112] The second execution unit is used to select a second sound source that matches the fingering of the current note from the first sound source according to the note information of the current note and the second technique rule if it does not belong to the target position.

[0113] In some specific embodiments, the position determination unit may specifically include:

[0114] a time difference determination unit, configured to select, if the fingering does not belong to the target position, a second sound source matching the fingering of the current note from the first sound source according to the note information of the current note and the second technique rule;

[0115] a phrase first note determination unit, configured to determine the current note as the phrase first note if the first time difference is greater than a beat;

[0116] The phrase last note determination unit is used to determine that the current note is the phrase last note if the second time difference is greater than the time of one beat.

[0117] In some specific embodiments, the position determination unit may specifically include:

[0118] A beat determination unit is used to determine that the current note is on a beat if the time difference between the current note and the starting point time of the beat in the current time signature is less than a preset duration threshold; the starting point time of the beat in the time signature is an element in a beat starting point time sequence; the beat starting point time sequence is generated based on the time signature, the music speed and a linear growth series.

[0119] In some specific embodiments, the target type musical instrument may be a guqin;

[0120] The music generating device may further include:

[0121] The range conversion unit is used to determine whether the range of the melody information is within the range of the guqin; if it is not within the range of the guqin, the range of the melody audio is converted to the range of the guqin by raising or lowering the octave.

[0122] In some specific embodiments, the music generating device may specifically include:

[0123] An interval query unit is used to query whether there are two consecutive notes of the target type instrument corresponding to the melody audio on the same string and the interval between the notes is less than an interval threshold after generating the music of the target type instrument corresponding to the melody audio;

[0124] A weakening processing unit is used for, if any, performing a fade-out process on the first note of the two consecutive notes.

[0125] Furthermore, the present application also discloses an electronic device, see Figure 3 As shown, the contents in the drawings should not be considered as any limitation on the scope of use of the present application.

[0126] Figure 3 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the music generation method disclosed in any of the aforementioned embodiments.

[0127] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0128] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including melody audio, etc. The storage method can be temporary storage or permanent storage.

[0129] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the music generation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0130] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the music generation method disclosed in any of the aforementioned embodiments are implemented.

[0131] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the music generation method disclosed in any of the aforementioned embodiments.

[0132] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0133] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0134] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0135] The above is a detailed introduction to a music generation method, device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in the field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A music generation method, characterized in that: include: Acquire melody audio, and extract music information of the melody audio; The music information includes note information of different notes in the melody audio, note context information and melody information; Traversing each note in the melody audio in turn, and selecting a first sound source that matches the pitch of the current note from a sound source database of a target type of musical instrument according to the note information of the current note; Based on the note information and note context information of the current note, and in accordance with the playing technique rules corresponding to the target type of musical instrument, a second sound source is selected from the first sound source; the playing technique rules include the corresponding relationship between playing fingerings and notes; Determine a matching sound source corresponding to the current note according to the second sound source; According to the matching sound source corresponding to each note in the melody audio and the melody information, music of a target type instrument corresponding to the melody audio is generated.

2. The music generation method according to claim 1, characterized in that: The playing technique rules include a first technique rule representing the correspondence between the target position of a note in a phrase or a beat and the fingering, and a second technique rule representing the correspondence between the force and the string number and the fingering; The selecting the second sound source from the first sound source based on the note information and the note context information of the current note and according to the playing technique rules corresponding to the target type of musical instrument includes: According to the note context information of the current note and the first technique rule, selecting a target first sound source that matches the fingering of the current note from the first sound source; According to the note information of the current note and in accordance with the second technique rule, a second sound source matching the fingering of the current note is selected from the target first sound source.

3. The music generation method according to claim 2, characterized in that: The target positions include the first note of a phrase, the last note of a phrase, and the note on the beat; Before selecting a target first sound source matching the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule, the method further includes: Determining position information of the current note according to the note context information of the current note; Determining whether the current note belongs to the target position according to the position information, and if it does, performing the step of selecting a target first sound source matching the fingering of the current note from the first sound source according to the note context information of the current note and the first technique rule; If it does not belong to the target position, then according to the note information of the current note and the second technique rule, a second sound source matching the fingering of the current note is screened out from the first sound source.

4. The music generation method according to claim 3, characterized in that: Determining the position information of the current note according to the note context information of the current note includes: Determining, according to the note context information of the current note, a first time difference between the current note and a previous note, and a second time difference between the current note and a subsequent note; If the first time difference is greater than the time of one beat, determining that the current note is the first note of the phrase; If the second time difference is greater than the time of one beat, it is determined that the current note is the last note of the phrase.

5. The music generation method according to claim 3, characterized in that: Determining the position information of the current note according to the note context information of the current note includes: If the time difference between the current note and the starting point of the beat in the current time signature is less than a preset duration threshold, it is determined that the current note is on the beat; The starting point time of the rebeat within the time signature is an element in the rebeat starting point time sequence; the rebeat starting point time sequence is generated according to the time signature, the music speed and a linear growth series.

6. The music generation method according to claim 1, characterized in that: The target type of musical instrument is guqin; Before traversing each note in the melody audio in sequence, it also includes: Determining whether the range of the melody information is within the range of the guqin; If it is not within the range of the guqin, the range of the melody audio is converted to the range of the guqin by raising or lowering the octave.

7. The music generation method according to claim 1, characterized in that: The process of generating the sound source database of the target type musical instrument includes: Acquire the sound source data of the target type of musical instrument; Performing sound source cleaning on the sound source data; the sound source cleaning includes checking the pitch value and checking the sound source starting point; The cleaned sound source data is expanded to obtain a sound source database.

8. The music generation method according to any one of claims 1 to 7, characterized in that: After generating the music of the target type instrument corresponding to the melody audio, the method further includes: Query whether there are two consecutive notes of the target type instrument corresponding to the melody audio on the same string and the interval between the notes is less than an interval threshold; If so, the first note of the two consecutive notes is faded out.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the music generation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein when the computer programs are executed by a processor, the music generating method as described in any one of claims 1 to 8 is implemented.

Images