Harmony automatic generation method and device, equipment, medium and product

Abstract

This invention provides a method, apparatus, device, medium, and product for automatic harmony generation. It establishes a mode template and sets the probability of each mode in the template migrating to other modes. Real-time pitch analysis is performed on the input vocal signal to obtain pitch information. Based on the pitch information in the vocal signal, the mode in the mode template is matched to calculate the probability that each mode is the current mode of the vocal signal. Based on the probability that each mode is the current mode of the vocal signal, and considering the probability of mode migration, the mode of the vocal signal is determined. Based on the mode of the vocal signal and preset harmony pattern rules, the corresponding harmonic notes are generated. This invention can automatically generate highly musical harmonies without external accompaniment or chord input, making it suitable for complex scenarios such as improvisation, a cappella, and movement. Furthermore, by considering the probability of mode migration, it avoids abrupt mode changes and improves the smoothness of the harmony.

Description

Technical Field

[0001] This invention belongs to the field of audio signal processing technology, and in particular to a method, apparatus, device, medium and product for automatic harmony generation. Background Technology

[0002] Automatic harmony refers to a vocal audio processing technology where devices or software perform real-time pitch processing and modulation on the input vocals to create harmonic effects. Existing automatic harmony generation technologies generally rely on at least one of the following:

[0003] (1) MIDI input or chord information explicitly marked by the user;

[0004] (2) External accompaniment audio signal, the current harmony is inferred through chord recognition algorithm;

[0005] (3) The pre-set mode and rhythm structure restricts the space for harmonic construction.

[0006] For dynamic harmony generation technologies that require MIDI input or user-defined chord information, users need music theory knowledge or pre-arranged chords, making them unusable for non-professional users and unsuitable for improvisational singing or mobile a cappella. For schemes that require external accompaniment audio signals and infer the current harmony through chord recognition algorithms, the system fails without accompaniment because the accompaniment audio must be played synchronously, limiting its applicability and making it unsuitable for a cappella solos or a cappella performances. Automatic harmony generation technologies based on preset key and rhythmic structures have fixed harmonic structures and cannot adapt to improvisational key changes or free rhythmic variations during singing, exhibiting poor compatibility with improvisational styles such as jazz and folk.

[0007] For example, if a user needs to simultaneously play accompaniment or MIDI control information while singing to determine what harmonies they require, these dependencies limit the application of existing automatic harmony systems in a cappella, improvisational, or mobile scenarios. For instance, non-professional singers, lacking music theory and chord knowledge, still desire pleasing harmonic support when singing a cappella. Similarly, singers performing a cappella solos require appropriate harmonic support, but existing automatic harmony generation technologies struggle to accurately determine the current pitch, key, and musical semantics to generate suitable harmonies.

[0008] There is an urgent need for a solution that can automatically generate highly musical harmonies without relying on external accompaniment or chord input. This would allow automatically generated harmonies to perfectly adapt to complex environments such as improvisation, a cappella singing, and music creation in mobile settings. For example, in traditional music performance scenarios, some special situations often arise. Imagine a drummer passionately playing the drums while simultaneously yearning to sing, wanting to add vocal appeal to their drumming. However, limited by the venue, the drummer can only focus on drumming and cannot play the guitar to provide chord cues. In this situation, if the drummer wants to add harmonies to their singing, they will be in trouble because without chord references, harmonies become extremely difficult to generate.

[0009] Therefore, there is an urgent need for a method that analyzes and infers modal structure and melodic intent solely based on the human voice itself, and then generates reasonable harmony, in order to expand the applicability of automatic harmony generation. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method, apparatus, device, medium, and product for automatic harmony generation, which can automatically generate harmony based solely on the analysis of the vocal signal itself without external chord prompts or accompaniment information. This invention is applicable to devices and application scenarios such as vocal effects processors, real-time performance processors, and digital audio workstation plugins.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] On the one hand, the present invention provides a method for automatic harmony generation, comprising the following steps:

[0013] Establish a mode template, which includes a series of modes, each containing its scale structure and root note information;

[0014] Set the probability of each key in the key template migrating to other keys;

[0015] Real-time pitch analysis is performed on the input human voice signal to obtain pitch information from the human voice signal;

[0016] Based on the pitch information in the human voice signal, the probability of each mode being the current mode of the human voice signal is calculated by matching the mode in the mode template.

[0017] The key of the vocal signal is determined based on the probability that each key is the current key of the vocal signal, while also considering the probability of key shifts between keys.

[0018] Based on the modulation of the human voice signal and the preset harmonic pattern rules, the corresponding harmonic notes of the human voice signal are generated.

[0019] On the other hand, the present invention provides an automatic harmony generation device, comprising:

[0020] The mode template building module is used to create mode templates, which include a series of modes, each containing its scale structure and root note information;

[0021] The mode migration probability setting module is used to set the probability of each mode in the mode template migrating to other modes;

[0022] The pitch analysis module is used to perform real-time pitch analysis on the input human voice signal and obtain pitch information in the human voice signal.

[0023] The mode matching module is used to match the pitch information in the human voice signal with the mode in the mode template, and calculate the probability that each mode is the current mode of the human voice signal.

[0024] The mode determination module is used to determine the mode of the human voice signal based on the probability that each mode is the current mode of the human voice signal, while also considering the probability of mode migration.

[0025] The harmony generation module is used to generate harmonic notes corresponding to the human voice signal based on the tone of the human voice signal and preset harmony pattern rules.

[0026] On the other hand, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described automatic harmony generation method.

[0027] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described automatic harmony generation method.

[0028] On the other hand, the present invention provides a computer program product stored on a computer-readable storage medium and including computer instructions that, when executed by a processor, cause a computer device to perform the steps of the above-described automatic harmony generation method.

[0029] Compared with the prior art, the technical effects of the present invention are as follows:

[0030] This invention establishes a mode template and sets the probability of each mode in the template migrating to other modes. It performs real-time pitch analysis on the input vocal signal to obtain pitch information. Based on the pitch information in the vocal signal, it matches the modes in the mode template to calculate the probability that each mode is the current mode of the vocal signal. Considering the probability of each mode being the current mode of the vocal signal, and taking into account the probability of mode migration, it determines the mode of the vocal signal. Based on the mode of the vocal signal and preset harmony pattern rules, it generates the corresponding harmonic notes. This invention can automatically generate highly musical harmonies without external accompaniment or chord input, making it suitable for complex scenarios such as improvisation, a cappella, and movement. Furthermore, by considering the probability of mode migration, it avoids abrupt mode changes, improving the smoothness of the generated harmonies.

[0031] In some musical performance scenarios, such as those mentioned in the background section, a drummer, while passionately playing the drums, may feel an urge to sing, yearning to add vocal charm to their beats. However, limited by the venue, the drummer can only focus on drumming and cannot simultaneously play the guitar to provide chord cues. In this situation, the automatic harmony generation method provided by this invention solves this problem. By configuring a microphone jack on the drum machine, the drummer simply connects their microphone to the jack and sings while playing the drums. The automatic harmony generation system captures the vocal signal in real time and automatically generates smooth and harmonious harmonies based on the aforementioned method. This innovative design not only avoids harmony generation problems caused by a lack of chord cues but also effectively avoids abrupt key changes by fully considering the probability of key shifts, significantly improving the smoothness of the generated harmonies and adding endless charm and possibilities to musical performances. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0033] Figure 1 This is a flowchart of an automatic harmony generation method in one embodiment. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0035] This invention analyzes real-time or recorded human voice signals to infer their melody direction, local pitch distribution, modal tendency, and structural characteristics, and dynamically generates a harmonic track that is consistent with the main melody.

[0036] A scale refers to a sequence of notes, not the starting point of the notes. For example, the natural major scale starts with a single note (the root), and ascends by 2, 4, 5, 7, 9, and 11 semitones. Adding this root note to the root note itself, this forms a seven-note arrangement within an octave, commonly known as do re mi fa so la ti. These seven notes repeat throughout all octaves, forming the natural major scale.

[0037] From a mathematical perspective, if the frequency of the root note is R (Hz), then this scale is composed of all notes with the following fundamental frequencies:

[0038]

[0039] so This set of integers from 0 to 11 defines a musical scale.

[0040] For example, the jazz melodic minor scale: the set {0,2,3,5,7,9,11}, which is la ti do re mi#fa #so. These seven notes repeat in all octaves, defining the jazz melodic minor scale.

[0041] Mode: A combination of the root note and the scale developed from it. For example, the natural major scale with C as the root note is denoted as C major.

[0042] Reference Figure 1 One embodiment provides a method for automatically generating harmony, comprising the following steps:

[0043] Establish a mode template, which includes a series of modes, each containing its scale structure and root note information;

[0044] Set the probability of each key in the key template migrating to other keys;

[0045] Real-time pitch analysis is performed on the input human voice signal to obtain pitch information from the human voice signal;

[0046] Based on the pitch information in the human voice signal, the probability of each mode being the current mode of the human voice signal is calculated by matching the mode in the mode template.

[0047] The key of the vocal signal is determined based on the probability that each key is the current key of the vocal signal, while also considering the probability of key shifts between keys.

[0048] Based on the modulation of the human voice signal and the preset harmonic pattern rules, the corresponding harmonic notes of the human voice signal are generated.

[0049] In one embodiment, a method for establishing a tuning template is proposed, comprising the following steps:

[0050] Divide an octave into equal parts N A portion, received N One root frequency, of which N Integers greater than 12;

[0051] choose M There are several types of scales, among which M ≥1;

[0052] For each root frequency and each type of scale, a mode is formed;

[0053] For each key, calculate the frequencies of all notes within a predetermined range;

[0054] Summarize the note frequencies of all modes to form a mode template.

[0055] An octave consists of 12 semitones, or 12 root notes. Since a cappella singing often doesn't fall on the exact correct notes—for example, singing a song in a major key doesn't necessarily fall on a specific major key, but rather somewhere between A major and A#—this solution establishes a more refined modal analysis than traditional modes, dividing an octave into equal parts. N A portion, received N One root frequency, of which N It is an integer greater than 12.

[0056] For example, the root frequency of A major is denoted as... The 12 traditional root sounds are ,

[0057] The root frequency used is: .for example N When the value is 48, it is equivalent to dividing each semitone into four equal parts.

[0058] Select M A musical scale, for example, can M=2 is used to select the natural major scale and the jazz melodic minor scale. These two scales are sufficient for the needs of the vast majority of songs. The scale, combined with the root note, forms... N M Each key can be used to calculate all the pitches of the notes within that key. Since the human vocal range is limited, we can focus only on the range that humans can sing. Preferably, the predetermined vocal range in the above embodiment is set to 40Hz to 1000Hz, covering the range that humans can sing. The note frequencies between 40Hz and 1000Hz for each key are summarized to form a key template.

[0059] Vocals may shift between different keys due to inaccurate singing or the demands of a song. However, the probability of shifting from C major to A major is very small, while the probability of shifting from C major to a key between C and C# due to off-key singing is very high. Similarly, the probability of shifting from C major to F major, G major, or A melodic minor due to musical reasons is also high. Therefore, it is necessary to set the probability of each key shifting to other keys. Due to the symmetry of keys, the probability of shifting from C major to F major is similar to the probability of shifting from F major to Bb major, because the distance between their root notes is consistent. Therefore, it is only necessary to set the probability of shifting from C major or C minor to other keys, and apply this to all keys. The probability of each key shifting to other keys in the key template can be statistically analyzed from a large amount of singing data, or it can be set directly based on empirical values.

[0060] Real-time pitch analysis is performed on the input human voice signal to obtain pitch information. Many existing pitch analysis methods exist, such as autocorrelation and cepstral methods, which can be selected based on experience by those skilled in the art. Without loss of generality, the method disclosed in publication number 119580764A, "Low-Delay Band Pitch Detection Method, Apparatus, and Device Based on Neural Networks," published on March 7, 2025, can also be used to achieve pitch analysis.

[0061] In one embodiment, the probability that each mode is the current mode of the voice signal is calculated by matching the pitch information in the voice signal with the mode in the mode template, including the following steps:

[0062] Acquire multiple pitch frequencies of human voice signals within a continuous time period;

[0063] Time weights are assigned to the plurality of pitch frequencies, wherein the pitch frequencies closer to the current time have higher weights;

[0064] For each mode in the mode template, perform the following operations: (i) for each pitch frequency, find the closest frequency among all note frequencies in the mode and calculate the logarithmic frequency distance between the pitch frequency and the closest frequency; (ii) multiply the logarithmic frequency distance of each pitch frequency by the corresponding time weight and sum them to obtain the distance between the mode and the vocal signal.

[0065] Based on the distance between each key and the human voice signal, the probability that each key is the current key of the human voice signal is calculated. The closer the key is to the human voice signal, the higher the probability that the human voice signal belongs to that key.

[0066] For a mode such as the first n The distance between a given key and the human voice signal can be calculated using the following formula:

[0067] ;

[0068] in Indicates the current t Keep moving forward i The fundamental frequency of a human voice at a given moment Indicates time weight, Indicates the first n In the first mode j The fundamental frequency of each note. Calculation results. Indicates the preceding t The human voice signal at the moment and the first n The distance of the key. The shorter the distance, the more the human voice signal belongs to the key. n The higher the probability weight of each mode, the better. Distance This can be converted into a probability, that is, the human voice signal belongs to the first... n The probability of a mode ,as follows:

[0069] .

[0070] That This represents the hyperbolic tangent function.

[0071] In one embodiment, based on the probability of each mode being the current mode of the voice signal, while also considering the probability of mode transitions, according to Determine the current t The modulation of the human voice signal at any given moment, among which

[0072]

[0073] in, They represent t time, t The mode number of the human voice signal at time -1 expresst The key number in the key template corresponding to the key number of the human voice signal at time -1. express Migrating to the first in the tuning template n The probability of a certain mode. express t The first moment in the human voice signal and mode template n The matching probability of a certain mode.

[0074] Finally, based on the key of the vocal signal and the preset harmonic pattern rules, the corresponding harmonic notes are generated. For example, if a three-part harmony is set, with notes a third higher and a fourth lower, the closest note in the key is found for the current pitch. This note is stabilized using a Schmitt trigger. Two notes above the current note are used as the pitch of the higher harmonic, and three notes below are used as the pitch of the lower harmonic. A common vocal pitch-shifting algorithm, such as but not limited to PSOLA and PhaseVocoder, is then called to perform pitch shifting and superimposed on the dry vocal signal to form harmonious harmonic notes.

[0075] In another embodiment, an automatic harmony generation device is provided, comprising:

[0076] The mode template building module is used to create mode templates, which include a series of modes, each containing its scale structure and root note information;

[0077] The mode migration probability setting module is used to set the probability of each mode in the mode template migrating to other modes;

[0078] The pitch analysis module is used to perform real-time pitch analysis on the input human voice signal and obtain pitch information in the human voice signal.

[0079] The mode matching module is used to match the pitch information in the human voice signal with the mode in the mode template, and calculate the probability that each mode is the current mode of the human voice signal.

[0080] The mode determination module is used to determine the mode of the human voice signal based on the probability that each mode is the current mode of the human voice signal, while also considering the probability of mode migration.

[0081] The harmony generation module is used to generate harmonic notes corresponding to the human voice signal based on the tone of the human voice signal and preset harmony pattern rules.

[0082] In another embodiment, the mode matching module in the automatic harmony generation device includes:

[0083] The pitch frequency acquisition module is used to acquire multiple pitch frequencies of human voice signals within a continuous time period;

[0084] The weighting module is used to assign time weights to the multiple pitch frequencies, wherein the pitch frequency closer to the current time has a higher weight;

[0085] The distance calculation module is used to perform the following operations for each mode in the mode template: (i) for each pitch frequency, find the closest frequency among all note frequencies of the mode, and calculate the logarithmic frequency distance between the pitch frequency and the closest frequency; (ii) multiply the logarithmic frequency distance of each pitch frequency by the corresponding time weight, and sum them to obtain the distance between the mode and the human voice signal.

[0086] The probability calculation module is used to calculate the probability that each mode is the current mode of the voice signal based on the distance between each mode and the voice signal. The closer the mode is to the voice signal, the higher the probability that the voice signal belongs to that mode.

[0087] On the other hand, the present invention provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the automatic harmony generation method provided in any of the above embodiments. The computer device may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device stores sample data. The network interface of the computer device is used for communication with external terminals via a network connection.

[0088] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the automatic harmony generation method provided in any of the above embodiments.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] Matters not covered in this invention are common knowledge.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically generating harmony, characterized in that, Includes the following steps: Establish a mode template, which includes a series of modes, each containing its scale structure and root note information; Set the probability of each key in the key template migrating to other keys; Real-time pitch analysis is performed on the input human voice signal to obtain pitch information from the human voice signal; Matching the pitch information in the human voice signal with the mode in the mode template, and calculating the probability that each mode is the current mode of the human voice signal, includes: obtaining multiple pitch frequencies of the human voice signal in a continuous time period; assigning time weights to the multiple pitch frequencies, wherein the pitch frequencies closer to the current time have higher weights; For each mode in the mode template, perform the following operations: (i) for each pitch frequency, find the closest frequency among all note frequencies in the mode, and calculate the logarithmic frequency distance between the pitch frequency and the closest frequency; (ii) multiply the logarithmic frequency distance of each pitch frequency by the corresponding time weight, and sum them to obtain the distance between the mode and the vocal signal; based on the distance between each mode and the vocal signal, calculate the probability that each mode is the current mode of the vocal signal, wherein the shorter the distance between the mode and the vocal signal, the higher the probability that the vocal signal belongs to the mode; The key of the vocal signal is determined based on the probability that each key is the current key of the vocal signal, while also considering the probability of key shifts between keys. Based on the modulation of the human voice signal and the preset harmonic pattern rules, the corresponding harmonic notes of the human voice signal are generated.

2. The automatic harmony generation method according to claim 1, characterized in that, Create a debugging template, including: Divide an octave into equal parts N Share, received N One root frequency, of which N Integers greater than 12; choose M There are several types of scales, among which M ≥1; For each root frequency and each type of scale, a mode is formed; For each key, calculate the frequencies of all notes within a predetermined range; Summarize the note frequencies of all modes to form a mode template.

3. The automatic harmony generation method according to claim 2, characterized in that, The intended vocal range is 40Hz to 1000Hz, covering the vocal range that humans can sing.

4. The automatic harmony generation method according to claim 1, 2, or 3, characterized in that, Based on the probability of each mode being the current mode of the vocal signal, and considering the probability of mode transitions, according to Determine the current The modulation of the human voice signal at any given moment, among which in, They represent time, The key signature of the human voice signal at any given moment. express The key number in the key template corresponding to the key number of the human voice signal at any given moment. express Migrating to the first in the tuning template The probability of a certain mode. express The first moment in the human voice signal and mode template The matching probability of a certain mode.

5. An automatic harmony generation device, characterized in that, include: The mode template building module is used to create mode templates, which include a series of modes, each containing its scale structure and root note information; The mode migration probability setting module is used to set the probability of each mode in the mode template migrating to other modes; The pitch analysis module is used to perform real-time pitch analysis on the input human voice signal and obtain pitch information in the human voice signal. A mode matching module is used to match the pitch information in the human voice signal with the modes in the mode template, and calculate the probability that each mode is the current mode of the human voice signal. The mode matching module includes: The system includes a pitch frequency acquisition module for acquiring multiple pitch frequencies of a human voice signal within a continuous time period; a weight allocation module for assigning time weights to the multiple pitch frequencies, wherein the pitch frequency closer to the current moment has a higher weight; a distance calculation module for performing the following operations for each mode in the mode template: (i) for each pitch frequency, finding the closest frequency among all note frequencies of the mode and calculating the logarithmic frequency distance between the pitch frequency and the closest frequency; (ii) multiplying the logarithmic frequency distance of each pitch frequency by the corresponding time weight and summing them to obtain the distance between the mode and the human voice signal; and a probability calculation module for calculating the probability that each mode is the current mode of the human voice signal based on the distance between each mode and the human voice signal, wherein the shorter the distance between the mode and the human voice signal, the higher the probability that the human voice signal belongs to the mode. The mode determination module is used to determine the mode of the human voice signal based on the probability that each mode is the current mode of the human voice signal, while also considering the probability of mode migration. The harmony generation module is used to generate harmonic notes corresponding to the human voice signal based on the tone of the human voice signal and preset harmony pattern rules.

6. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The processor executes the steps of the automatic harmony generation method as described in claim 1 when executing a computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the automatic harmony generation method as described in claim 1.

8. A computer program product, said computer program product being stored on a computer-readable storage medium and comprising computer instructions, characterized in that, The computer instructions, when executed by the processor, cause the computer device to perform the steps of the automatic harmony generation method as described in claim 1.

Images