A method and system for generating audio from sheet music

Abstract

The present application relates to the technical field of music score, and particularly relates to a music score audio generation method and system. The method comprises the following steps: obtaining a to-be-adjusted music score, wherein the to-be-adjusted music score comprises at least one music unit; identifying edge notes in the to-be-adjusted music score; wherein the edge notes refer to notes with a pitch higher than a preset upper limit threshold of the pitch or a pitch lower than a preset lower limit threshold of the pitch; selecting an adjustment area according to the number or proportion of the edge notes in the corresponding music unit; the adjustment area is used to define the number or proportion of notes allowed or suggested to be modified; adjusting the pitch of the to-be-adjusted notes to obtain a music score pitch, generating an adjusted music score according to the music score pitch and the to-be-adjusted music score; and generating a music score audio for the adjusted music score. The present application adjusts the music score audio generation mode moderately, so that the music score prompt can adapt to the singing ability of a person, and the training effect and the vocal comfort are taken into account.

Description

Technical Field

[0001] This invention relates to the field of music notation technology, and in particular to a method and system for generating audio for music notation. Background Technology

[0002] Singing from sheet music is an important part of piano learning and daily practice, and it is also a key link in connecting sight-reading, pitch, rhythm, and auditory perception. By singing the sheet music aloud, learners can simultaneously clarify note values, dynamic logic, and melodic direction, establishing a linked memory between the sheet music, their hearing, and their finger playing. This avoids problems such as unclear sight-reading, rhythm confusion, and weak pitch sense caused by simply mechanically touching the piano.

[0003] In response, existing technologies have proposed some solutions to help users improve their sight-singing abilities.

[0004] For example, patent application CN116737994A discloses a method, apparatus, device, and medium for synchronously playing video, sung audio, and sheet music. One specific embodiment of the method includes: matching a demonstration video with the electronic sheet music corresponding to a target song to obtain first matching information; acquiring the sung audio of the target song; in response to determining that the sung audio is sung by a real person, matching the electronic sheet music and the sung audio to obtain second matching information; determining third matching information based on the first and second matching information; using the third matching information to stretch the sung audio to obtain stretched sung audio; and in response to detecting a preset operation, synchronously playing the demonstration video and the stretched sung audio using the first matching information, and synchronously displaying the electronic sheet music.

[0005] For example, patent application CN110889787A proposes a music teaching assistance system, which belongs to the field of teaching and includes a music library management module, a teaching module, a sight-singing learning module, an ear training learning module, a user data module, a human-computer interaction module, a processing unit, and a database module.

[0006] However, the above-mentioned technical solutions cannot be applied to complex real-world musical instrument learning scenarios. Therefore, there is an urgent need for a more intelligent method for generating sheet music audio. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for generating audio scores, which partially solves or alleviates the above-mentioned shortcomings in the prior art. By appropriately adjusting the method of generating audio scores, the scoring prompts can be adapted to human singing ability, taking into account both training effect and vocal comfort.

[0008] To solve the aforementioned technical problems, the present invention specifically adopts the following technical solution: A first aspect of the present invention is to provide a method for generating audio for singing scores, comprising: S101, Obtain the score to be adjusted, wherein the score to be adjusted includes at least one musical unit; S102, Identify edge notes in the score to be adjusted; wherein, the edge notes refer to notes whose pitch is higher than a preset upper limit threshold or lower than a preset lower limit threshold. S103, Select an adjustment area based on the number or proportion of edge notes in the corresponding musical unit; the adjustment area is used to define the number or proportion of notes that are allowed or suggested to be modified; S103 includes: S1031, determine whether the number or proportion of the edge notes is greater than a preset number or preset proportion; If S1031 is yes, then the note in the first adjustment region is selected as the note to be adjusted; If S1031 is not, then the note in the second adjustment area is selected as the note to be adjusted; Wherein, the first adjustment region is larger than the second adjustment region; S104, adjust the pitch of the note to be adjusted to obtain the musical score pitch, and generate an adjusted score based on the musical score pitch and the musical score to be adjusted; S105, generate the singing audio for the adjusted score.

[0009] In some embodiments, S105 includes: Obtain the sound attributes of the music unit, including frequency and / or volume; The musical score audio is generated based on the musical unit and the sound attributes.

[0010] In some embodiments, the step of obtaining the sound attributes of the music unit includes: Identify the pitch difference between the reference sung pitch of at least one adjustment unit and the reference sung pitch of an adjacent musical unit; wherein, the adjustment unit is a musical unit that has undergone pitch adjustment, and the reference sung pitch refers to a representative pitch used to characterize the overall pitch characteristics of a single musical unit; If at least one pitch difference is less than or equal to a set first pitch threshold, then proceed with the following steps: Update the sound attributes for the corresponding adjustment unit and / or the adjacent music unit.

[0011] In some embodiments, S104 includes: The pitch of the note to be adjusted is decreased or increased according to the suggested adjustment amount; and the pitch difference between the reference pitch of the corresponding adjustment unit and the reference pitch of the adjacent music unit is greater than the set second pitch threshold.

[0012] In some embodiments, including: The second pitch threshold is updated based on the complexity of the score to be adjusted; the complexity is determined based on the number of voice parts. The more voice parts a sound has, the higher the complexity, and correspondingly the smaller the second pitch threshold.

[0013] In some embodiments, the method further includes: The corresponding preset pitch upper limit threshold and preset pitch lower limit threshold are determined based on the selected sound material.

[0014] In some embodiments, including: The suggested adjustment amount is updated based on the preset upper pitch threshold or the preset lower pitch threshold; including: Identify the pitch difference between the edge note and its adjacent preset pitch upper limit threshold or preset pitch lower limit threshold; The suggested adjustment amount is updated based on the pitch difference; Wherein: the smaller the pitch difference, the smaller the suggested adjustment amount.

[0015] In some embodiments, it also includes: A sound material library is pre-established, and the sound material library stores at least one sound material. In response to the user's selection command, the corresponding sound material is selected as the sound material; Correspondingly, in S105, the singing score audio is generated based on the selected sound material and the adjusted score.

[0016] A second aspect of the present invention is to provide a music score audio generation system, comprising: The music score acquisition module is used to acquire the music score to be adjusted, and the music score to be adjusted includes at least one musical unit; An edge recognition module is used to identify edge notes in the score to be adjusted; wherein, the edge note refers to a note whose pitch is higher than a preset upper limit threshold or lower than a preset lower limit threshold. The adjustment selection module is used to select an adjustment area based on the number or proportion of edge notes in the corresponding musical unit; the adjustment area is used to define the number or proportion of notes that are allowed or suggested to be modified; The adjustment selection module includes: The judgment unit is used to determine whether the number or proportion of the edge notes is greater than a preset number or preset proportion; If the result of the judgment unit is yes, then the note in the first adjustment area is selected as the note to be adjusted; If the result of the judgment unit is negative, then the note in the second adjustment area is selected as the note to be adjusted; Wherein, the first adjustment region is larger than the second adjustment region; The music score generation module is used to adjust the pitch of the notes to be adjusted to obtain the pitch of the musical score, and to generate an adjusted music score based on the pitch of the musical score and the music score to be adjusted. An audio generation module is used to generate a vocal score audio for the adjusted musical score.

[0017] In some embodiments, the audio generation module is used to perform: Obtain the sound attributes of the music unit, including frequency and / or volume; The musical score audio is generated based on the musical unit and the sound attributes. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0019] Figure 1 Example diagram showing the selection of a vocal part in an exemplary embodiment of this application; Figure 2 Example diagram of solfège selection in an exemplary embodiment of this application; Figure 3 This is an example diagram illustrating the selection of sheet music timbre in an exemplary embodiment of this application; Figure 4 This is an example diagram of pitch adjustment in an exemplary embodiment of this application; Figure 5 This is yet another example diagram of pitch adjustment in an exemplary embodiment of this application; Figure 6 This is a flowchart illustrating the method for generating audio scores in an exemplary embodiment of this application. Figure 7 This is a schematic diagram of the structure of a music score audio generation system in an exemplary embodiment of this application.

[0020] Attached label: 1. Preset lower pitch threshold; 2. Preset upper pitch threshold; 3. Edge note. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" may be used interchangeably.

[0023] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0024] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0025] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0026] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0027] As used in this specification, the term "about" typically means + / -5% of the value, more typically + / -4% of the value, more typically + / -3% of the value, more typically + / -2% of the value, even more typically + / -1% of the value, and even more typically + / -0.5% of the value.

[0028] In this specification, certain embodiments may be disclosed in a range-bound format. It should be understood that this "range-bound" description is merely for convenience and brevity and should not be construed as a rigid limitation on the disclosed range. Therefore, the description of a range should be considered as having specifically disclosed all possible subranges and the individual numerical values ​​within those ranges. For example, a description of the range 1-6 should be considered as having specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and the individual numbers within those ranges, such as 1, 2, 3, 4, 5, and 6. This rule applies regardless of the breadth of the range.

[0029] Definition of the noun: Sight-singing, also known as music fundamentals training, involves learning to read aloud the corresponding solfège syllables (do, re, mi, etc.) while referring to sheet music such as simplified notation or staff notation (or the corresponding melody). Simultaneously and accurately reproduces the pitch, rhythm, dynamics, phrasing, and expression markings on the sheet music, transforming written musical symbols into a coherent melody. Sight-singing requires training in sight-reading, pitch accuracy, rhythm, and musicality. It serves as a crucial bridge connecting music theory, inner hearing, and vocal and instrumental performance, and is a fundamental skill that every instrument learner must master.

[0030] Singing from sheet music is a crucial link between visual symbols and auditory experience in piano learning. Through singing, learners can develop an auditory understanding and perception of pitch, rhythm, and phrasing before playing, effectively improving the accuracy and fluency of their performance. Simultaneously, the active awareness of melodic direction, breath control, and dynamic changes during the singing process helps performers transform abstract sheet music into concrete musical expression, enhancing their understanding of the work's structure and emotional delivery. Consistent singing practice not only significantly shortens sight-reading time but also allows learners to hear the expected sound effect before their fingers touch the keys, making their performance more lyrical and expressive. It is an indispensable foundation for comprehensively improving the quality of piano playing.

[0031] Example 1: Sight-singing is a fundamental training module in music education. However, the applicant noted that beginners often find it difficult to quickly form melodies in their minds when reading music, and their understanding of pitch relationships, phrasing, and musical expression is relatively weak. Singing music often becomes a major learning obstacle for beginners.

[0032] In order to help users (especially beginners) quickly understand and become familiar with the melody of the score, this application chooses to automatically generate the audio of the singing score based on the score.

[0033] For example, the process of automatically generating audio scores is as follows: Pre-collect a certain amount of audio material (e.g., collect sound material of an object at different pitches); The system identifies multiple pitches on a musical score, selects the corresponding pitch material based on these pitches, and then merges these pitch materials into a single audio recording.

[0034] However, the applicant noted that both actual sheet music practice and the generation of sheet music audio often encounter objective limitations: the sheet music frequently contains notes in extremely high or low registers, exceeding the comfortable vocal range of ordinary people. Forcing a performance in the original key not only produces a harsh and unpleasant tone but also easily disrupts pitch judgment due to the deliberate strain on the vocal cords.

[0035] In other words, the human voices collected often have a specific vocal range. If the voices are outside this range, the timbre may be poor, or the subject may not be able to produce sound at all. This can lead to a lack of material in the low or high registers when generating audio scores.

[0036] Therefore, for notes or passages that exceed the range of human voice, this application proposes a method for generating audio scores that can adapt to human singing ability without losing melodic lines (such as pitch fluctuations) and reducing changes in pitch logic, while taking into account both training effectiveness and vocal comfort.

[0037] Specifically, when there is a note with a very high pitch (such as greater than the 70th key), it can be sung in a lower octave (such as a pitch reduced by 12).

[0038] However, it's worth noting that adjusting a musical fragment might cause it to become incompatible with other parts of the musical phrase. For example, the adjusted notes might overlap in pitch with the original notes in the middle register.

[0039] In this application, when a pitch exceeding the preset upper / lower pitch threshold is fitted to a human voice, problems such as distortion, weak sound, and compression are likely to occur, which can be regarded as poor sound production effect. Pitches within the preset upper / lower pitch threshold can be regarded as good sound production effect.

[0040] For this, please see Figure 6 This application proposes a method for generating audio scores, including: S101, Obtain the score to be adjusted, wherein the score to be adjusted includes at least one musical unit; S102, Identify edge notes in the score to be adjusted; wherein, the edge notes refer to notes whose pitch is higher than a preset upper limit threshold or lower than a preset lower limit threshold. S103, Select an adjustment area based on the number or proportion of edge notes in the corresponding musical unit; the adjustment area is used to define the number or proportion of notes that are allowed or suggested to be modified; S103 includes: S1031, determine whether the number or proportion of the edge notes is greater than a preset number or preset proportion; If S1031 is yes, then the note in the first adjustment region is selected as the note to be adjusted; If S1031 is not, then the note in the second adjustment area is selected as the note to be adjusted; Wherein, the first adjustment region is larger than the second adjustment region; S104, adjust the pitch of the note to be adjusted to obtain the musical score pitch, and generate an adjusted score based on the musical score pitch and the musical score to be adjusted; S105, generate the singing audio for the adjusted score.

[0041] In some embodiments, a musical unit may refer to a measure, phrase, section, or voice part.

[0042] In some embodiments, S103 includes: the greater the number or proportion of edge notes in the corresponding musical unit, the larger the range of the adjustment area.

[0043] In some embodiments, a musical unit with at least one edge note can be considered as a target musical unit.

[0044] Correspondingly, the first adjustment area can refer to all notes in the target musical unit. The second adjustment area can refer to all edge notes.

[0045] In other embodiments, if the number of non-edge notes between two adjacent edge notes in the current music unit is less than a preset threshold, the second adjustment area can include the non-edge notes between the edge notes as well as the notes to be adjusted, in addition to selecting the edge notes.

[0046] For example, if the first and third notes in the current musical unit are edge notes, and the second note is a non-edge note between them, then the first, second, and third notes can be placed together in the second adjustment area, that is, their pitch can be adjusted together.

[0047] It should be understood that this application aims to both modify edge notes to fit the human vocal range and minimize alterations to the original score while preserving the original melodic direction. To this end, this embodiment provides a collaborative mechanism for local and overall pitch adjustment of edge notes. Specifically, when the number or proportion of edge notes is large, the overall (or large-scale) pitch adjustment of the target musical unit is performed; while when the number or proportion of edge notes is small, the local (or small-scale) pitch adjustment of the target musical unit is performed.

[0048] It is worth noting that when the number or proportion of edge notes is small, local pitch adjustment of the target musical unit (such as adjusting the pitch of only one edge note) will have a reasonable impact on the overall melody and the fluctuation of musical phrases, thus reducing the difficulty of singing the score while effectively avoiding obvious disharmony.

[0049] For example, a measure (i.e., the target musical unit) has 8 notes, of which 7 notes have pitches between the preset upper pitch threshold 2 and the preset lower pitch threshold 1; 1 note is above the preset upper pitch threshold 2, and this note can be identified as an edge note 3. In this target musical unit, there is only one edge note 3, which is less than the preset proportion. For this, the second adjustment area can be selected. If only this single edge note needs to have its pitch lowered locally, please refer to [link to relevant documentation]. Figure 4 or Figure 5 For example, the pitch of the edge note 3 can be shifted down by an octave (i.e., without changing the solfège, only the pitch is lowered by 12 semitones, such as adjusting high do (C5) to middle do (C4)).

[0050] In this embodiment, the original pitches of the remaining 7 notes in the measure remain unchanged, and only the pitch of a single edge note is slightly adjusted by lowering it by an octave. The overall melody of the measure after adjustment will not sound too abrupt.

[0051] In other embodiments, when the number or proportion of edge notes is relatively large (e.g., 7 out of 8 notes in the target musical unit are edge notes), the pitch of all or most notes is synchronously shifted as a whole (e.g., the same pitch adjustment range). The relative intervals (i.e. pitch differences) between notes hardly change, which can reduce the difficulty of singing the score while reducing the impact of pitch adjustment on the original melody.

[0052] It should be understood that sight-reading itself has a relatively high barrier to entry for music beginners, requiring users to have a certain foundation in music theory. Using audio recordings of sheet music allows beginners to quickly grasp the melodic outline of a complete piece, effectively reducing the learning pressure when practicing unfamiliar music for the first time.

[0053] This application is particularly suitable for scenarios where it automatically generates vocal-adapted audio scores for users to sing along and imitate, thus familiarizing themselves with the melody. This application can both assist users in quickly mastering the melody and improving the efficiency of practicing and learning music through the generated audio scores, and reduce the difficulty of singing along by appropriately adjusting the pitch.

[0054] For example, the pitch adjustment amount may include: raising / lowering by 8 degrees, or raising / lowering by 15 degrees.

[0055] Preferably, for notes to be adjusted that are in the same target musical unit, the pitch adjustment range is the same (e.g., all are raised by an octave).

[0056] In some embodiments, S104 may include: adjusting the pitch of the notes to be adjusted in the score to obtain the pitch of the singing score, keeping the original pitch of the other notes unchanged, and combining the adjusted notes and the unadjusted notes according to the original sequence in the score to obtain the adjusted score.

[0057] For example, the score to be adjusted (e.g., with one musical unit and four notes): C5, D5, E5, G5. The preset upper limit threshold for pitch is C5. Therefore, D5, E5, and G5 are considered edge notes, and since the proportion of edge notes exceeds the preset ratio, the first adjustment area is selected. D5, E5, and G5 are uniformly lowered by an octave to obtain the corresponding musical notation pitches (D4, E4, G4); C5 remains unchanged, retaining its original pitch. The notes are then concatenated in the original note order to generate the adjusted score: C5, D4, E4, G4.

[0058] In some embodiments, S105 includes: Obtain the sound attributes of the music unit, including frequency and / or volume; The musical score audio is generated based on the musical unit and the sound attributes.

[0059] For example, the sound attribute can be a preset default parameter for the score generation process, such as setting a recommended frequency and volume for at least one musical note unit.

[0060] In some embodiments, the step of obtaining the sound attributes of the music unit includes: Identify the pitch difference between the reference syllable pitch of at least one adjustment unit and the reference syllable pitch of an adjacent musical unit; wherein the adjustment unit is a musical unit that has undergone pitch adjustment.

[0061] If at least one pitch difference is less than or equal to the set first pitch threshold, then proceed with the following steps: Update the sound attributes for the corresponding adjustment unit and / or the adjacent music unit.

[0062] In other words, in this embodiment, when the pitch difference between two adjacent units is not obvious due to pitch adjustment, the auditory distinction can be made by adjusting the sound attributes of the two units.

[0063] The reference pitch is the size of the register in which the musical unit is located. Alternatively, the reference pitch can be used to characterize the overall pitch of the register in which the musical unit is located.

[0064] In some embodiments, a human vocal melody can be synthesized by adjusting the pitch, duration, and other musical information of the notes in the score, thereby obtaining the audio of the sung score.

[0065] In some embodiments, music units generally have default settings. If a music unit has its own sound attributes, the settings can be adjusted according to the sound attributes (such as increasing the frequency and / or volume) to synthesize the score audio.

[0066] In some embodiments, the reference pitch can refer to a representative pitch used to characterize the overall pitch characteristics of a single musical unit. For example, the reference pitch can refer to the median of the pitches of the first note, the last note, or all notes within a musical unit.

[0067] For example, the note sequence of a certain musical unit (measure) is: D4, E4, G4, C5.

[0068] If the first note is taken as the reference pitch, then the reference pitch for this musical unit is D4. If the last note is taken as the reference pitch, then the reference pitch for this musical unit is C5. If the mid-range note is taken as the reference pitch, then the reference pitch for this musical unit is E4.

[0069] If the reference pitch of the adjacent musical unit is A4, then the pitch difference between the adjustment unit and the adjacent musical unit is the interval between D4 and A4 (or C5 and A4, E4 and A4).

[0070] Alternatively, in other embodiments, the vocal range or the range of a standard instrument (such as the full 88-key range of a piano) can be divided into multiple consecutive pitch ranges according to pitch intervals, and each pitch range is assigned a unique pitch range number. When all the notes in a target musical unit fall within the same pitch range, the entire musical unit can be assigned to the corresponding pitch range, and the pitch range number can be used as the reference pitch of the musical unit to represent the overall pitch of the musical unit.

[0071] In some embodiments, an adjacent musical unit can refer to a musical unit that is adjacent to the adjustment unit in terms of sound production time. For example, in the same vocal part, the preceding or following phrase of the adjustment unit.

[0072] In other embodiments, an adjacent musical unit may also refer to a musical unit in a multi-part score that sounds simultaneously with the adjustment unit. For example, if the adjustment unit is located in the main melody part, the adjacent musical unit may be a musical unit in the accompaniment or harmony part that sounds during the same period.

[0073] In some embodiments, the sound attribute of update frequency can refer to high- or low-cutting of audio. For example, low-cutting refers to low-pass filtering of audio, that is, filtering out low-frequency components of audio, making the sound clear and prominent; high-cutting refers to high-pass filtering of audio, that is, filtering out high-frequency components of audio, making the sound deep and resonant.

[0074] In this embodiment, if the pitch difference between the adjustment unit after pitch adjustment and the reference score of the adjacent musical unit is too small, it may affect the singing effect due to the indistinguishability of melody layers. For example, when multiple parts are played simultaneously, the pitch ranges of two melodies may partially overlap, and the melodic lines of the parts may interfere with each other; or, for example, adjacent musical units in the same part may have insufficient pitch difference between the preceding and following phrases, making it difficult for the user to clearly distinguish the approximate pitch range of the corresponding phrase.

[0075] In this regard, this application chooses to update the sound attributes of the adjustment unit and / or the adjacent music unit. Based on this, even if the pitch range between the adjustment unit and the adjacent music unit is similar, the user can easily and directly distinguish the two music units by the difference in volume or frequency.

[0076] In other words, for scores with high complexity, this application proposes a sound attribute update mechanism on the basis of pitch adjustment. That is, when the pitch difference is not obvious, different musical units are distinguished by changes in timbre or volume, so as to make the generated singing score audio sound more consistent with the layering of the original score in a simpler way.

[0077] Therefore, this application achieves a good balance between ensuring that the audio of the score can be sung normally and ensuring that the voice parts / phrases can be clearly distinguished. Specifically, on the one hand, this application adjusts the pitch of the edge notes to adjust the notes to the effective vocal range of the selected sound material, which can effectively avoid the situation that the notes are beyond the human voice's ability to be sung (or the singing effect is very poor); on the other hand, by using audio sound attribute compensation, it ensures that there is sufficient auditory distinction between different musical units and different voice parts, and prevents the overlap of voice ranges from causing melody confusion.

[0078] In other words, pitch adjustment ensures that the melody of the score can be sung normally from the primary dimension (pitch) (i.e., ensuring that the user can sing it, or making the generated score audio sound like it conforms to auditory habits). Updating sound attributes can further fine-tune the relative relationships between musical units from the secondary dimensions (volume and frequency), making the musical units and voices within the melody more distinct, thereby reducing the impact of pitch adjustment on the original score and improving the overall quality of the score audio.

[0079] In some embodiments, S104 includes: decreasing or increasing the pitch of the note to be adjusted by a suggested adjustment amount; and ensuring that the pitch difference between the reference pitch of the corresponding adjustment unit and the reference pitch of the adjacent music unit is greater than a set second pitch threshold.

[0080] Preferably, the first pitch threshold is greater than the second pitch threshold.

[0081] In other words, in this embodiment, for some complex musical scores, the second pitch threshold can be set very small, that is, to reduce the restriction on the adjustment scale during the pitch adjustment process.

[0082] The suggested adjustment amount refers to the range of pitch adjustment, which can be quantified in octaves. For example, the suggested adjustment amount could be raising or lowering the pitch of a note by one or two octaves.

[0083] In some embodiments, pitch adjustment may be performed at least once according to the recommended adjustment amount. Alternatively, pitch adjustment may be performed multiple times according to the recommended adjustment amount.

[0084] In this embodiment, the present application ensures that there is a sufficient pitch difference between the adjustment unit and the adjacent musical unit, which helps to make the generated musical score melody more in line with the actual trend of melody change.

[0085] The following will describe a complete process of generating audio for sheet music in conjunction with specific embodiments: S1 adjusts the pitch of edge notes; You can try adjusting it multiple times, and each time it is recommended to adjust it by 8 degrees or 15 degrees. Each pitch adjustment yields several adjusted pitches, such as adjusted pitch a after the first pitch adjustment, adjusted pitch b after the second pitch adjustment, and adjusted pitch c after the third pitch adjustment. S2, after each adjustment attempt, determine whether the adjusted pitch obtained by the latest adjustment falls within the pitch range between the preset upper limit threshold and the preset lower limit threshold; S3, if the result of S2 is negative, continue to try adjusting; S4. If the result of S2 is yes, then pitch adjustment can be paused, and it can be determined whether the pitch difference is greater than the second pitch threshold.

[0086] If the result of S4 is yes, then proceed to S105.

[0087] If the S4 judgment result is negative, the user can choose to force a fit (e.g., directly fit the adjusted pitch b after the previous pitch adjustment using human voice), meaning that the user can choose to accept a pitch b that is slightly beyond the normal range of human voice. Alternatively, the user can choose to accept an adjusted pitch c that is slightly less than satisfactory in terms of audio discrimination, while triggering audio compensation.

[0088] In this embodiment, the audio generation process of the score should meet two conditions as much as possible: 1) Absolute condition, that is, the pitch of the score is preferably within the normal range of human voice; 2) Relative condition, that is, the pitch difference between the reference score pitch of the adjustment unit and the adjacent music unit must be at least greater than the second pitch threshold.

[0089] In some embodiments, including: The second pitch threshold is updated based on the complexity of the score to be adjusted; the complexity is determined based on the number of voice parts. The more voice parts a sound has, the higher the complexity, and correspondingly the smaller the second pitch threshold.

[0090] In this embodiment, if the score is highly complex, it may require a higher degree of differentiation between musical units / parts (if the difference in sound between the adjustment unit and adjacent musical units is not obvious, it may lead to muddy harmony). In this case, this application chooses to reduce the second pitch threshold, making it easier to trigger audio compensation (i.e., volume and / or frequency compensation) to improve the differentiation of different musical units / parts through volume or frequency differentiation, thereby improving the quality of the sung audio.

[0091] Alternatively, if the score is highly complex, by reducing the second pitch threshold, the number of pitch adjustments can be effectively controlled within a reasonable range, avoiding excessive pitch adjustments from having too much impact on the already complex original melody. Thus, this application can achieve a good balance between ensuring that the audio of the score can be sung normally and ensuring that the voice parts / musical phrases can be clearly distinguished.

[0092] In other words, considering the difficulty of adjusting pitch in complex scores—for example, where more than half of the musical units in a complex score require pitch adjustment—it's difficult to guarantee that the sung audio will retain the original melody. Or, for complex scores, excessive pitch adjustments may cause unpredictable changes to the overall sound of multiple musical units and parts. Therefore, this embodiment chooses to lower the second pitch threshold and introduce more external audio compensation. This helps the sung audio to maintain sufficient distinction with the help of external factors after fewer pitch adjustments.

[0093] When the musical score is less complex, this application chooses to maintain or increase the second pitch threshold. That is to say, for simpler musical scores, pitch correction is also simpler. Therefore, this application chooses to maintain or increase the second pitch threshold, so that without or with only a small degree of intervention from external factors (i.e., updating the sound attributes with audio compensation), the required singing audio can be generated by pitch correction.

[0094] In other words, based on pitch correction, this application can achieve a better balance between the sound production effect of the score audio and the distinction between voice parts / phrases through dynamic control of external factors.

[0095] In some embodiments, it also includes: A sound material library is pre-established, and the sound material library stores at least one sound material. In response to the user's selection command, the corresponding sound material is selected as the sound material; Correspondingly, in S105, the singing score audio is generated based on the selected sound material and the adjusted score.

[0096] For example, if a user selects a female alto voice as the selected sound material, the corresponding vocal sample segment can be retrieved according to the pitch of the notes in the score, and finally multiple vocal sample segments can be spliced ​​together to form the audio of the score.

[0097] Alternatively, an audio generation AI model can be used to generate a musical score audio based on the selected sound material and the adjusted musical score.

[0098] For example, an AI model for audio generation could be a VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) singing synthesis model.

[0099] For example, you can input human voice audio (such as clean human voice) and corresponding musical notation (such as notes corresponding to pitch) into the model, and the model will output a standard singing audio segment that conforms to the musical notation and is sung by that human voice.

[0100] In this embodiment, the sound material library stores different singer timbres, and users can select their desired timbres to generate sheet music audio, thereby helping to enhance the fun of sheet music audio generation.

[0101] In some embodiments, selected sound materials can be selected from existing sound libraries, or a self-made sound library can be recorded or a self-developed sound library can be generated. Based on the self-made sound library or the self-developed sound library, an AI model is trained to generate an audio track corresponding to the score, further optimizing the singing timbre, pitch and rhythm performance, and improving the fit and realism of the score audio.

[0102] In some embodiments, the method further includes: The corresponding preset pitch upper limit threshold and preset pitch lower limit threshold are determined based on the selected sound material.

[0103] Among them, acoustic analysis can be performed on selected sound materials to extract the stable and natural vocal range of the human voice, and then the highest note of the vocal range is used as the preset upper limit threshold and the lowest note is used as the preset lower limit threshold.

[0104] For example, vocal range can refer to the pitch range within which a voice will not crack or become breathy. Alternatively, vocal range can also refer to the pitch range between the highest and lowest notes that a person's voice can produce.

[0105] In some embodiments, including: The suggested adjustment amount is updated based on the preset upper pitch threshold or the preset lower pitch threshold; including: Identify the pitch difference between the edge note and its adjacent preset pitch upper limit threshold or preset pitch lower limit threshold; The suggested adjustment amount is updated based on the pitch difference; Wherein: the smaller the pitch difference, the smaller the suggested adjustment amount.

[0106] Alternatively, in other embodiments, the suggested adjustment amount is a pre-set fixed value.

[0107] In this embodiment, considering the differences in the vocal range of different timbres, this application also provides a personalized update mechanism for pitch adjustment. Specifically, if the pitch of an edge note is far from the current vocal range limit, a larger suggested adjustment amount (e.g., raising / lowering by 15 degrees each time) can be applied to resolve the issue of the edge note's pitch exceeding the limit with fewer adjustments. If the pitch of an edge note is close to the current vocal range limit, a smaller suggested adjustment amount (e.g., raising / lowering by 8 degrees each time) can be applied, i.e., a more conservative pitch adjustment can be made to avoid more serious listening problems caused by improper adjustment.

[0108] In other words, this application can achieve a good balance between the adjustment effect and adjustment efficiency of edge notes.

[0109] Please see Figure 7 This application proposes a system for generating audio scores, comprising: The music score acquisition module is used to acquire the music score to be adjusted, and the music score to be adjusted includes at least one musical unit; An edge recognition module is used to identify edge notes in the score to be adjusted; wherein, the edge note refers to a note whose pitch is higher than a preset upper limit threshold or lower than a preset lower limit threshold. The adjustment selection module is used to select an adjustment area based on the number or proportion of edge notes in the corresponding musical unit; the adjustment area is used to define the number or proportion of notes that are allowed or suggested to be modified; The adjustment selection module includes: The judgment unit is used to determine whether the number or proportion of the edge notes is greater than a preset number or preset proportion; If the result of the judgment unit is yes, then the note in the first adjustment area is selected as the note to be adjusted; If the result of the judgment unit is negative, then the note in the second adjustment area is selected as the note to be adjusted; Wherein, the first adjustment region is larger than the second adjustment region; The music score generation module is used to adjust the pitch of the notes to be adjusted to obtain the pitch of the musical score, and to generate an adjusted music score based on the pitch of the musical score and the music score to be adjusted. An audio generation module is used to generate a vocal score audio for the adjusted musical score.

[0110] In some embodiments, the audio generation module is used to perform: Obtain the sound attributes of the music unit, including frequency and / or volume; The musical score audio is generated based on the musical unit and the sound attributes.

[0111] In some embodiments, if the extreme note is in a musical phrase (e.g., the notes are connected together), and the notes in the phrase are all relatively far from the original notes in the middle register, the entire phrase can be adjusted together. However, if this extreme note belongs to a musical phrase but is close to an existing note in the middle register, adjusting the entire phrase may cause auditory conflict. In this case, you can adjust only this extreme note.

[0112] In some embodiments, different timbre libraries / style libraries can be pre-collected, and different ranges of the middle register can be preset for different timbres / styles. For example, male singers have a lower timbre and a narrower vocal range, so the coverage of their middle register can be narrowed.

[0113] For example, in some preferred embodiments, the system can pre-complete the offline acquisition and standardized modeling storage of multiple types of timbre libraries and singing style libraries for users to select, and can individually configure differentiated intermediate voice range threshold intervals and effective coverage parameters for each type of timbre and each singing style in the library.

[0114] In the early data collection phase, vocal samples of different genders, voice types, and singing styles can be collected in batches. The inherent acoustic properties of various vocal types are analyzed through audio feature extraction algorithms to divide the appropriate intermediate vocal range and complete parameter presets.

[0115] Taking male vocal timbre as an example, male voices generally have a lower fundamental frequency and a thicker, deeper tone. Their natural effective vocal range is narrower overall, and their ability to support high notes is limited. If a universal standard middle register is used, problems such as abrupt transitions, timbre distortion, and voice cracking will occur. Therefore, for male singer-specific timbre samples, the upper and lower limits of their middle register can be narrowed to reduce the suitable vocal range. This allows the algorithm to prioritize melody generation and audio rendering within the pitch range that aligns with the physiological comfort zone of male voices when running on this timbre.

[0116] Correspondingly, it can also match different specifications of the middle range of the vocal range, such as wide, narrow, high, and low, for other timbres / styles, such as female voices, children's voices, deep opera voices, clear pop voices, and rock screams. When calling the corresponding timbre library, the system automatically loads the pre-configured range parameters without the need for manual real-time adjustment. This ensures that the timbre is natural and uniform and the pitch is stable when various human voices are sung, and also avoids the dissonance and distortion caused by exceeding the comfortable range of human voices, greatly improving the realism of AI singing and the fit of human voices.

[0117] Further, please see Figure 1 , Figure 2 , Figure 3 Users can preset different voice parts, solfège names (such as fixed solfège, movable solfège, note name solfège, etc.), and timbre, and then generate the corresponding musical score audio based on the selection.

[0118] Example 2: After adjusting the pitch, further determine whether there is a problem of pitch overlap (or too close pitch). If there is no conflict of pitch overlap (or too close pitch), the adjustment plan can be accepted. However, if there are conflicts due to overlapping (or excessively close) pitches, audio compensation solutions can be used (in the case of multiple parts, compensation can be achieved by adjusting volume and high / low cut): Taking volume as an example: increase the volume of the melody part (right hand main melody) and decrease the volume of the accompaniment part (left hand bass); Alternatively, the priority of the vocal parts can be determined in other ways, and the volume of the higher priority parts can be increased. Taking high-low cut as an example, the main melody part raises the mid-frequency (the frequency range that the human ear is sensitive to), while the accompaniment part lowers the mid-frequency, thus increasing the recognizability of overlapping notes in terms of timbre and auditory perception.

[0119] It should be noted that the sound in this embodiment can originate from a pre-built database: First, select piano pieces with diverse styles and comprehensive voice parts as recording tracks and prepare the scores.

[0120] To expand the database, recording personnel with different genders, ages, and singing levels can be selected.

[0121] Configure recording equipment in an acoustically compliant recording environment, acquire data in a lossless format, and preset the sampling rate, bit depth, recording distance, and gain level.

[0122] The recording staff sang under the guidance of the accompaniment, and the dry vocals without post-production processing and the auxiliary audio track with accompaniment reference notes were recorded simultaneously.

[0123] The same singer performs the same passage multiple times with different emotional tones, vocal registers, and dynamics to cover the diversity of singing styles.

[0124] During the acquisition process, the effective singing range is detected in real time. For segments with clipping, excessive background noise, or interruptions, a re-recording prompt is given, and invalid data is removed.

[0125] Specifically, the data can undergo the following preprocessing steps: The original audio is processed by removing DC offset, removing silence, and normalizing loudness, and the dry vocals are stored separately from the accompaniment reference track (for audio extracted from an existing music library).

[0126] Each piece of singing data is assigned an emotion category and a singing style label.

[0127] The system calculates pitch and rhythm deviations in singing, assesses pronunciation clarity using a speech recognition model, and discards or re-records unqualified data.

[0128] The qualified data is stored hierarchically according to dimensions such as singer, song, and style, and metadata indexes are recorded to build a database of vocal materials.

[0129] In some embodiments, this application may support users in selecting different methods for generating sheet music audio, such as note names, solfège syllables, etc.

[0130] In some embodiments, the audio of the aria can be synchronized with the vectorized score, allowing users to practice phrase by phrase / measure by measure / slowly / in a loop.

[0131] In some embodiments, the audio of singing sheet music can be upgraded into a teaching tool through methods such as sentence demonstration, singing along and scoring, listening, and imitating the score.

[0132] Example 3: In some embodiments, to achieve simultaneous production of monophonic intervals, chords, and multiple voices, a multi-track simultaneous playback scheme can be adopted: that is, by assigning an independent MIDI track to each sound point, the multi-voice content can be played in parallel at the same time.

[0133] For example, analyze the XML information of each score individually to count the number of notes in the densest column for each voice part in the entire piece (i.e., the maximum number of notes simultaneously played in a single column for that voice part). Add up the number of notes in the densest single column for all voice parts; the result is the total number of tracks required for all voice parts to play simultaneously. During actual playback, the total number of notes playing simultaneously at most points in time is less than this total number of tracks; unassigned tracks play silently and do not output any valid sound.

[0134] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a computer terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0136] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for generating audio for singing scores, characterized in that, include: S101, Obtain the score to be adjusted, wherein the score to be adjusted includes at least one musical unit; S102, Identify edge notes in the score to be adjusted; wherein, the edge notes refer to notes whose pitch is higher than a preset upper limit threshold or lower than a preset lower limit threshold. S103, Select an adjustment area based on the number or proportion of edge notes in the corresponding musical unit; the adjustment area is used to define the number or proportion of notes that are allowed or suggested to be modified; S103 includes: S1031, determine whether the number or proportion of the edge notes is greater than a preset number or preset proportion; If S1031 is yes, then the note in the first adjustment region is selected as the note to be adjusted; If S1031 is not, then the note in the second adjustment area is selected as the note to be adjusted; Wherein, the first adjustment region is larger than the second adjustment region; S104, adjust the pitch of the note to be adjusted to obtain the musical score pitch, and generate an adjusted score based on the musical score pitch and the musical score to be adjusted; S104 includes: decreasing or increasing the pitch of the note to be adjusted according to the suggested adjustment amount; and ensuring that the pitch difference between the reference pitch of the corresponding adjustment unit and the reference pitch of the adjacent music unit is greater than the set second pitch threshold. The suggested adjustment amount is updated based on the preset upper pitch threshold or the preset lower pitch threshold; including: identifying the pitch difference between the edge note and its adjacent preset upper pitch threshold or preset lower pitch threshold; updating the suggested adjustment amount based on the pitch difference; wherein: the smaller the pitch difference, the smaller the suggested adjustment amount; S105, generate the singing audio for the adjusted score.

2. The method for generating audio scores according to claim 1, characterized in that, S105 includes: Obtain the sound attributes of the music unit, including frequency and / or volume; The musical score audio is generated based on the musical unit and the sound attributes.

3. The method for generating audio scores according to claim 2, characterized in that, The steps for obtaining the sound attributes of the music unit include: Identify the pitch difference between the reference sung pitch of at least one adjustment unit and the reference sung pitch of an adjacent musical unit; wherein, the adjustment unit is a musical unit that has undergone pitch adjustment, and the reference sung pitch refers to a representative pitch used to characterize the overall pitch characteristics of a single musical unit; If at least one pitch difference is less than or equal to a set first pitch threshold, then proceed with the following steps: Update the sound attributes for the corresponding adjustment unit and / or the adjacent music unit.

4. The method for generating audio scores according to claim 1, characterized in that, include: The second pitch threshold is updated based on the complexity of the score to be adjusted; The complexity is determined based on the number of voice parts; The more voice parts a sound has, the higher the complexity, and correspondingly the smaller the second pitch threshold.

5. The method for generating audio scores according to claim 1, characterized in that, The method further includes: The corresponding preset pitch upper limit threshold and preset pitch lower limit threshold are determined based on the selected sound material.

6. The method for generating audio scores according to claim 1, characterized in that, Also includes: A sound material library is pre-established, and the sound material library stores at least one sound material. In response to the user's selection command, the corresponding sound material is selected as the sound material; Correspondingly, in S105, the singing score audio is generated based on the selected sound material and the adjusted score.

7. A system for generating audio scores, characterized in that, include: The music score acquisition module is used to acquire the music score to be adjusted, and the music score to be adjusted includes at least one musical unit; An edge recognition module is used to identify edge notes in the score to be adjusted; wherein, the edge note refers to a note whose pitch is higher than a preset upper limit threshold or lower than a preset lower limit threshold. The adjustment selection module is used to select an adjustment area based on the number or proportion of edge notes in the corresponding musical unit; the adjustment area is used to define the number or proportion of notes that are allowed or suggested to be modified; The adjustment selection module includes: The judgment unit is used to determine whether the number or proportion of the edge notes is greater than a preset number or preset proportion; If the result of the judgment unit is yes, then the note in the first adjustment area is selected as the note to be adjusted; If the result of the judgment unit is negative, then the note in the second adjustment area is selected as the note to be adjusted; Wherein, the first adjustment region is larger than the second adjustment region; The music score generation module is used to adjust the pitch of the notes to be adjusted to obtain the pitch of the musical score, and to generate an adjusted music score based on the pitch of the musical score and the music score to be adjusted. The audio generation module is used to generate the audio score for the adjusted musical score; The system is used to perform: The pitch of the note to be adjusted is reduced or increased as suggested; and the pitch difference between the reference pitch of the corresponding adjustment unit and the reference pitch of the adjacent music unit is greater than the set second pitch threshold. The suggested adjustment amount is updated based on the preset upper pitch threshold or the preset lower pitch threshold; including: identifying the pitch difference between the edge note and its adjacent preset upper pitch threshold or preset lower pitch threshold; updating the suggested adjustment amount based on the pitch difference; wherein: the smaller the pitch difference, the smaller the suggested adjustment amount.

8. The audio generation system for sheet music according to claim 7, characterized in that, The audio generation module is used to perform: Obtain the sound attributes of the music unit, including frequency and / or volume; The musical score audio is generated based on the musical unit and the sound attributes.

Images