Vocal synthesis methods, systems and computer equipment
By adding sound reflex information to the singing vocal synthesis technology, the problem of insufficient expression of singing vocal synthesis in the existing technology is solved, and singing vocal synthesis with more personal characteristics and rich emotions is achieved, adapting to a variety of synthesis models.
Patent Information
- Application Number
- CN202210529792.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-16
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-05-16
AI Technical Summary
The existing singing vocal synthesis technology is not as good as the vocals in terms of expressiveness and nature, and lacks personal characteristics. It requires large recording volume and high coverage requirements, and consumes a lot of manpower and material resources.
By dividing the digital music score into multiple music score segments, obtain the frame number sequence and pitch ID sequence, enter the sound prediction model to obtain the sound position information and pitch information, and add the sound information to the music score, and finally enter the song synthesis model Perform synthesis.
It improves the expressiveness of the synthetic vocals, makes it more personal, conveys rich emotions, and adapts to various vocal synthesis models without additional training on the vocal synthesis model.
Smart Images
Figure CN114974183B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical fields related to artificial intelligence and vocal processing, and more specifically, to a singing synthesis method, system and computer equipment. Background Art
[0002] Singing is not only a medium for people to convey information, but also an art form that conveys the thoughts of creators and singers. With the continuous development of artificial intelligence and computer technology, enabling computers to sing songs as beautifully as humans is an essential research topic in artificial intelligence, and it is also one of the hot research areas of computer music.
[0003] Singing synthesis technology is a computer-generated method of synthesizing human singing based on musical scores. Today, singing synthesis technology has brought certain changes to modern life. For example, the appearance of virtual singers who are popular all over the world, such as Hatsune Miku and Luo Tianyi, has lowered the threshold for music production, allowing many creators to simply use their own laptops without having to hire professional bands or singers to create their own songs. At present, many popular songs have been created with the help of virtual singers.
[0004] However, except for the original singing synthesis technology based on unit selection and waveform splicing, the current synthesized singing is not as expressive and natural as human voice, and also lacks some personal characteristics. In addition, splicing synthesis technology requires a large amount of recordings and high coverage requirements, which consumes a lot of manpower and material resources. Therefore, how to improve the expressiveness of synthesized singing is an important research topic at present so that singing synthesis technology can be put into market applications (such as virtual singers, computer music creation, music education, etc.). Summary of the invention
[0005] Based on the above, in order to at least improve the expressiveness of synthesized singing, in a first aspect, an embodiment of the present application provides a singing synthesis method, the method comprising:
[0006] Segment the digital music score to obtain multiple music score segments, and obtain a frame number sequence and a pitch ID sequence of each music score segment;
[0007] Inputting the frame number sequence and the pitch ID sequence of each of the music score segments into the appoggiatura prediction model in sequence to obtain the appoggiatura position information and the appoggiatura pitch information of the digital music score;
[0008] Adding corresponding grace note information to the digital music score according to the grace note position information and grace note pitch information;
[0009] The digital music score with the added appoggiatura information is input into the singing synthesis model for singing synthesis.
[0010] In an implementation manner of the first aspect, the digital music score is segmented to obtain a plurality of music score segments, including:
[0011] Extracting the scale information and duration information of each note from the digital music score;
[0012] Pre-processing the digital music score according to the scale information and duration information;
[0013] The digital music score is segmented according to the rests and breaths in the pre-processed digital music score to obtain a plurality of music score segments, wherein each music score segment includes at least three notes, and the beginning and the end of each music score segment respectively include a breath note duration and a rest duration.
[0014] In an implementation manner of the first aspect, pre-processing the digital music score according to the scale information and the duration information includes:
[0015] Merging consecutive characters of the same word with the same musical scale in the digital music score, and taking the sum of the durations of the characters before merging as the duration of the merged character; and / or
[0016] Consecutive rests and consecutive breath notes are merged, and the sum of the durations of the rest notes before the merger is used as the duration of the rest note after the merger, and the sum of the durations of the breath notes before the merger is used as the duration of the breath note after the merger.
[0017] According to an implementation of the first aspect, obtaining a frame number sequence and a pitch ID sequence of each music score segment and adding corresponding appoggiatura information includes:
[0018] For each music score segment, convert each scale in the music score segment into a corresponding pitch ID to obtain a pitch ID sequence;
[0019] The floating-point duration of each scale in the music score segment is converted into a frame number to obtain a frame number sequence.
[0020] Based on an implementation manner of the first aspect, the appoggiatura prediction model includes an embedding layer, a bidirectional long short-term memory layer, a fully connected layer, and a conditional random field layer, wherein the frame number sequence and the pitch ID sequence of each of the music score segments are sequentially input into the appoggiatura prediction model to obtain the appoggiatura position information and the appoggiatura pitch information of the digital music score, including:
[0021] Inputting the frame number sequence and the pitch ID sequence of the music score segment into the embedding layer, performing vector mapping on the frame number sequence and the pitch ID sequence to obtain the duration vector and the pitch vector of each note of the music score segment, and adding the duration vector and the pitch vector of each note to obtain the input feature vector corresponding to each note;
[0022] The input feature vectors corresponding to the notes are input into the bidirectional long short-term memory layer for processing, and the output of the bidirectional long short-term memory layer is sent to the fully connected layer to obtain the scores of each input feature vector for multiple prediction labels; the scores of each input feature vector for each prediction label are input into the conditional random field layer, and the conditional random field layer obtains the appoggiatura position sequence and the appoggiatura pitch difference sequence corresponding to the music score segment according to the scores of each input feature vector for each prediction label; the appoggiatura position sequence includes an identifier indicating whether the corresponding position contains an appoggiatura, and the appoggiatura pitch difference sequence includes the appoggiatura pitch difference at each position in the music score segment.
[0023] Based on an implementation manner of the first aspect, the prediction label includes an appoggiatura prediction label and a pitch prediction label, the appoggiatura prediction label includes four labels of "sentence beginning", "sentence end", "appoggiatura", and "non-appoggiatura", and the input feature vector is predicted to be a probability value of each of the appoggiatura prediction labels as a score of the input feature vector for each of the appoggiatura prediction labels; the pitch prediction label includes five pitch prediction labels of "beginning", "end", "pitch difference is 0", "pitch difference is 1", and "pitch difference is 2", and the input feature vector is predicted to be a probability value of each of the pitch prediction labels as a score of the input feature vector for each of the pitch prediction labels; a pitch difference of 0 indicates that the corresponding note position does not contain an appoggiatura, a pitch difference of 1 indicates that the appoggiatura at the corresponding note position is one semitone lower than the pitch of the subsequent main tone, and a pitch difference of 2 indicates that the appoggiatura at the corresponding note position is one whole tone lower than the pitch of the subsequent main tone.
[0024] According to an implementation of the first aspect, adding corresponding grace note information to the digital music score according to the grace note position information and the grace note pitch information includes:
[0025] According to the sequence of appoggiatura positions, adding an appoggiatura scale to the notes corresponding to each predicted appoggiatura position in each of the music score segments;
[0026] Setting a corresponding duration for the note at the appoggiatura position according to the duration of a main tone after the appoggiatura position, and adjusting the duration of the main tone to the difference between the original duration of the main tone and the duration of the note at the appoggiatura position;
[0027] The lyrics of the main note are moved to the position of the previous appoggiatura, and the lyrics of the main note are changed to a fermata symbol to obtain a music score segment after the appoggiatura processing;
[0028] The rests and breath notes at the beginning of each music score fragment after the appoggiatura processing are removed and then spliced to obtain a digital music score with appoggiatura information added.
[0029] According to an implementation of the first aspect, the method further includes:
[0030] Acquire a music score data set, the music score data set comprising a plurality of digital music score segment samples with labels, the label of each digital music score segment sample comprising a flag sequence of whether each note position has an appoggiatura and a pitch difference sequence of the appoggiatura and the tonic when each note position has an appoggiatura, wherein if there is no appoggiatura at a certain position, the corresponding pitch difference is 0, and if there is an appoggiatura, the pitch difference is 1 or 2, 1 indicating that the appoggiatura at the corresponding position is one semitone lower than the subsequent tonic, and 2 indicating that the appoggiatura at the corresponding position is one whole tone lower than the subsequent tonic;
[0031] Each digital music score fragment sample is input into a pre-selected AI model in sequence, a loss function is calculated according to the result output by the AI model and the label of the digital music score fragment sample, and the parameters of the AI model are adjusted by the loss function until the training converges, thereby obtaining the trained AI model as the appoggiatura prediction model.
[0032] In a second aspect, an embodiment of the present application further provides a singing voice synthesis system, which is applied to a computer device, and the singing voice synthesis system includes:
[0033] A segmentation module is used to segment the digital music score to obtain multiple music score segments, and obtain a frame number sequence and a pitch ID sequence of each music score segment;
[0034] A prediction module, for sequentially inputting the frame number sequence and the pitch ID sequence of each of the music score segments into the appoggiatura prediction model to obtain the appoggiatura position information and the appoggiatura pitch information of the digital music score;
[0035] An adding module, configured to add corresponding grace note information in the digital music score according to the grace note position information and grace note pitch information;
[0036] The synthesis module is used to input the digital music score with the appoggiatura information added into the singing synthesis model for singing synthesis.
[0037] In a third aspect, an embodiment of the present application also provides a computer device, comprising a machine-readable storage medium and one or more processors, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the one or more processors, the above-mentioned singing synthesis method is implemented.
[0038] Based on the above contents of the embodiments of the present application, compared with the prior art, the singing voice synthesis method, system and computer device provided by the embodiments of the present application predict the score appoggiatura information by using the scale and duration on the original digital music score. The predicted appoggiatura pitch can be integrated into the original music score information and directly sent to the singing voice synthesis model as input, and no additional training is required for the singing voice synthesis model, and it can adapt to various singing voice synthesis models. At the same time, based on the predicted appoggiatura information, the singing voice synthesis model can synthesize a singing voice that is more expressive and can convey richer emotions, so that the synthesized singing voice appears to be more personal, rather than completely in line with the rigid changes of the music score. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 It is a flow chart of a singing synthesis method provided in an embodiment of the present application.
[0041] Figure 2 yes Figure 1 Schematic diagram of the implementation process of step S100.
[0042] Figure 3 It is a structural schematic diagram of an appoggiatura prediction model provided in an embodiment of the present application.
[0043] Figure 4 yes Figure 1 Schematic diagram of the implementation process of step S200.
[0044] Figure 5 yes Figure 1 Schematic diagram of the implementation process of step S300.
[0045] Figure 6 It is a schematic diagram of an existing singing synthesis method.
[0046] Figure 7 It is a schematic diagram of the singing synthesis method corresponding to the embodiment of the present application.
[0047] Figure 8 It is a schematic diagram of the functional modules of the singing synthesis system provided in the embodiment of the present application.
[0048] Fig. 9 It is a schematic diagram of a computer device for implementing the above-mentioned singing synthesis method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0050] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0052] In the description of the present application, the terms “first”, “second”, “third”, etc. are only used to distinguish the descriptions and should not be understood as indicating or implying relative importance.
[0053] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0054] Based on the technical problems mentioned in the aforementioned background technology, the inventors of the present application discovered that a synthesized singing voice containing appoggiatura would be more expressive, could convey richer emotions, make the singing voice more pleasant to the ear, and could allow the synthesized singing voice to have pitch changes that are different from the original score while conforming to the overall rhythm of the original score, making the synthesized singing voice appear more personal rather than completely conforming to the rigid changes of the score.
[0055] In addition, some existing singing synthesis models have made improvements to the vibrato in the ornaments. For example, Park et al. controlled the pitch curve through parameters such as vibrato degree and vibrato rate. When Bonada et al. used the mixed parameter statistical method and neural network method to predict the pitch curve, they also used the Gaussian function to simulate the vibrato phenomenon. However, there are no related applications and technologies for appoggiatura in singing synthesis.
[0056] In the research and development process of this application, people with a certain music theory background were organized to listen to some selected songs, and they were asked to make music scores according to the actual singing effect of the songs, including pitch, lyrics text, lyrics pinyin and whether there were appoggiatura. Then, all music scores except Mandarin were deleted. For music scores containing polyphonic or multi-staff, they were modified to monophonic and single stave. After obtaining clean data, songs were synthesized with music scores with appoggiatura labels and music scores without appoggiatura labels, and ABX tests were performed on vocal performance. The final test result was that the vast majority of people believed that singing with appoggiatura was more expressive. When simply showing music scores with or without appoggiatura, almost all testers believed that the singing effect of music scores with appoggiatura would be better. Because the song synthesis model used was not optimized for the situation of appoggiatura, some synthesized songs could not show the effect of appoggiatura well. Nevertheless, ABX tests showed that appoggiatura can effectively improve the expressiveness of synthesized singing.
[0057] Based on the above background, the present application proposes a singing voice synthesis method, system and computer device based on appoggiatura prediction to provide a subsequent singing voice synthesis model with a music score containing appoggiatura, so as to facilitate the synthesis of highly expressive singing voice containing appoggiatura. The relevant schemes of the present application will be exemplarily described below in conjunction with the accompanying drawings.
[0058] For ease of understanding, before introducing the technical solution of the present application, some relevant technical terms involved in the present application are first explained.
[0059] Singing synthesis refers to the conversion of musical score information into standard and fluent singing.
[0060] Grace note: Grace notes can be divided into long grace notes and short grace notes. Long grace notes mainly appear in classical works and are extremely rare in current music notation.
[0061] Acciaccatura is the general concept of appoggiatura in our daily conversation. It is not the main note in melody or harmony. Its duration is included in the tonic it relies on, and it is usually one semitone or a whole tone lower than the tonic.
[0062] BiLSTM (Bi-directional Long Short-Term Memory) is composed of forward LSTM and backward LSTM.
[0063] CRF (Conditional Random Fields) is a conditional probability distribution model that obtains another set of output sequences under a given set of input sequences.
[0064] Embedding: Vector mapping, a way to convert discrete variables into continuous vector representation.
[0065] The implementation scheme of the embodiment of the present application will be exemplarily introduced below with reference to the accompanying drawings.
[0066] like Figure 1 , which is a flow chart of a singing voice synthesis method provided in an embodiment of the present application. In this embodiment, the singing voice synthesis method can be executed and implemented by a computer device. It should be understood that the order of some of the steps included in the singing voice synthesis method provided in this embodiment can be interchanged according to actual needs during actual implementation, or some of the steps can be omitted or deleted, and this embodiment does not specifically limit this.
[0067] Combine the following Figure 1 Each step of the singing voice synthesis method of this embodiment is described in detail by way of example. Figure 1 As shown, the method may include the relevant contents described in the following steps S100 to S400.
[0068] Step S100, segmenting the digital music score to obtain a plurality of music score segments, and obtaining a frame number sequence and a pitch ID sequence of each music score segment.
[0069] In this embodiment, the digital music score may be the original music score provided by the user, including lyrics, scale, duration and other information. The digital music score may be a digital music score in the form of musicxml corresponding to a simplified musical notation or a five-line musical notation containing scale and duration information. For each music score segment, each scale in the music score segment may be converted into a corresponding pitch ID to obtain a pitch ID sequence. For example, each scale of the music score segment may be converted into a corresponding pitch ID by looking up a table, thereby obtaining a pitch ID sequence corresponding to the music score segment. For example, the pitch ID corresponding to the scale may be C0-12, C#0-13, etc. by looking up a table.
[0070] In addition, the floating-point duration of each scale in each score segment can be converted into a frame number to obtain a frame number sequence. For example, the floating-point duration of each scale in each score segment can be converted into a frame number by the following formula, thereby obtaining a frame number sequence:
[0071]
[0072] Among them, sr is the sampling rate set for sampling the audio of the music score segment, dur is the floating point duration of the scale, win refers to the window length of each frame when the audio is framed, and hop refers to the window movement length, which indicates how long it takes to take a frame when framing.
[0073] Regarding step S100, in a possible implementation of this embodiment, as follows Figure 2 As shown, in step S100, the digital music score is segmented to obtain a plurality of music score segments, which can be achieved through the following S110-S130.
[0074] Step S110: extracting the scale information and duration information of each note from the digital music score.
[0075] Among them, this embodiment can realize the extraction of the scale information and duration information through an existing scale duration extractor, which will not be described in detail in this embodiment.
[0076] Step S120: pre-processing the digital music score according to the scale information and duration information.
[0077] In a possible implementation of this embodiment, pre-processing the digital music score includes at least one of the following two processing methods.
[0078] Method 1: merging consecutive characters of the same musical scale in the digital music score, and taking the sum of the durations of the characters before merging as the duration of the merged character.
[0079] Method 2: Combine consecutive rests and consecutive breaths, and use the sum of the durations of the rests before the combination as the duration of the combined rest, and use the sum of the durations of the breaths before the combination as the duration of the combined breath.
[0080] Step S130, segmenting the digital music score according to the rests and breaths in the pre-processed digital music score to obtain a plurality of music score segments. In this embodiment, each music score segment includes at least three notes, and the beginning and the end of each music score segment include a breath duration and a rest duration respectively.
[0081] The digital music score is pre-processed by the above two methods so that after the duration and notes of the entire digital music score are merged, the entire digital music score can be segmented according to the rests and breaths in the music score. During the segmentation process, it is necessary to ensure that each segment has at least 3 notes, and the duration of the rests and breaths is copied and placed at the end of the previous music score segment and the beginning of the next music score segment respectively.
[0082] Step S200, inputting the frame number sequence and pitch ID sequence of each of the music score segments into the appoggiatura prediction model in sequence, and obtaining the appoggiatura position information and appoggiatura pitch information of the digital music score.
[0083] In this embodiment, the appoggiatura prediction model can be obtained by training a set AI model (such as a BiLSTM-CRF model) with a pre-collected music score data set.
[0084] As described in the aforementioned technical background, after clarifying the data validity and the necessity of appoggiatura prediction, data analysis can be performed on the data set to find the input features of model training. The data directly extracted from the digital music score can have Chinese characters, pinyin, pitch, duration, whether it is appoggiatura, these 5 kinds of information. In order to determine whether the appoggiatura position is semantically relevant, the part of speech and rhythmic label of each word can also be extracted. For the convenience of data analysis, the pinyin is split into initial consonants, finals, and tones, and the floating point number of duration is converted into the number of frames, and they are converted into id together with the pitch. After normalization, they are analyzed by correlation coefficients with whether there is appoggiatura, and finally it is verified that the pitch and duration in the above-mentioned features are the two features with the highest correlation with the appoggiatura position, so the two features are mainly used in the embodiment of the present application to predict the appoggiatura.
[0085] In addition to the correlation analysis of the position of the appoggiatura, we also analyzed the pitch of the appoggiatura. In the common sense of musicians, the pitch of the appoggiatura is generally one whole tone or one semitone lower than the pitch of the tonic. In the database, the pitch of the tonic minus the pitch of the appoggiatura accounts for about 74% of 2, indicating a difference of one whole tone, and 20% of 1, indicating a difference of one semitone. These two situations cover 94% of the appoggiaturas.
[0086] From the above analysis, it can be seen that the prediction of the appoggiatura position is mainly based on the duration information, while the prediction of the appoggiatura pitch is mainly based on the pitch information. Therefore, this application decides to split the appoggiatura prediction into two tasks, one is to predict the appoggiatura position, and the other is to predict the appoggiatura pitch.
[0087] Based on the above content, in this embodiment, the music score data set may include a large number of labeled digital music score fragment samples. The label of each digital music score fragment sample includes a flag sequence indicating whether each note position contains an appoggiatura and a pitch difference sequence between the appoggiatura and the main tone when each note position is assumed to have an appoggiatura, wherein if there is no appoggiatura at a certain position, the corresponding pitch difference is 0, and if there is an appoggiatura, the pitch difference is 1 or 2, 1 indicating that the appoggiatura at the corresponding position is one semitone lower than the next main tone, and 2 indicating that the appoggiatura at the corresponding position is one full tone lower than the next main tone. During model training, each digital music score fragment sample can be sequentially input into the AI model, and the loss function can be calculated based on the results output by the model and the labels of the digital music score fragment samples. The parameters of the AI model can be adjusted by the loss function until the training converges, and the trained AI model is obtained as the appoggiatura prediction model.
[0088] In the embodiments of the present application, for example Figure 3 As shown, the appoggiatura prediction model may include an embedding layer, a bidirectional long short-term memory layer (BiLSTM layer), a full connecting layer (Full connecting layer), and a conditional random field layer (CRF layer). Figure 4 As shown, in step S200, the frame number sequence and pitch ID sequence of each of the music score segments are sequentially input into the appoggiatura prediction model to obtain the appoggiatura position information and appoggiatura pitch information of the digital music score, which may include the following steps S210-S230.
[0089] Step S210, input the frame number sequence and the pitch ID sequence of the score segment into the embedding layer, perform vector mapping on the frame number sequence and the pitch ID sequence, obtain the duration vector and the pitch vector of each note in the score segment, and add the duration vector and the pitch vector of each note to obtain the input feature vector corresponding to each note. For example, the embedding layer can convert each pitch / scale (pitch) in the input score segment into a corresponding pitch ID by looking up a table, thereby obtaining the pitch ID sequence, and convert the corresponding floating point duration into a frame number by vector mapping, thereby obtaining the frame number sequence.
[0090] Step S220, input the input feature vectors corresponding to each note into the bidirectional long short-term memory layer for processing, and send the output of the bidirectional long short-term memory layer into the fully connected layer to obtain the scores of each input feature vector for multiple prediction labels. Wherein, the prediction label is pre-set, including a grace note prediction label and a pitch prediction label. The grace note prediction label may include four labels of "beginning of sentence", "end of sentence", "grace note", and "non-grace note", and the probability value of the input feature vector predicted as each grace note prediction label is used as the score of the input feature vector for each grace note prediction label. The pitch prediction label may include five pitch prediction labels such as "beginning", "end", "pitch difference is 0", "pitch difference is 1", and "pitch difference is 2", and the probability value of the input feature vector predicted as each pitch prediction label is used as the score of the input feature vector for each pitch prediction label. Among them, a pitch difference of 0 indicates that the corresponding note position does not contain an appoggiatura, a pitch difference of 1 indicates that the appoggiatura at the corresponding note position is one semitone lower than the pitch of the subsequent main note, and a pitch difference of 2 indicates that the appoggiatura at the corresponding note position is one whole tone lower than the pitch of the subsequent main note.
[0091] Step S230, input the score of each input feature vector for each predicted label into the conditional random field layer, and the conditional random field layer obtains the appoggiatura position sequence and the appoggiatura pitch difference sequence corresponding to the music score fragment according to the score of each input feature vector for each predicted label, the appoggiatura position sequence includes an identifier representing whether the corresponding position contains an appoggiatura (such as 0 or 1, 0 indicates no appoggiatura, 1 indicates the presence of appoggiatura), and the appoggiatura pitch difference sequence includes the appoggiatura pitch difference of each position in the music score fragment.
[0092] Step S300: adding corresponding appoggiatura information to the digital music score according to the appoggiatura position information and the appoggiatura pitch information.
[0093] In a possible implementation of this embodiment, for example Figure 5 As shown, the step S300 can be implemented through the following steps S310-S340, which are exemplarily introduced below.
[0094] Step S310: adding an appoggiatura scale to the notes corresponding to each predicted appoggiatura position of each of the music score segments according to the appoggiatura position sequence.
[0095] Step S320, according to the duration of a main note after the appoggiatura position, a corresponding duration is set for the note at the appoggiatura position, and the duration of the main note is adjusted to the difference between the original duration of the main note and the duration of the note at the appoggiatura position. For example, the duration of the note at the appoggiatura position can be set to 1 / 8 of the following main note, and the duration of the following main note is set to the difference between the original duration and the duration of the newly added appoggiatura (such as 7 / 8).
[0096] Step S330, moving the lyrics of the main note to the previous appoggiatura position, and changing the lyrics of the main note to a fermata symbol, so as to obtain a music score segment after the appoggiatura processing.
[0097] Step S340, removing the rests and breath notes at the beginning of each music score segment after the appoggiatura processing and then splicing them together to obtain a digital music score with the appoggiatura information added.
[0098] Step S400, inputting the digital music score with the added appoggiatura information into the singing voice synthesis model for singing voice synthesis.
[0099] Among them, in the embodiment of the present application, the singing synthesis model can be any existing singing synthesis model. According to the singing synthesis method of this embodiment, the singing synthesized by the singing synthesis model contains appoggiatura information and has better expressiveness.
[0100] In conjunction with the above, please refer to Figure 6 and Figure 7 As shown, Figure 6 is a schematic diagram of an existing singing synthesis method, Figure 7 Schematic diagram of the singing voice synthesis method corresponding to the embodiment of the present application. Compared to it can be seen that in the existing method, generally directly synthesizing the a cappella singing voice through the equipment such as computer or server according to the music score information input by the user, and then the singing audio singing voice with singing effect can be obtained by increasing the accompaniment information. In the present application, the music score information input by the user will first be processed by pitch duration extraction, appoggiatura position prediction, appoggiatura pitch prediction, etc., to obtain the pitch and duration information of each music score fragment (which can be a sentence) after segmentation, and then the corresponding appoggiatura information will be added in each music score fragment, and finally the music score fragments after the appoggiatura information are added will be spliced into a complete music score, for use as the input of the subsequent singing voice synthesis model. In this way, by using the scale and duration on the original digital music score to predict the music score appoggiatura information, the predicted appoggiatura pitch can be integrated into the original music score information, directly sent to the singing voice synthesis model as input, and there is no need to carry out additional training to the singing voice synthesis model, and various singing voice synthesis models can be adapted. At the same time, based on the predicted appoggiatura information, the singing synthesis model can synthesize singing that is more expressive and can convey richer emotions, making the synthesized singing appear more personal rather than completely conforming to the rigid changes of the music score.
[0101] like Figure 8 FIG. 1 is a schematic diagram of a singing voice synthesis system for implementing the singing voice synthesis method provided in an embodiment of the present application. In this embodiment, the singing voice synthesis system can be applied to Fig. 9The computer device 100 shown. In detail, the computer device 100 may include one or more processors 110, a machine-readable storage medium 120 and a singing synthesis system 130. The processor 110 and the machine-readable storage medium 120 may be connected to each other via a system bus. The machine-readable storage medium 120 stores machine-executable instructions, and the processor 110 implements the singing synthesis method described above by reading and executing the machine-executable instructions in the machine-readable storage medium 120. In this embodiment, the computer device 100 may be a cloud server for executing the various functional modules included in the front end of the singing synthesis system, or a user client for executing the various functional modules included in the back end of the singing synthesis system, or a cloud server for executing the various functional modules included in the front end and back end of the singing synthesis system at the same time, or a combination of a cloud server and a user client for respectively executing the various functional modules included in the front end and back end of the singing synthesis system, and this embodiment does not limit this.
[0102] The machine-readable storage medium 120 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The machine-readable storage medium 120 is used to store a program, and the processor 110 executes the program after receiving an execution instruction.
[0103] The processor 110 may be an integrated circuit chip with signal processing capability. The processor may be, but is not limited to, a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.
[0104] In this embodiment, the singing synthesis system 130 may include a segmentation module 131 , a prediction module 132 , an addition module 133 , and a synthesis module 134 .
[0105] The segmentation module 131 is used to segment the digital music score to obtain a plurality of music score segments, and obtain a frame number sequence and a pitch ID sequence of each music score segment.
[0106] In this embodiment, the segmentation module 131 may be used to execute the above step S100. For more detailed contents of the segmentation module 131, reference may be made to the relevant contents of the above step S100, which will not be described in detail here.
[0107] The prediction module 132 is used to sequentially input the frame number sequence and pitch ID sequence of each of the music score segments into the appoggiatura prediction model to obtain the appoggiatura position information and appoggiatura pitch information of the digital music score.
[0108] In this embodiment, the prediction module 132 may be used to execute the above step S200. For more detailed contents of the prediction module 132, reference may be made to the relevant contents of the above step S200, which will not be described again here.
[0109] The adding module 133 is used to add corresponding grace note information in the digital music score according to the grace note position information and grace note pitch information.
[0110] In this embodiment, the adding module 133 can be used to execute the above step S300. For more detailed content of the adding module 133, refer to the relevant content of the above step S300, which will not be repeated here.
[0111] The synthesis module 134 is used to input the digital music score with the added appoggiatura information into the singing synthesis model to perform singing synthesis.
[0112] In this embodiment, the synthesis module 134 can be used to execute the above step S400. For more detailed content of the synthesis module 134, please refer to the relevant content of the above step S400, which will not be repeated here.
[0113] In summary, the singing voice synthesis method, system and computer device provided by the embodiment of the present application predict the music score appoggiatura information by using the scale and duration on the original digital music score. The predicted appoggiatura pitch can be integrated into the original music score information and directly sent to the singing voice synthesis model as input, and no additional training is required for the singing voice synthesis model, and it can adapt to various singing voice synthesis models. At the same time, based on the predicted appoggiatura information, the singing voice synthesis model can synthesize a singing voice that is more expressive and can convey richer emotions, so that the synthesized singing voice appears to be more personal, rather than completely in line with the rigid changes of the music score.
[0114] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0115] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0116] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0117] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0118] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A singing voice synthesis method, characterized in that: The method comprises: Segment the digital music score to obtain multiple music score segments, and obtain a frame number sequence and a pitch ID sequence of each music score segment; Inputting the frame number sequence and the pitch ID sequence of each of the music score segments into the appoggiatura prediction model in sequence to obtain the appoggiatura position information and the appoggiatura pitch information of the digital music score; Adding corresponding grace note information to the digital music score according to the grace note position information and grace note pitch information; Inputting the digital music score with the added appoggiatura information into the singing voice synthesis model to perform singing voice synthesis; The appoggiatura prediction model includes an embedding layer, a bidirectional long short-term memory layer, a fully connected layer, and a conditional random field layer. The frame number sequence and the pitch ID sequence of each score segment are sequentially input into the appoggiatura prediction model to obtain the appoggiatura position information and the appoggiatura pitch information of the digital music score, including: Inputting the frame number sequence and the pitch ID sequence of the music score segment into the embedding layer, performing vector mapping on the frame number sequence and the pitch ID sequence to obtain the duration vector and the pitch vector of each note of the music score segment, and adding the duration vector and the pitch vector of each note to obtain the input feature vector corresponding to each note; Inputting the input feature vectors corresponding to the notes into the bidirectional long short-term memory layer for processing, and sending the output of the bidirectional long short-term memory layer into the fully connected layer to obtain the scores of each input feature vector for multiple predicted labels; The scores of each input feature vector for each predicted label are input into the conditional random field layer, and the conditional random field layer obtains the appoggiatura position sequence and the appoggiatura pitch difference sequence corresponding to the music score segment according to the scores of each input feature vector for each predicted label; the appoggiatura position sequence includes an identifier representing whether the corresponding position contains the appoggiatura, and the appoggiatura pitch difference sequence includes the appoggiatura pitch difference of each position in the music score segment.
2. The singing voice synthesis method according to claim 1, characterized in that: The digital music score is segmented to obtain multiple music score fragments, including: Extracting the scale information and duration information of each note from the digital music score; Pre-processing the digital music score according to the scale information and duration information; The digital music score is segmented according to the rests and breaths in the pre-processed digital music score to obtain a plurality of music score segments, wherein each music score segment includes at least three notes, and the beginning and the end of each music score segment respectively include a breath note duration and a rest duration.
3. The singing voice synthesis method according to claim 2, characterized in that: Pre-processing the digital music score according to the scale information and the duration information includes: Merging consecutive characters of the same word with the same musical scale in the digital music score, and taking the sum of the durations of the characters before merging as the duration of the merged character; and / or Consecutive rests and consecutive breath notes are merged, and the sum of the durations of the rest notes before the merger is used as the duration of the rest note after the merger, and the sum of the durations of the breath notes before the merger is used as the duration of the breath note after the merger.
4. The singing voice synthesis method according to claim 1, characterized in that: Get the frame number sequence and pitch ID sequence of each music score segment and add the corresponding appoggiatura information, including: For each music score segment, convert each scale in the music score segment into a corresponding pitch ID to obtain a pitch ID sequence; The floating-point duration of each scale in the music score segment is converted into a frame number to obtain a frame number sequence.
5. The singing voice synthesis method according to any one of claims 1 to 4, characterized in that: The prediction labels include appoggiatura prediction labels and pitch prediction labels, and the appoggiatura prediction labels include four labels: "beginning of sentence", "end of sentence", "appoggiatura", and "non-appoggiatura", and the input feature vector is predicted to be the probability value of each of the appoggiatura prediction labels as the score of the input feature vector for each of the appoggiatura prediction labels; the pitch prediction labels include five pitch prediction labels: "beginning", "end", "pitch difference is 0", "pitch difference is 1", and "pitch difference is 2", and the input feature vector is predicted to be the probability value of each of the pitch prediction labels as the score of the input feature vector for each of the pitch prediction labels; a pitch difference of 0 indicates that the corresponding note position does not contain appoggiatura, a pitch difference of 1 indicates that the appoggiatura at the corresponding note position is one semitone lower than the pitch of the subsequent main tone, and a pitch difference of 2 indicates that the appoggiatura at the corresponding note position is one whole tone lower than the pitch of the subsequent main tone.
6. The singing voice synthesis method according to claim 5, characterized in that: Adding corresponding grace note information to the digital music score according to the grace note position information and the grace note pitch information includes: According to the sequence of appoggiatura positions, adding an appoggiatura scale to the notes corresponding to each predicted appoggiatura position in each of the music score segments; Setting a corresponding duration for the note at the appoggiatura position according to the duration of a main tone after the appoggiatura position, and adjusting the duration of the main tone to the difference between the original duration of the main tone and the duration of the note at the appoggiatura position; The lyrics of the main note are moved to the position of the previous appoggiatura, and the lyrics of the main note are changed to a fermata symbol to obtain a music score segment after the appoggiatura processing; The rests and breath notes at the beginning of each music score fragment after the appoggiatura processing are removed and then spliced to obtain a digital music score with appoggiatura information added.
7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a music score data set, the music score data set comprising a plurality of digital music score segment samples with labels, the label of each digital music score segment sample comprising a flag sequence of whether each note position has an appoggiatura and a pitch difference sequence of the appoggiatura and the tonic when each note position has an appoggiatura, wherein if there is no appoggiatura at a certain position, the corresponding pitch difference is 0, and if there is an appoggiatura, the pitch difference is 1 or 2, 1 indicating that the appoggiatura at the corresponding position is one semitone lower than the subsequent tonic, and 2 indicating that the appoggiatura at the corresponding position is one whole tone lower than the subsequent tonic; Each digital music score fragment sample is input into a pre-selected AI model in sequence, a loss function is calculated according to the result output by the AI model and the label of the digital music score fragment sample, and the parameters of the AI model are adjusted by the loss function until the training converges, thereby obtaining the trained AI model as the appoggiatura prediction model.
8. A singing voice synthesis system, applied to a computer device, characterized in that: The singing voice synthesis system comprises: A segmentation module is used to segment the digital music score to obtain multiple music score segments, and obtain a frame number sequence and a pitch ID sequence of each music score segment; A prediction module, for sequentially inputting the frame number sequence and the pitch ID sequence of each of the music score segments into the appoggiatura prediction model to obtain the appoggiatura position information and the appoggiatura pitch information of the digital music score; An adding module, configured to add corresponding grace note information in the digital music score according to the grace note position information and grace note pitch information; A synthesis module, used for inputting the digital music score with added appoggiatura information into the singing synthesis model to perform singing synthesis; The appoggiatura prediction model includes an embedding layer, a bidirectional long short-term memory layer, a fully connected layer, and a conditional random field layer. The prediction module is specifically used for: Inputting the frame number sequence and the pitch ID sequence of the music score segment into the embedding layer, performing vector mapping on the frame number sequence and the pitch ID sequence to obtain the duration vector and the pitch vector of each note of the music score segment, and adding the duration vector and the pitch vector of each note to obtain the input feature vector corresponding to each note; Inputting the input feature vectors corresponding to the notes into the bidirectional long short-term memory layer for processing, and sending the output of the bidirectional long short-term memory layer into the fully connected layer to obtain the scores of each input feature vector for multiple predicted labels; The scores of each input feature vector for each predicted label are input into the conditional random field layer, and the conditional random field layer obtains the appoggiatura position sequence and the appoggiatura pitch difference sequence corresponding to the music score segment according to the scores of each input feature vector for each predicted label; the appoggiatura position sequence includes an identifier representing whether the corresponding position contains the appoggiatura, and the appoggiatura pitch difference sequence includes the appoggiatura pitch difference of each position in the music score segment.
9. A computer device, characterized in that: It comprises a machine-readable storage medium and one or more processors, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the one or more processors, the singing synthesis method according to any one of claims 1 to 7 is implemented.