Song synthesis method and device
By creating a note display box in the music score display interface and receiving lyrics, the simple and easy operation of song synthesis is achieved, and the complex problem of song synthesis operations in the existing technology is solved, which improves the user experience.
Patent Information
- Application Number
- CN202011255647.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-11-11
AI Technical Summary
The existing song synthesis software is complex in operation and difficult to use, making it difficult for ordinary users to synthesize songs.
Provide a song synthesis method. By receiving the score of the song to be synthesized, creating a note display box, and displaying the note display box in the score display interface. The user can enter lyrics into the note display box to synthesize the song.
It lowers the threshold for song synthesis, allowing any user to easily synthesize songs without mastering complex software usage techniques, improving user experience.
Smart Images

Figure CN114550690B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of audio processing technology, and in particular to a song synthesis method and device. Background Art
[0002] With the rapid development of audio processing technology, song synthesis technology has also developed, and many song synthesis software have been born. In the prior art, if a user needs to synthesize a song, he needs to first download and install the corresponding song synthesis software, and then learn the software usage method and related production skills of the song synthesis software. It can be seen that the existing song synthesis process is complicated and difficult to operate, and thus a simpler and more convenient method is needed to operate or process song synthesis. Summary of the invention
[0003] In view of this, the embodiment of this specification provides a song synthesis method. This specification also relates to a song synthesis device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.
[0004] According to a first aspect of an embodiment of this specification, a song synthesis method is provided, the method comprising:
[0005] Receiving a song to be synthesized, and determining a music score of the song to be synthesized;
[0006] Creating a note display frame for the notes included in the music score, and displaying the note display frame on a music score display interface;
[0007] Receiving lyrics corresponding to the musical note through the musical note display frame;
[0008] The song to be synthesized is synthesized according to the music score and the lyrics corresponding to each note included in the music score.
[0009] Optionally, creating a note display frame for the notes included in the music score includes:
[0010] For any note included in the music score, determining a duration feature of the note;
[0011] Determining the length of the note display frame corresponding to the note according to the duration feature;
[0012] Create a note display frame corresponding to the note according to the length.
[0013] Optionally, the music score display interface includes a tone, and displaying the note display frame on the music score display interface includes:
[0014] Determining the pitch of the note corresponding to the note display frame;
[0015] Determining the starting position of the note in the music score display interface according to the starting time of the note in the music score;
[0016] Determining a display position of a note display frame corresponding to the note in the music score display interface according to the pitch of the note and the starting position of the note in the music score display interface;
[0017] The note display frame is displayed at the display position in the music score display interface.
[0018] Optionally, synthesizing the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score includes:
[0019] Synthesizing a voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score;
[0020] The voice file is synthesized with the music score to obtain the song to be synthesized.
[0021] Optionally, before synthesizing the voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score, the method further includes:
[0022] An imported lyrics audio library is received, where the lyrics audio library is an audio collection including at least one monosyllabic audio.
[0023] Optionally, synthesizing the voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score includes:
[0024] Acquire the monosyllabic audio of the lyrics corresponding to each note included in the music score from the lyrics audio library;
[0025] adjusting the monosyllabic audio to the target audio of the note in the musical score;
[0026] The target audio is synthesized to obtain a voice file of the song to be synthesized.
[0027] Optionally, adjusting the monosyllabic audio to a target audio of the note in the music score includes:
[0028] Determining the original duration and original pitch of the monosyllabic audio;
[0029] Determining a target duration and a target pitch of the note in the musical score;
[0030] The original pitch is adjusted to the target pitch, and the original duration is adjusted to the target duration, so as to obtain the target audio of the note in the music score.
[0031] Optionally, determining the original pitch of the monosyllabic audio includes:
[0032] In the case where the monosyllabic audio carries a pitch, the pitch carried in the monosyllabic audio is read, and the pitch is determined as the original pitch of the monosyllabic audio.
[0033] Optionally, determining the original pitch of the monosyllabic audio includes:
[0034] In the case where the monosyllabic audio does not carry a pitch, determining an average frequency of the monosyllabic audio;
[0035] The original pitch of the monosyllabic audio is determined according to the average frequency.
[0036] Optionally, adjusting the original duration to the target duration includes:
[0037] When the original duration is equal to the target duration, the original duration is determined as the target duration;
[0038] When the original duration and the target duration are not equal, determine whether the original duration exceeds the target duration. If so, cut the original duration to the target duration; if not, stretch the original duration to the target duration.
[0039] Optionally, stretching the original duration to the target duration includes:
[0040] In a case where the monosyllabic audio carries a phoneme segmentation identifier, reading the phoneme segmentation identifier carried in the monosyllabic audio, where the phoneme segmentation identifier is used to segment the phonemes included in the monosyllabic audio;
[0041] Determine the vowels and consonants in the monosyllabic audio according to the phoneme segmentation identifier;
[0042] The original duration of the consonant is maintained, and the original duration of the vowel is stretched until the target duration is reached.
[0043] Optionally, stretching the original duration to the target duration includes:
[0044] In the case that the monosyllabic audio does not carry a phoneme segmentation identifier, each phoneme included in the monosyllabic audio is stretched until the target duration is reached.
[0045] Optionally, the receiving of the song to be synthesized includes:
[0046] Receive imported songs to be synthesized; or,
[0047] A selection instruction for a target song in a preset song library is received, and the target song is determined as the song to be synthesized.
[0048] Optionally, after displaying the note display frame at the display position in the music score display interface, the method further includes:
[0049] Receive movement instructions for a target note display frame;
[0050] According to the movement instruction, the target note display frame is moved from the current display position to the target display position, and the target display position is the tone position indicated by the movement instruction.
[0051] Optionally, synthesizing the voice file with the music score to obtain the song to be synthesized includes:
[0052] Calculate the duration of a measure of the music score according to the number of beats per minute of the music score;
[0053] According to the measure duration, the voice file is rhythmically aligned with the music score, and the song to be synthesized is synthesized.
[0054] Optionally, after synthesizing the voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score, the method further includes:
[0055] When a pre-listening instruction for the voice file is received, the synthesized voice file is played.
[0056] Optionally, synthesizing the voice file with the music score to obtain the song to be synthesized includes:
[0057] When receiving a music score adding instruction, adding the music score to the voice file;
[0058] receiving an offset set for the score;
[0059] offsetting the music score according to the offset;
[0060] The shifted music score and the voice file are synthesized to obtain the song to be synthesized.
[0061] Optionally, after synthesizing the shifted music score and the voice file to obtain the song to be synthesized, the method further includes:
[0062] In case of receiving an export instruction for the song to be synthesized, determining a target file format indicated by the export instruction for export;
[0063] The synthesized song to be synthesized is exported according to the target file format.
[0064] Optionally, the step of acquiring, from the lyrics audio library, a monosyllabic audio corresponding to each note included in the music score comprises:
[0065] Determine whether the lyrics received through the musical note display frame are pinyin;
[0066] If yes, then obtain the monosyllabic audio corresponding to the pinyin from the lyrics audio library;
[0067] If not, the lyrics are converted into pinyin, and then the monosyllabic audio corresponding to the pinyin is obtained from the lyrics audio library.
[0068] Optionally, creating a note display frame for the notes included in the music score includes:
[0069] For any note included in the music score, determining the duration characteristics and default lyrics of the note;
[0070] Determining the length of the note display frame corresponding to the note according to the duration feature;
[0071] According to the length, a note display frame corresponding to the note is created, and the default lyrics are displayed in the note display frame.
[0072] Optionally, receiving lyrics corresponding to the note through the note display frame includes:
[0073] For any of the musical note display frames, determining whether an input instruction is received;
[0074] If yes, receiving the input updated lyrics through the note display box, and determining the updated lyrics as the lyrics corresponding to the note;
[0075] If not, the default lyrics of the note display frame are determined as the lyrics corresponding to the note.
[0076] Optionally, synthesizing the target audio to obtain a voice file of the song to be synthesized includes:
[0077] If two adjacent notes overlap, the target audio corresponding to the previous note is faded out, and the target audio corresponding to the next note is faded in.
[0078] According to a second aspect of the embodiments of this specification, a song synthesis device is provided, the device comprising:
[0079] A determination module is configured to receive a song to be synthesized and determine a music score of the song to be synthesized;
[0080] A display module, configured to create a note display frame for the notes included in the music score, and display the note display frame on the music score display interface;
[0081] A receiving module is configured to receive lyrics corresponding to the musical note through the musical note display frame;
[0082] The synthesis module is configured to synthesize the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score.
[0083] According to a third aspect of an embodiment of this specification, a computing device is provided, including:
[0084] Memory and processor;
[0085] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the following method:
[0086] Receiving a song to be synthesized, and determining a music score of the song to be synthesized;
[0087] Creating a note display frame for the notes included in the music score, and displaying the note display frame on a music score display interface;
[0088] Receiving lyrics corresponding to the musical note through the musical note display frame;
[0089] The song to be synthesized is synthesized according to the music score and the lyrics corresponding to each note included in the music score.
[0090] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the song synthesis method are implemented.
[0091] The song synthesis method provided in this specification can first receive the song to be synthesized, determine the score of the song to be synthesized, then create a note display box for the notes included in the score, and display the note display box on the score display interface; then, the lyrics corresponding to the notes can be received through the note display box, and finally, the song to be synthesized can be synthesized according to the score and the lyrics corresponding to each note included in the score. In this case, a score display interface will be displayed, and the note display box will be displayed on the score display interface. The user only needs to enter the lyrics in the note display box to synthesize the song he wants to synthesize. The song synthesis process is simple and easy to operate, so that any user who does not have any song synthesis software usage skills can easily synthesize songs, which lowers the threshold for song synthesis, facilitates user use, and improves user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0092] Figure 1is a flow chart of a song synthesis method provided in an embodiment of this specification;
[0093] Figure 2 is a schematic diagram of a music score display interface provided in an embodiment of this specification;
[0094] Figure 3 is a schematic diagram of another music score display interface provided in an embodiment of this specification;
[0095] Figure 4 This is a schematic diagram of an interface for importing lyrics audio library provided in an embodiment of this specification;
[0096] Figure 5 is a schematic diagram of stretching a monosyllabic audio provided by an embodiment of this specification;
[0097] Figure 6 It is a processing flow chart of a song synthesis method applied to a browser provided in an embodiment of this specification;
[0098] Figure 7 This is a processing flow chart of a song synthesis method applied to a ghost animal scene provided by an embodiment of this specification;
[0099] Figure 8 It is a structural schematic diagram of a song synthesis device provided in an embodiment of this specification;
[0100] Fig. 9 It is a structural block diagram of a computing device provided in an embodiment of this specification. DETAILED DESCRIPTION
[0101] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0102] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0103] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0104] First, the terms involved in one or more embodiments of this specification are explained.
[0105] Pitch: Pitch refers to various sounds of different pitches, that is, the height of the sound, one of the basic characteristics of the sound. The pitch of the sound is determined by the vibration frequency, and the two are positively correlated. The higher the frequency (that is, the number of vibrations per unit time), the higher the sound, and vice versa.
[0106] Phoneme: The smallest speech unit divided according to the natural properties of speech. From the perspective of acoustic properties, phonemes are the smallest speech units divided from the perspective of sound quality. From the perspective of physiological properties, one pronunciation action forms a phoneme. Phonemes are generally divided into two categories: vowels and consonants.
[0107] MIDI (Musical Instrument Digital Interface): It was proposed in the early 1980s to solve the communication problem between electro-acoustic instruments. MIDI is the most widely used music standard format in the music arrangement industry and can be called "music scores that computers can understand". It uses digital control signals of notes to record music. A complete MIDI music is only tens of KB in size and can contain dozens of music tracks. Almost all modern music is synthesized using MIDI plus a sound library. MIDI does not transmit sound signals, but instructions such as notes and control parameters. It instructs MIDI devices what to do and how to do it, such as which notes to play and how loud they should be. They are uniformly represented as MIDI messages.
[0108] Web Audio API: Provides a very effective and general system for controlling audio on the Web, allowing developers to decode, process and output audio, such as selecting audio sources, adding special effects to audio, visualizing audio, adding spatial effects (such as panning), and so on.
[0109] Audio Buffer interface: represents a short audio resource in memory.
[0110] Web Assembly: is a new type of code that runs in modern web browsers and provides new performance features and effects.
[0111] WAV: One of the most common sound file formats, it is a specially developed standard digital audio file that can record various mono or stereo sound information and ensure that the sound is not distorted.
[0112] AAC (Advanced Audio Coding): A file compression format designed specifically for sound data, using a new algorithm for encoding, which is more efficient and has a higher "cost performance". Using the AAC format, people can feel that the sound quality is not significantly reduced, while being more compact.
[0113] In this specification, a song synthesis method is provided. This specification also relates to a song synthesis device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0114] Figure 1 A flowchart of a song synthesis method provided according to an embodiment of this specification is shown, which specifically includes the following steps:
[0115] Step 102: receiving a song to be synthesized and determining a music score of the song to be synthesized.
[0116] In practical applications, people's work and life are inseparable from songs. In addition to playing existing songs, users can also synthesize lyrics and music scores according to their preferences and needs to obtain new synthesized songs. At present, song synthesis is generally performed through professional song synthesis software developed by professionals. However, song synthesis software needs to be downloaded and installed in advance, and users need to master a lot of software usage skills to synthesize songs. The song synthesis process is complicated and difficult to operate. For example, the singing synthesis tool UTAU is a well-known software in the field of human VOCALOID. It uses WAVE format audio recorded by oneself or others. After the software database is imported, the sound can be used to synthesize songs. Before using UTAU to synthesize songs, users need to download and install UTAU software, and learn and master the software's song synthesis skills through a large number of tutorials, resulting in only users with a certain audio and software foundation can use the software to synthesize songs.
[0117] In order to reduce the operational difficulty of song synthesis, thereby lowering the threshold of song synthesis and allowing ordinary users to achieve song synthesis, this manual provides a song synthesis method, which can first receive the song to be synthesized and determine the score of the song to be synthesized, then create a note display box for the notes included in the score, and display the note display box on the score display interface; then, the lyrics corresponding to the notes can be received through the note display box, and finally the song to be synthesized can be synthesized according to the score and the lyrics corresponding to each note included in the score. In this case, a score display interface will be displayed, and the note display box will be displayed on the score display interface. The user only needs to enter the lyrics in the note display box to synthesize the song he wants to synthesize. The song synthesis process is simple and easy to operate, so that any user who does not have any skills in using song synthesis software can easily synthesize songs.
[0118] It should be noted that a song generally includes music score and lyrics. The music score can be understood as accompaniment, and the lyrics can be understood as voice files, that is, the human voice obtained by recording the lyrics. The goal of vocal synthesis is a melody with lyrics. Usually, the melody is represented by music score. In the computer, a digital method is required to represent it. This manual uses a music score representation method similar to the MIDI standard as an input of the synthesis method. The music score usually consists of multiple notes, and each note records its pitch, starting position and duration, all in digital form.
[0119] Generally speaking, when synthesizing a song, the lyrics created by the user are often synthesized on the original music score, or the original music score is only fine-tuned and then synthesized with the lyrics created by the user. Therefore, when synthesizing a song, the music score basically remains unchanged. In this manual, when synthesizing a song, it is first necessary to determine the music score of the song to be synthesized as the basis for subsequent song synthesis.
[0120] In an optional implementation of this embodiment, before performing song synthesis, it is first necessary to receive the song to be synthesized. The specific implementation process of receiving the song to be synthesized can be as follows:
[0121] Receive imported songs to be synthesized; or,
[0122] A selection instruction for a target song in a preset song library is received, and the target song is determined as a song to be synthesized.
[0123] The preset song library is a pre-generated song collection including at least one song. The preset song library may include songs that are frequently synthesized. When a user needs to synthesize a song, he or she may directly select the song from the preset song library. If the preset song library does not contain the song that the user wants to synthesize, the user may also import the song that the user wants to synthesize.
[0124] In this manual, a preset song library can be provided to the user, so that the user can directly select the song he wants to synthesize from the preset song library. In this way, the user does not need to prepare the songs to be synthesized by himself, which reduces the user's operation and makes it easier for the user to synthesize songs. In addition, the user is not restricted to selecting the songs to be synthesized in the preset song library. The user can also import the songs he wants to synthesize by himself. The method of determining the songs to be synthesized is flexible and can adapt to the different needs of different users, thereby improving the user experience.
[0125] Step 104: creating a note display frame for the notes included in the music score, and displaying the note display frame on the music score display interface.
[0126] Specifically, on the basis of determining the score of the song to be synthesized, further, a note display frame will be created for the notes included in the score, and the note display frame will be displayed on the score display interface. The note display frame is an editable display frame created for the note, and the note display frame is used to represent the corresponding note. By editing the note display frame, the user can set the lyrics corresponding to the note represented by the note display frame; a corresponding note display frame is created for each note in the score of the song to be synthesized, and then displayed to the user through the score display interface. The score display interface is an interface for displaying multiple created note display frames, that is, an interface for displaying each note included in the score of the song to be synthesized.
[0127] In an optional implementation of this embodiment, since the note display frame is used to represent the corresponding note, the note display frame should be created according to the audio features (i.e., time length features) of the corresponding note, that is, the note display frame is created for the note included in the music score. The specific implementation process can be as follows:
[0128] For any note included in the music score, determine the duration feature of the note;
[0129] According to the duration feature, determining the length of the note display frame corresponding to the note;
[0130] Create a note display frame corresponding to the note according to the length.
[0131] The duration feature refers to the duration of a note. For example, if a note lasts for 1.5 seconds, then the duration feature is 1.5 seconds. Generally speaking, a basic length corresponding to a basic duration is predetermined, and then the length of the note display frame corresponding to each note is determined with reference to the basic length. For example, if the length of the note display frame corresponding to a duration of 1 second is 1 cm, then the length of the note display frame corresponding to a duration of 1.5 seconds is 1.5 cm, and the length of the note display frame corresponding to a duration of 0.5 seconds is 0.5 cm, and so on, the length of the audio display frame corresponding to each note included in the score is determined, thereby creating a corresponding audio display frame for each note included in the score.
[0132] It should be noted that after determining the length of the corresponding note display frame according to the duration characteristics of the note, when creating the note display frame, a blank note display frame can be created to wait for the user to enter the corresponding lyrics; or, a note display frame with a note logo can be created to prompt the user of the note represented by the note display frame, so that the user can enter the lyrics corresponding to the note in the note display frame according to the note logo.
[0133] Alternatively, the default lyrics corresponding to the notes represented by the notes can be directly displayed in the note display box. In this case, a note display box is created for the notes included in the score. The specific implementation process can be as follows:
[0134] For any note included in the score, determine the duration characteristics and default lyrics of the note;
[0135] Determine the length of the note display frame corresponding to the note according to the duration feature;
[0136] According to the length, a note display frame corresponding to the note is created, and the default lyrics are displayed in the note display frame.
[0137] The default lyrics refer to the original lyrics corresponding to the note in the song to be synthesized. It should be noted that the corresponding default lyrics are directly displayed in the note display box, and the user can modify the default lyrics or not. In this way, the user does not need to set the corresponding lyrics for each note display box, but only needs to enter the modified lyrics in the note display box corresponding to the note he wants to modify, which greatly improves the efficiency of song synthesis, facilitates user operation, and improves user experience.
[0138] In an optional implementation of this embodiment, the music score display interface may include a tone. After creating a note display frame for each note included in the music score of the song to be synthesized, each created note display frame needs to be displayed at the corresponding tone in the music score display interface. At this time, the note display frame is displayed on the music score display interface. The specific implementation process may be as follows:
[0139] Determine the pitch of the note corresponding to the note display frame;
[0140] Determine the starting position of the note in the music score display interface according to the starting time of the note in the music score;
[0141] Determine the display position of the note display frame corresponding to the note in the music score display interface according to the pitch of the note and the starting position of the note in the music score display interface;
[0142] A note display frame is displayed at a display position in the music score display interface.
[0143] It should be noted that different pitches correspond to different heights in the score display interface, and different start times correspond to different horizontal axis positions in the score display interface. Therefore, the display position of the note in the score display interface can be uniquely determined by the pitch of the note and the start position of the note, and then the note display frame corresponding to the note can be displayed at the display position. The display position of each created note display frame in the score display interface can be determined by the pitch of the note and the start position of the note, and then each created note display frame can be displayed at the corresponding display position.
[0144] For example, Figure 2 is a schematic diagram of a music score display interface provided in an embodiment of this specification, such as Figure 2 As shown in FIG. 1 , it is assumed that the score of the song to be synthesized includes 7 notes, and corresponding note display frames are created for the 7 notes. The 7 note display frames are blank and editable display frames. For the 7 note display frames, the display positions are determined according to the pitch and starting position of the corresponding notes, and displayed in the score display interface, as shown in FIG. Figure 2 shown.
[0145] Figure 3 is a schematic diagram of another music score display interface provided in an embodiment of this specification, such as Figure 3 As shown in the figure, assuming that the score of the song to be synthesized includes 7 notes, corresponding note display frames are created for the 7 notes. The 7 note display frames have corresponding default lyrics (the default lyrics are editable), and the default lyrics are yi, shan, yi, shan, liang, jing, jing. For the 7 note display frames, the display positions are determined according to the pitch and starting position of the corresponding notes, and displayed in the score display interface, as shown in the following figure. Figure 3 shown.
[0146] Furthermore, after the musical note display frame is displayed on the music score display interface, the user can also adjust the position of some musical note display frames, that is, the user can change the musical score melody of the song to be synthesized, so after the musical note display frame is displayed at the display position in the music score display interface, it also includes:
[0147] Receive movement instructions for a target note display frame;
[0148] According to the moving instruction, the target note display frame is moved from the current display position to the target display position, and the target display position is the tone position indicated by the moving instruction.
[0149] The move instruction is an instruction triggered by a preset move operation, and the move instruction is used to move the target note display frame from the current position to the target position, and the target note display frame refers to the note display frame selected by the move instruction. This manual does not limit the preset move operation, and any operation that can move the note display frame is acceptable, such as dragging a note display frame from the current position to another position (target display position), or selecting a note display frame, right-clicking, and moving the note display frame through the pop-up function options.
[0150] In this manual, after the note display frame is displayed on the music score display interface, a move instruction for the target note display frame can also be received to adjust the display position of the note display frame in the music score display interface, thereby changing the melody of the music score, so that the user can not only synthesize according to the original score of the song to be synthesized, but also change the music score of the song to be synthesized and re-make the music score to synthesize the song. The synthesis method of the song to be synthesized is flexible and can adapt to the different needs of different users, thereby improving the user experience.
[0151] Step 106: Receive the lyrics corresponding to the musical note through the musical note display frame.
[0152] Specifically, after creating a note display frame for the notes included in the score and displaying the note display frame on the score display interface, further, the lyrics corresponding to the notes will be received through the note display frame. Among them, the note display frame is editable, and the user can set the lyrics corresponding to the notes represented by the note display frame by editing the note display frame. If the note display frame displayed in the score display interface is blank or only has a note mark, then each note display frame needs to be selected by the user, and then the corresponding lyrics are entered.
[0153] In addition, in an optional implementation of this embodiment, if the note display box displayed in the music score display interface has default lyrics, the user can enter the modified lyrics only for the note display box that he wants to modify. At this time, the lyrics entered by the user will replace the default lyrics. If the user does not modify the default lyrics, the subsequent song synthesis steps are directly performed based on the default lyrics. Therefore, if the note display box has default lyrics, the lyrics corresponding to the note are received through the note display box. The specific implementation process can be as follows:
[0154] For any note display frame, determine whether an input instruction is received;
[0155] If yes, then receiving the updated lyrics inputted through the note display box, and determining the updated lyrics as the lyrics corresponding to the note;
[0156] If not, the default lyrics of the note display frame are determined as the lyrics corresponding to the note.
[0157] It should be noted that if the user enters updated lyrics in the note display box, the updated lyrics can be determined as the lyrics corresponding to the note represented by the note display box. If the user does not enter updated lyrics in the note display box, the default lyrics in the note display box can be determined as the lyrics corresponding to the note represented by the note display box and used directly in subsequent song synthesis.
[0158] Step 108: synthesize the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score.
[0159] Specifically, based on receiving the lyrics corresponding to the notes through the note display frame, the song to be synthesized is further synthesized according to the music score and the lyrics corresponding to each note included in the music score.
[0160] In an optional implementation of this embodiment, the song to be synthesized is synthesized according to the music score and the lyrics corresponding to each note included in the music score. The specific implementation process may be as follows:
[0161] Synthesize a voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score;
[0162] The voice file is synthesized with the music score to obtain the song to be synthesized.
[0163] Before synthesizing the voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score, it is also necessary to prepare the audio required for synthesizing the voice file, that is, before synthesizing the voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score, it also includes:
[0164] An imported lyrics audio library is received, where the lyrics audio library is an audio collection including at least one monosyllabic audio.
[0165] It should be noted that monosyllabic audio refers to the audio recording of a Chinese character pinyin, that is, one Chinese character pinyin corresponds to one monosyllabic audio. Before synthesizing a song, the user needs to prepare the audio material required for synthesizing the song. One possible implementation method is that the user can record the monosyllabic audio corresponding to the required lyrics by himself, and then combine the recorded monosyllabic audios into a lyrics audio library and import it; another possible implementation method is that the user can directly download the monosyllabic audio corresponding to the required lyrics from a mature audio material library on the Internet, and then combine the downloaded monosyllabic audios into a lyrics audio library and import it.
[0166] In addition, this manual does not limit the monosyllabic audio included in the imported lyrics audio library. When synthesizing a song, the user generally knows clearly which lyrics are needed for the song he wants to synthesize. Therefore, the lyrics audio library can be composed of only the monosyllabic audio corresponding to the lyrics required for the synthesized song; or, in order to prevent the user from temporarily wanting to use other lyrics during the song synthesis process, the lyrics audio library can also be composed of a large number of monosyllabic audio corresponding to the lyrics commonly used in synthesized songs. When synthesizing songs later, the audio corresponding to the required lyrics can be searched therefrom.
[0167] For example, Figure 4 This is a schematic diagram of an interface of a lyrics audio library provided in an embodiment of this specification, such as Figure 4 As shown, the received lyrics audio library includes 9 monosyllabic audios, namely yi, shan, liang, jing, chan, chang, che, chen, and cheng.
[0168] In this manual, before synthesizing a song, when preparing audio materials, the user can either record the required lyrics audio library by himself or download the required lyrics audio library directly from the Internet. The preparation method of the lyrics audio library of the song to be synthesized is flexible and can adapt to the different needs of different users, thereby improving the user experience.
[0169] In the specific implementation, according to the lyrics corresponding to each note included in the music score, the voice file of the song to be synthesized is synthesized, including:
[0170] Obtaining monosyllabic audio of lyrics corresponding to each note included in the music score from the lyrics audio library;
[0171] Adjust the monosyllable audio to the target audio of the note in the score;
[0172] The target audio is synthesized to obtain the voice file of the song to be synthesized.
[0173] It should be noted that, generally speaking, the monosyllabic audio obtained from the lyrics audio library is the original audio, and the audio features (such as pitch, duration, etc.) of the original audio and the target audio of the corresponding note in the music score may be different. Therefore, after obtaining the monosyllabic audio of the lyrics corresponding to each note included in the music score from the lyrics audio library, each monosyllabic audio obtained needs to be adjusted to the target audio of the corresponding note in the music score.
[0174] For example, the score includes 7 notes, the lyrics corresponding to note 1 are yi, the lyrics corresponding to note 2 are shan, the lyrics corresponding to note 3 are yi, the lyrics corresponding to note 4 are shan, the lyrics corresponding to note 5 are liang, the lyrics corresponding to note 6 are jing, and the lyrics corresponding to note 7 are jing. After obtaining the monosyllabic audios of the lyrics corresponding to the above 7 notes from the lyrics audio library, the above 7 monosyllabic audios are adjusted to the target audios of the 7 notes in the score.
[0175] In addition, when receiving the lyrics corresponding to the notes through the note display box, the received lyrics may be in Chinese characters or directly in pinyin, and the monosyllabic audio stored in the lyrics audio library is a separate recording audio of Chinese pinyin. Therefore, it is necessary to search in the lyrics audio library according to the pinyin of the lyrics to obtain the monosyllabic audio corresponding to the pinyin, that is, to obtain the monosyllabic audio corresponding to each note included in the music score from the lyrics audio library. The specific implementation process can be as follows:
[0176] Determine whether the lyrics received through the note display frame are pinyin;
[0177] If yes, then obtain the monosyllabic audio corresponding to the pinyin from the lyrics audio library;
[0178] If not, the lyrics are converted into pinyin, and then the monosyllabic audio corresponding to the pinyin is obtained from the lyrics audio library.
[0179] It should be noted that if the lyrics received through the musical note display box are in pinyin, you can directly search in the lyrics audio library according to the pinyin to obtain the monosyllabic audio corresponding to the pinyin; and if the lyrics received through the musical note display box are in Chinese characters, you can convert the Chinese characters into pinyin through simple recognition, and then search in the lyrics audio library according to the pinyin to obtain the monosyllabic audio corresponding to the Chinese characters.
[0180] In this manual, users can directly enter the Chinese characters or pinyin of the lyrics they want to synthesize in the note display box, which adapts to the input habits of different users and improves the user experience.
[0181] In an optional implementation of this embodiment, the monosyllabic audio is adjusted to the target audio of the note in the music score, and the specific implementation process can be as follows:
[0182] Determine the original duration and original pitch of monosyllabic audio;
[0183] Determine the target duration and target pitch of the note in the score;
[0184] The original pitch is adjusted to the target pitch, and the original duration is adjusted to the target duration, so as to obtain the target audio of the note in the score.
[0185] The original pitch of the monosyllabic audio is determined, and the specific implementation process can be as follows:
[0186] In the case where the monosyllabic audio carries a pitch, reading the pitch carried in the monosyllabic audio, and determining the pitch as the original pitch of the monosyllabic audio;
[0187] In the case where the monosyllabic audio does not carry any pitch, the average frequency of the monosyllabic audio is determined, and the original pitch of the monosyllabic audio is determined based on the average frequency.
[0188] It should be noted that if the received lyrics audio library is downloaded by the user from a mature audio material library on the Internet, then the monosyllabic audio included in the lyrics audio library will generally carry the corresponding pitch directly, and can be directly read at this time; if the received lyrics audio library is recorded by the user himself, then the monosyllabic audio included in the lyrics audio library will generally not carry the corresponding pitch. At this time, the pitch of the monosyllabic audio can be identified by a simple pitch recognition algorithm, such as by determining the average frequency of the monosyllabic audio to determine the original pitch of the monosyllabic audio.
[0189] For example, Figure 4 As shown, the 9 monosyllabic audios included in the received lyrics audio library all carry corresponding pitches, the original pitch of yi is 170Hz, the original pitch of shan is 177Hz, the original pitch of liang is 180Hz, the original pitch of jing is 178Hz, the original pitch of chan is 181Hz, the original pitch of chang is 188Hz, the original pitch of che is 172Hz, the original pitch of chen is 177Hz, and the original pitch of cheng is 174Hz.
[0190] In addition, the original duration is adjusted to the target duration. The specific implementation process can be as follows:
[0191] When the original duration and the target duration are equal, the original duration is determined as the target duration;
[0192] When the original duration and the target duration are not equal, determine whether the original duration exceeds the target duration. If so, cut the original duration to the target duration; if not, stretch the original duration to the target duration.
[0193] Among them, stretching the original duration to the target duration can be specifically implemented as follows:
[0194] When there is a phoneme segmentation identifier carried in the monosyllabic audio, read the phoneme segmentation identifier carried in the monosyllabic audio, where the phoneme segmentation identifier is used to segment the phonemes included in the monosyllabic audio; determine the vowels and consonants in the monosyllabic audio according to the phoneme segmentation identifier, keep the original duration of the consonants, and stretch the original duration of the vowels until the target duration is reached;
[0195] When there is no phoneme segmentation identifier carried in the monosyllabic audio, stretch each phoneme included in the monosyllabic audio until the target duration is reached.
[0196] Specifically, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phoneme. Phonemes are divided into two major categories: vowels and consonants. For example, the Chinese syllable "啊 (ā)" has only one phoneme, "爱 (ài)" has two phonemes, and "代 (dài)" has three phonemes. In addition, the phoneme segmentation identifier refers to the identifier that segments the vowels and consonants in the monosyllabic audio. For example, if the monosyllabic audio is "zhang", a phoneme segmentation identifier can be set between "zh" and "ang" at this time to separate "zh" and "ang".
[0197] If the received lyric audio library is downloaded by the user from a mature audio material library on the Internet, then the monosyllabic audio included in the lyric audio library generally carries a phoneme segmentation identifier directly. At this time, it can be directly read, and then the monosyllabic audio is segmented according to the phoneme segmentation identifier, keeping the consonants unchanged and only stretching the vowels; if the received lyric audio library is recorded by the user himself, then the monosyllabic audio included in the lyric audio library generally does not carry a phoneme segmentation identifier. At this time, the entire monosyllabic audio can be directly stretched.
[0198] It should be noted that, for a single note, during the synthesis process, the sound source in the sound source library (i.e., the lyrics audio library) will first be searched to obtain the corresponding audio file (i.e., monosyllabic audio). The sound source itself will have a basic pitch, and the current note will have a target pitch. In order to generate a melody, the current pitch of the sound source will be adjusted to the target pitch through a pitch change algorithm. For the duration of the note (i.e., time length), if the length of the original audio exceeds the duration of the current note, the original audio will be directly cropped to remove the excess part. If the length of the original audio is less than the required duration, the audio will be stretched. There are two possible situations for the stretching operation. The first situation is to stretch directly, and the audio will show the effect of variable speed playback (such as stretching to 2 times, the effect is equivalent to 0.5 times the speed of playback); the second situation is to distinguish the stretching of vowels and consonants according to the phoneme segmentation mark. For the consonant part, it will not change during the stretching, and for the vowel part, the method equivalent to the first situation will be used to stretch until the required duration can be filled.
[0199] For example, Figure 5 is a schematic diagram of stretching a monosyllabic audio provided in an embodiment of this specification, such as Figure 5 As shown, the monosyllabic audio is zhang. Assuming that there is a phoneme segmentation mark between zh and ang, zh is kept unchanged and ang is stretched to the target duration; assuming that the monosyllabic audio zhang does not carry a phoneme segmentation mark, the entire zhang is stretched to the target duration.
[0200] In an optional implementation of this embodiment, the target audio is synthesized to obtain a voice file of the song to be synthesized. The specific implementation process may be as follows:
[0201] If two adjacent notes overlap, the target audio corresponding to the previous note is faded out, and the target audio corresponding to the next note is faded in.
[0202] It should be noted that in order to ensure the continuity of the pronunciation of the synthesized song, for some specific pinyin, there is an overlapping part between the pronunciation of the two notes before and after. When processing the overlapping part, the audio of the previous note can be faded out and the audio of the next note can be faded in to ensure a smooth transition of the sound.
[0203] In addition, the above song synthesis implementation can be implemented in the browser. Basic audio operations (such as cropping and splicing) can be performed based on the Audio Buffer interface in the Web Audio API. Some algorithms, such as tuning and speed change algorithms, can be implemented through Web Assembly.
[0204] In an optional implementation of this embodiment, after synthesizing the voice file of the song to be synthesized, a trial listening may be performed, that is, according to the lyrics corresponding to each note included in the music score, after synthesizing the voice file of the song to be synthesized, the following may be further included:
[0205] When a pre-listening instruction for a voice file is received, the synthesized voice file is played.
[0206] It should be noted that in the actual process of synthesizing a song, the user can fill in the corresponding single-word pinyin in each note display box according to the prepared lyrics. After all the pinyin is filled in, the synthesis result can be previewed. Then, each note will be traversed according to the music score for synthesis, and the user can hear the changes in the synthesized content in real time.
[0207] In an optional implementation of this embodiment, when synthesizing a song to be synthesized, it is necessary to align the synthesized voice file and the music score, and then synthesize them, so that the voice file and the music score are synthesized to obtain the song to be synthesized. The specific implementation process can be as follows:
[0208] Calculate the length of the measure of the music score based on the number of beats per minute of the music score;
[0209] According to the length of the measure, the beats of the speech file and the music score are aligned to obtain the song to be synthesized.
[0210] It should be noted that in this manual, the actual duration of a measure of the score can be automatically calculated based on the BPM (Beats per Minutes) of the score, so that the synthesized voice file part can be aligned with the prepared score (i.e., accompaniment) in the beat.
[0211] In an optional implementation of this embodiment, after the synthesis effect of the voice file part is determined, the user can select the music score (i.e., accompaniment) of the song. Since there may be an offset in the beat between the music score and the synthesized voice file, the user can also manually set the offset for the music score to ensure the effect of synthesizing the song, that is, synthesizing the voice file with the music score to obtain the song to be synthesized. The specific implementation process can be as follows:
[0212] When receiving a music score adding instruction, adding the music score to the voice file;
[0213] Receives the offset set for the score;
[0214] The music score is shifted according to the shift amount;
[0215] The shifted music score and voice file are synthesized to obtain the song to be synthesized.
[0216] The offset refers to the offset time, such as moving the music score forward by 0.5 seconds (offset is 0.5 seconds) to align with the voice file; or moving the music score backward by 1 second (offset is 1 second) to align with the voice file.
[0217] It should be noted that in order to ensure the synthesis effect, the user can listen to the synthesized voice file without adding accompaniment (i.e., music score), and only listen to the pure voice file. After determining the effect of the voice file, the corresponding music score is added to the voice file. The music score here refers to the music score corresponding to the previous note display box. In order to adjust the offset of the music score and the synthesized voice file in the beat, the corresponding offset can be set when adding the music score, and the music score is adjusted, and then the music score and the voice file are synthesized to obtain the synthesized song.
[0218] In an optional implementation of this embodiment, after synthesizing the shifted music score and the voice file to obtain the song to be synthesized, the following step is further included:
[0219] In case of receiving an export instruction for the song to be synthesized, determining a target file format indicated by the export instruction for export;
[0220] Export the synthesized song to be synthesized according to the target file format.
[0221] It should be noted that after determining the effect of the overall song synthesis, the user can choose to export the synthesized song. The voice file part and the selected music score will be automatically merged during export. In addition, the user can also set the export format by himself, such as exporting audio files in wav or aac format.
[0222] The song synthesis method provided in this specification can first receive the song to be synthesized, determine the score of the song to be synthesized, then create a note display box for the notes included in the score, and display the note display box on the score display interface; then, the lyrics corresponding to the notes can be received through the note display box, and finally, the song to be synthesized can be synthesized according to the score and the lyrics corresponding to each note included in the score. In this case, a score display interface will be displayed, and the note display box will be displayed on the score display interface. The user only needs to enter the lyrics in the note display box to synthesize the song he wants to synthesize. The song synthesis process is simple and easy to operate, so that any user who does not have any song synthesis software usage skills can easily synthesize songs, which lowers the threshold for song synthesis, facilitates user use, and improves user experience.
[0223] The following combination Figure 6 , taking the application of the song synthesis method provided in this specification in a browser as an example, the song synthesis method is further described. Figure 6A processing flow chart of a song synthesis method applied to a browser provided in an embodiment of this specification is shown, which specifically includes the following steps:
[0224] Step 602: Receive a lyrics audio library imported by a user through a song creation interface provided by a browser, where the lyrics audio library is an audio collection including at least one monosyllabic audio.
[0225] In actual applications, song synthesis software needs to be downloaded and installed in advance, and users need to master a lot of software usage skills to synthesize songs. The song synthesis process is complicated and difficult to operate. Therefore, this specification provides a song synthesis method applied to a browser, which can display a music score display interface on the browser interface, and display a note display box on the music score display interface. Users only need to enter lyrics in the note display box to synthesize the song they want to synthesize. The song synthesis process is simple and easy to operate, so that any user who does not have any song synthesis software usage skills can easily synthesize a complete Chinese song on the browser.
[0226] Step 604: Receive the song to be synthesized imported by the user through the song creation interface provided by the browser.
[0227] Step 606: Receive a selection instruction for a target song in a preset song library displayed on a song creation interface provided by the browser, and determine the target song as the song to be synthesized.
[0228] It should be noted that both step 604 and step 606 are steps for determining the song to be synthesized. In actual implementation, only one of the two can be selected, and both do not need to be executed.
[0229] Step 608: Determine the music score of the song to be synthesized.
[0230] Step 610: Create a note display frame for the notes included in the music score, and display the note display frame on the music score display interface in the browser.
[0231] Step 612: Receive lyrics corresponding to the musical note through the musical note display frame.
[0232] Step 614: synthesize the voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score; synthesize the voice file with the music score to obtain the synthesized song.
[0233] The song synthesis method provided in this specification receives a lyrics audio library imported by a user through a song creation interface provided by a browser, and the lyrics audio library is an audio collection including at least one monosyllabic audio; then receives a song to be synthesized imported by a user through the song creation interface provided by the browser, or receives a selection instruction for a target song in a preset song library displayed on the song creation interface provided by the browser, and determines the target song as the song to be synthesized; then, determines the score of the song to be synthesized, creates a note display box for the notes included in the score, and displays the note display box on the score display interface in the browser, and receives the lyrics corresponding to the notes through the note display box; finally, according to the lyrics corresponding to each note included in the score, a voice file of the song to be synthesized can be synthesized with the score, and the voice file is synthesized with the score to obtain the song to be synthesized. In this case, a score display interface can be displayed on the browser interface, and a note display box can be displayed on the score display interface. The user only needs to enter the lyrics in the note display box to synthesize the song he wants to synthesize. The song synthesis process is simple and easy to operate, so that any user who does not have any song synthesis software usage skills can easily synthesize a complete Chinese song on the browser.
[0234] The following combination Figure 7 , taking the application of the song synthesis method provided in this specification in the ghost animal scene as an example, the song synthesis method is further explained. Among them, Figure 7 The processing flow chart of a song synthesis method applied to a ghost animal scene provided by an embodiment of this specification is shown. It should be noted that the song synthesis method can be specifically applied in the production process of ghost animal songs. Ghost animal songs often simulate the effect of real people singing by sampling real people's recordings and performing later sound adjustment. Specifically, the following steps may be included:
[0235] Step 702: Receive a ghost animal lyrics sound source library imported by a user through a song creation interface provided by a browser, where the ghost animal lyrics sound source library is a collection of audios including at least one ghost animal lyric.
[0236] It should be noted that step 702 is the material preparation stage for making the ghost animal song. After the user has created the lyrics of the ghost animal song, he can read the Chinese characters included in the lyrics, record the audio of the ghost animal lyrics (i.e., the recording of the voice file), and then merge the recorded audios into the ghost animal lyrics sound source library, and import them into the singing creation module in the browser. Among them, when the user records the audio of the ghost animal lyrics, he can prepare for the pinyin of the single words used in the required lyrics, because in the singing process of Chinese songs, the pronunciation of the lyrics is "first tone", so the voice file recording can use "first tone" for recording.
[0237] Step 704: Receive the music score of the ghost song to be produced imported by the user through the song creation interface provided by the browser.
[0238] Specifically, the music score of the ghost animal song to be produced is the accompaniment of the ghost animal song to be produced.
[0239] Step 706: Create a note display frame for the notes included in the music score, and display the note display frame on the music score display interface in the browser.
[0240] It should be noted that for the music score, the browser will draw each note in the music score in an interactive graphical interface (music score display interface), the high and low positions of the note display frame represent the corresponding different pitches, and the length of the note display frame represents the different durations of the notes.
[0241] Step 708: Receive the ghost lyrics corresponding to the notes through the note display box.
[0242] Step 710: synthesize the voice file of the ghost song to be produced according to the ghost lyrics corresponding to each note included in the music score; synthesize the voice file with the music score to obtain the completed ghost song.
[0243] The song synthesis method provided in this specification receives the ghost animal lyrics sound source library imported by the user through the song creation interface provided by the browser, and receives the music score of the ghost animal song to be produced imported by the user through the song creation interface provided by the browser; then creates a note display frame for the notes included in the music score, and displays the note display frame on the music score display interface in the browser; after that, receives the ghost animal lyrics corresponding to the notes through the note display frame, synthesizes the voice file of the ghost animal song to be produced according to the ghost animal lyrics corresponding to each note included in the music score, synthesizes the voice file with the music score, and obtains the finished ghost animal song. In this case, a music score display interface can be displayed on the browser interface, and a note display frame can be displayed on the music score display interface. The user only needs to enter the lyrics in the note display box to synthesize the ghost animal song he wants to produce. The ghost animal song production process is simple and easy to operate, so that any user who does not have any song synthesis software usage skills can easily make a complete ghost animal song on the browser.
[0244] Corresponding to the above method embodiment, this specification also provides a song synthesis device embodiment, Figure 8 FIG. 1 shows a schematic diagram of the structure of a song synthesis device provided in an embodiment of the present specification. Figure 8 As shown, the device comprises:
[0245] The determination module 802 is configured to receive a song to be synthesized and determine a music score of the song to be synthesized;
[0246] A display module 804 is configured to create a note display frame for the notes included in the music score, and display the note display frame on the music score display interface;
[0247] The first receiving module 806 is configured to receive lyrics corresponding to the musical note through the musical note display frame;
[0248] The synthesis module 808 is configured to synthesize the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score.
[0249] Optionally, the display module 804 is further configured to:
[0250] For any note included in the music score, determining a duration feature of the note;
[0251] Determining the length of the note display frame corresponding to the note according to the duration feature;
[0252] Create a note display frame corresponding to the note according to the length.
[0253] Optionally, the display module 804 is further configured to:
[0254] Determining the pitch of the note corresponding to the note display frame;
[0255] Determining the starting position of the note in the music score display interface according to the starting time of the note in the music score;
[0256] Determining a display position of a note display frame corresponding to the note in the music score display interface according to the pitch of the note and the starting position of the note in the music score display interface;
[0257] The note display frame is displayed at the display position in the music score display interface.
[0258] Optionally, the synthesis module 808 is further configured to:
[0259] Synthesizing a voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score;
[0260] The voice file is synthesized with the music score to obtain the song to be synthesized.
[0261] Optionally, the synthesis module 808 is further configured to:
[0262] An imported lyrics audio library is received, where the lyrics audio library is an audio collection including at least one monosyllabic audio.
[0263] Optionally, the synthesis module 808 is further configured to:
[0264] Acquire the monosyllabic audio of the lyrics corresponding to each note included in the music score from the lyrics audio library;
[0265] adjusting the monosyllabic audio to the target audio of the note in the musical score;
[0266] The target audio is synthesized to obtain a voice file of the song to be synthesized.
[0267] Optionally, the synthesis module 808 is further configured to:
[0268] Determining the original duration and original pitch of the monosyllabic audio;
[0269] Determining a target duration and a target pitch of the note in the musical score;
[0270] The original pitch is adjusted to the target pitch, and the original duration is adjusted to the target duration, so as to obtain the target audio of the note in the music score.
[0271] Optionally, the synthesis module 808 is further configured to:
[0272] In the case where the monosyllabic audio carries a pitch, the pitch carried in the monosyllabic audio is read, and the pitch is determined as the original pitch of the monosyllabic audio.
[0273] Optionally, the synthesis module 808 is further configured to:
[0274] In the case where the monosyllabic audio does not carry a pitch, determining an average frequency of the monosyllabic audio;
[0275] The original pitch of the monosyllabic audio is determined according to the average frequency.
[0276] Optionally, the synthesis module 808 is further configured to:
[0277] When the original duration is equal to the target duration, the original duration is determined as the target duration;
[0278] When the original duration and the target duration are not equal, determine whether the original duration exceeds the target duration. If so, cut the original duration to the target duration; if not, stretch the original duration to the target duration.
[0279] Optionally, the synthesis module 808 is further configured to:
[0280] In a case where the monosyllabic audio carries a phoneme segmentation identifier, reading the phoneme segmentation identifier carried in the monosyllabic audio, where the phoneme segmentation identifier is used to segment the phonemes included in the monosyllabic audio;
[0281] Determine the vowels and consonants in the monosyllabic audio according to the phoneme segmentation identifier;
[0282] The original duration of the consonant is maintained, and the original duration of the vowel is stretched until the target duration is reached.
[0283] Optionally, the synthesis module 808 is further configured to:
[0284] In the case that the monosyllabic audio does not carry a phoneme segmentation identifier, each phoneme included in the monosyllabic audio is stretched until the target duration is reached.
[0285] Optionally, the device further comprises:
[0286] The second receiving module is configured to receive an imported song to be synthesized; or,
[0287] The third receiving module is configured to receive a selection instruction for a target song in a preset song library, and determine the target song as the song to be synthesized.
[0288] Optionally, the display module 804 is further configured to:
[0289] Receive movement instructions for a target note display frame;
[0290] According to the movement instruction, the target note display frame is moved from the current display position to the target display position, and the target display position is the tone position indicated by the movement instruction.
[0291] Optionally, the display module 804 is further configured to:
[0292] Calculate the duration of a measure of the music score according to the number of beats per minute of the music score;
[0293] According to the measure duration, the voice file is rhythmically aligned with the music score, and the song to be synthesized is synthesized.
[0294] Optionally, the display module 804 is further configured to:
[0295] When a pre-listening instruction for the voice file is received, the synthesized voice file is played.
[0296] Optionally, the display module 804 is further configured to:
[0297] When receiving a music score adding instruction, adding the music score to the voice file;
[0298] receiving an offset set for the score;
[0299] offsetting the music score according to the offset;
[0300] The shifted music score and the voice file are synthesized to obtain the song to be synthesized.
[0301] Optionally, the display module 804 is further configured to:
[0302] In case of receiving an export instruction for the song to be synthesized, determining a target file format indicated by the export instruction for export;
[0303] The synthesized song to be synthesized is exported according to the target file format.
[0304] Optionally, the synthesis module 808 is further configured to:
[0305] Determine whether the lyrics received through the musical note display frame are pinyin;
[0306] If yes, then obtain the monosyllabic audio corresponding to the pinyin from the lyrics audio library;
[0307] If not, the lyrics are converted into pinyin, and then the monosyllabic audio corresponding to the pinyin is obtained from the lyrics audio library.
[0308] Optionally, the display module 804 is further configured to:
[0309] For any note included in the music score, determining the duration characteristics and default lyrics of the note;
[0310] Determining the length of the note display frame corresponding to the note according to the duration feature;
[0311] According to the length, a note display frame corresponding to the note is created, and the default lyrics are displayed in the note display frame.
[0312] Optionally, the display module 804 is further configured to:
[0313] For any of the musical note display frames, determining whether an input instruction is received;
[0314] If yes, receiving the input updated lyrics through the note display box, and determining the updated lyrics as the lyrics corresponding to the note;
[0315] If not, the default lyrics of the note display frame are determined as the lyrics corresponding to the note.
[0316] Optionally, the synthesis module 808 is further configured to:
[0317] If two adjacent notes overlap, the target audio corresponding to the previous note is faded out, and the target audio corresponding to the next note is faded in.
[0318] The song synthesis device provided in this specification can first receive the song to be synthesized and determine the score of the song to be synthesized, then create a note display frame for the notes included in the score, and display the note display frame on the score display interface; after that, the lyrics corresponding to the notes can be received through the note display frame, and finally the song to be synthesized can be synthesized according to the score and the lyrics corresponding to each note included in the score. In this case, a score display interface will be displayed, and the note display frame will be displayed on the score display interface. The user only needs to enter the lyrics in the note display box to synthesize the song he wants to synthesize. The song synthesis process is simple and easy to operate, so that any user who does not have any skills in using song synthesis software can easily synthesize songs, which lowers the threshold for song synthesis, facilitates user use, and improves user experience.
[0319] The above is a schematic scheme of a song synthesis device of this embodiment. It should be noted that the technical scheme of the song synthesis device and the technical scheme of the song synthesis method described above are of the same concept, and the details not described in detail in the technical scheme of the song synthesis device can be referred to the description of the technical scheme of the song synthesis method described above.
[0320] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.
[0321] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0322] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0323] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 900 may also be a mobile or stationary server.
[0324] The processor 920 is used to execute the following computer executable instructions to implement the following method:
[0325] Receiving a song to be synthesized, and determining a music score of the song to be synthesized;
[0326] Creating a note display frame for the notes included in the music score, and displaying the note display frame on a music score display interface;
[0327] Receiving lyrics corresponding to the musical note through the musical note display frame;
[0328] The song to be synthesized is synthesized according to the music score and the lyrics corresponding to each note included in the music score.
[0329] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the song synthesis method described above are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the song synthesis method described above.
[0330] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which are used to implement the following method when executed by a processor:
[0331] Receiving a song to be synthesized, and determining a music score of the song to be synthesized;
[0332] Creating a note display frame for the notes included in the music score, and displaying the note display frame on a music score display interface;
[0333] Receiving lyrics corresponding to the musical note through the musical note display frame;
[0334] The song to be synthesized is synthesized according to the music score and the lyrics corresponding to each note included in the music score.
[0335] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the song synthesis method described above are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the song synthesis method described above.
[0336] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0337] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0338] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.
[0339] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0340] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.
Claims
1. A song synthesis method, It is characterized in that The method comprises: Receiving a song to be synthesized, and determining a music score of the song to be synthesized; Creating a note display frame for the notes included in the music score, and displaying the note display frame on a music score display interface; Receiving lyrics corresponding to the musical note through the musical note display frame; synthesizing the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score; The step of synthesizing the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score comprises: Synthesizing a voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score; The voice file is synthesized with the music score to obtain the song to be synthesized.
2. The song synthesis method according to claim 1, It is characterized in that The step of creating a note display frame for the notes included in the music score comprises: For any note included in the music score, determining a duration feature of the note; Determining the length of the note display frame corresponding to the note according to the duration feature; Create a note display frame corresponding to the note according to the length.
3. The song synthesis method according to claim 1 or 2, It is characterized in that The music score display interface includes a tone, and the displaying of the note display frame on the music score display interface includes: Determining the pitch of the note corresponding to the note display frame; Determining the starting position of the note in the music score display interface according to the starting time of the note in the music score; Determining a display position of a note display frame corresponding to the note in the music score display interface according to the pitch of the note and the starting position of the note in the music score display interface; The note display frame is displayed at the display position in the music score display interface.
4. The song synthesis method according to claim 1, It is characterized in that Before synthesizing the voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score, the method further includes: An imported lyrics audio library is received, where the lyrics audio library is an audio collection including at least one monosyllabic audio.
5. The song synthesis method according to claim 4, It is characterized in that The step of synthesizing the voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score comprises: Acquire the monosyllabic audio of the lyrics corresponding to each note included in the music score from the lyrics audio library; adjusting the monosyllabic audio to the target audio of the note in the musical score; The target audio is synthesized to obtain a voice file of the song to be synthesized.
6. The song synthesis method according to claim 5, It is characterized in that The step of adjusting the monosyllabic audio to a target audio of the note in the music score comprises: Determining the original duration and original pitch of the monosyllabic audio; Determining a target duration and a target pitch of the note in the musical score; The original pitch is adjusted to the target pitch, and the original duration is adjusted to the target duration, so as to obtain the target audio of the note in the music score.
7. The song synthesis method according to claim 6, It is characterized in that The adjusting the original duration to the target duration includes: In a case where the monosyllabic audio carries a phoneme segmentation identifier, reading the phoneme segmentation identifier carried in the monosyllabic audio, where the phoneme segmentation identifier is used to segment the phonemes included in the monosyllabic audio; Determine the vowels and consonants in the monosyllabic audio according to the phoneme segmentation identifier; The original duration of the consonant is maintained, and the original duration of the vowel is stretched until the target duration is reached.
8. The song synthesis method according to claim 6, It is characterized in that The adjusting the original duration to the target duration includes: In the case where the monosyllabic audio does not carry a phoneme segmentation identifier, each phoneme included in the monosyllabic audio is stretched until the target duration is reached.
9. The song synthesis method according to claim 1 or 2, It is characterized in that The receiving of the song to be synthesized comprises: Receive imported songs to be synthesized; or, A selection instruction for a target song in a preset song library is received, and the target song is determined as the song to be synthesized.
10. The song synthesis method according to claim 3, It is characterized in that After the note display frame is displayed at the display position in the music score display interface, the method further includes: Receive movement instructions for a target note display frame; According to the movement instruction, the target note display frame is moved from the current display position to the target display position, and the target display position is the tone position indicated by the movement instruction.
11. The song synthesis method according to claim 1, It is characterized in that The step of synthesizing the voice file with the music score to obtain the song to be synthesized includes: Calculate the duration of a measure of the music score according to the number of beats per minute of the music score; According to the measure duration, the voice file is rhythmically aligned with the music score to obtain the song to be synthesized.
12. The song synthesis method according to claim 1, It is characterized in that The step of synthesizing the voice file with the music score to obtain the song to be synthesized includes: When receiving a music score adding instruction, adding the music score to the voice file; receiving an offset set for the score; offsetting the music score according to the offset; The shifted music score and the voice file are synthesized to obtain the song to be synthesized.
13. The song synthesis method according to claim 5, It is characterized in that The step of obtaining the monosyllabic audio corresponding to each note included in the music score from the lyrics audio library comprises: Determine whether the lyrics received through the musical note display frame are pinyin; If yes, then obtain the monosyllabic audio corresponding to the pinyin from the lyrics audio library; If not, the lyrics are converted into pinyin, and then the monosyllabic audio corresponding to the pinyin is obtained from the lyrics audio library.
14. The song synthesis method according to claim 1, It is characterized in that The step of creating a note display frame for the notes included in the music score comprises: For any note included in the music score, determining the duration characteristics and default lyrics of the note; Determining the length of the note display frame corresponding to the note according to the duration feature; According to the length, a note display frame corresponding to the note is created, and the default lyrics are displayed in the note display frame.
15. The song synthesis method according to claim 5, It is characterized in that The target audio is synthesized, Obtaining the voice file of the song to be synthesized, including: If two adjacent notes overlap, the target audio corresponding to the previous note is faded out, and the target audio corresponding to the next note is faded out. Fade in the target audio corresponding to each note.
16. A song synthesis device, It is characterized in that The device comprises: A determination module is configured to receive a song to be synthesized and determine a music score of the song to be synthesized; A display module, configured to create a note display frame for the notes included in the music score, and display the note display frame on the music score display interface; A receiving module is configured to receive lyrics corresponding to the musical note through the musical note display frame; A synthesis module is configured to synthesize the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score; Wherein, the synthesis module is also used for: Synthesizing a voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score; The voice file is synthesized with the music score to obtain the song to be synthesized.
17. A computing device, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the following method: Receiving a song to be synthesized, and determining a music score of the song to be synthesized; Creating a note display frame for the notes included in the music score, and displaying the note display frame on a music score display interface; Receiving lyrics corresponding to the musical note through the musical note display frame; synthesizing the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score; The step of synthesizing the song to be synthesized according to the music score and the lyrics corresponding to each note included in the music score comprises: Synthesizing a voice file of the song to be synthesized according to the lyrics corresponding to each note included in the music score; The voice file is synthesized with the music score to obtain the song to be synthesized.
18. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the song synthesis method described in any one of claims 1 to 15.
19. A computer program product comprising computer instructions, It is characterized in that When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 15 are implemented.
