Singing sound generation method and device, electronic equipment and storage medium

Abstract

The invention belongs to the technical field of human voice generation, and particularly relates to a singing sound generation method and device, electronic equipment and a storage medium. The method comprises the following steps: inputting a lyric to be voiced and a required singing style into a first vector model to obtain a lyric tone vector and a style tone vector; inputting the required singing style into a second vector model to obtain a background sound vector; obtaining a final human voice based on the lyric voice vector and the style voice vector; and obtaining a final background sound based on the background sound vector, and determining a singing sound by combining the final human voice and the final background sound. Through application of the method and the device, the user can specify the singer sound by providing the required singing style and the lyrics generated by the language model according to own preferences and requirements, and the singing characteristics of the singer are reserved, so that the style of the music work with the real person sound is customized, and the user experience is improved. Moreover, the automatic generation process greatly improves the efficiency of music creation, and reduces the complexity and time consumption of manual creation.

Description

Technical Field

[0001] The present application belongs to the technical field of human voice generation, and in particular, relates to a method, device, electronic device and storage medium for generating singing voice. Background Art

[0002] With the continuous optimization of deep learning algorithms, speech synthesis technology has been able to achieve very close to natural human speech generation. These technologies train large-scale neural network models to learn the characteristics and rules of human speech, so as to simulate various voice styles and emotional expressions.

[0003] Apart from the suno.ai platform, there is no mature way to generate human voices. The current mainstream approach is transposition (i.e. adjusting the frequency based on an existing song to achieve a similarity to a certain person); there are few methods to generate human voices from a single melody, and there is basically no method that can maintain the characteristics of the singer. Currently, the generation of lyrics and singing voices is a combination of lyrics + MIDI, giving each word of the lyrics a different sound, but this method is difficult to consider the integrity of the entire song, and because it is MIDI sound, the sound produced is in a mechanical pronunciation state, and the generated sound cannot be transformed into different styles according to user needs.

[0004] Therefore, how to retain the characteristics of real human voices during the sound generation process and transform the generated sounds into different styles as needed has become an urgent problem that needs to be solved. Summary of the invention

[0005] The present application provides a method, device, electronic device and storage medium for generating singing voices, which can generate sounds of different styles according to different needs, and can specify a singer while retaining the singer's singing characteristics while improving the generation quality.

[0006] In a first aspect, an embodiment of the present application provides a method for generating singing, the method comprising: obtaining lyrics to be voiced and a required singing style, the lyrics to be voiced being obtained through a language model; determining a lyrics sound vector and a style sound vector through a first vector model based on the lyrics to be voiced and the required singing style, the first vector model being obtained through real vocal training; determining a background sound vector through a second vector model based on the required singing style, the second vector model being obtained through pure music training; converting the background sound vector into MIDI format to determine the MIDI background sound; determining the final background sound based on the MIDI background sound and a preset background audio library; merging the lyrics sound vector and the style sound vector, and converting them into MIDI format to determine the MIDI vocal; determining the final vocal based on the MIDI vocal and a preset vocal audio library; determining the singing based on the final background sound and the final vocal.

[0007] In one possible implementation, constructing the first vector model includes: acquiring singing data, the singing data including multiple real human voice singing dry sounds and original lyrics; preprocessing the singing dry sounds to determine preprocessed dry sounds; based on the preprocessed dry sounds, determining the MIDI dry sound file corresponding to the preprocessed dry sounds and the preprocessed dry sound style; creating a first mapping relationship between the MIDI dry sound file, the original lyrics and the preprocessed dry sound style; and training the first mapping relationship through a diffusion model to obtain the first vector model.

[0008] In a possible implementation, constructing the second vector model includes: acquiring a plurality of pure music data; determining preprocessed pure music by preprocessing the plurality of pure music data; determining a MIDI pure music file and the pure music style corresponding to the preprocessed pure music based on the preprocessed pure music; creating a second mapping relationship between the MIDI pure music file and the pure music style; and obtaining the second vector model by training the second mapping relationship through a diffusion model.

[0009] In a possible implementation, the method for constructing the preset vocal audio library includes: associating the preprocessed dry sound and the MIDI dry sound file to determine the vocal unit; and determining the preset vocal audio library based on the vocal unit and the preprocessed dry sound.

[0010] In a possible implementation, the method for constructing the preset background audio library includes: associating the preprocessed pure music and the MIDI pure music file to determine the background sound unit; and determining the preset background audio library based on the background sound unit and the preprocessed pure music.

[0011] In a possible implementation, after the final background sound is determined based on the MIDI background sound and the preset background audio library, the method further includes: based on the final background sound, reverse encoding into adjustment sound; accordingly, merging the lyrics sound vector and the style sound vector, and converting them into MIDI format, and determining the MIDI vocals, including: merging the adjustment sound, the lyrics sound vector and the style sound vector, and converting them into MIDI format, and determining the MIDI vocals.

[0012] In a possible implementation, the merging of the lyrics sound vector and the style sound vector, and converting them into MIDI format to determine the MIDI vocals, includes: merging the lyrics sound vector and the style sound vector through a preset influence coefficient to determine the merged sound vector, wherein the preset influence coefficient is used to characterize the proportion of the style sound vector in the merged sound vector; converting the merged sound vector into MIDI format to determine the MIDI vocals.

[0013] In a second aspect, an embodiment of the present application provides a device for generating singing voice, the device comprising: an acquisition module for acquiring lyrics to be voiced and a required singing style, the lyrics to be voiced being acquired through a language model; a determination module for determining a lyrics sound vector and a style sound vector through a first vector model based on the lyrics to be voiced and the required singing style, the first vector model being obtained through real vocal training; the determination module is also used to determine a background sound vector through a second vector model based on the required singing style, the second vector model being obtained through pure music training; a conversion module is used to convert the background sound vector into a MIDI format to determine the MIDI background sound; the determination module is also used to determine the final background sound based on the MIDI background sound and a preset background audio library; the conversion module is also used to merge the lyrics sound vector and the style sound vector, and convert them into a MIDI format to determine the MIDI vocal; the determination module is also used to determine the final vocal based on the MIDI vocal and a preset vocal audio library; the determination module is also used to determine the singing voice based on the final background sound and the final vocal.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or any one of the implementation methods thereof is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect or any one of the implementation methods thereof is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the method described in the first aspect or any one of the implementation methods thereof.

[0017] The solution provided by the present application is to obtain a lyrics sound vector and a style sound vector by inputting the lyrics to be voiced and the required singing style into a first vector model; to obtain a background sound vector by inputting the required singing style into a second vector model; to obtain a final human voice based on the lyrics sound vector and the style sound vector; to obtain the final background sound based on the background sound vector, and to determine the singing voice by combining the final human voice and the final background sound. Among them, the first vector model is obtained by training with real human voices, and the second vector model is obtained by training with pure music data. Compared with the state of mechanical pronunciation of the sound produced in the prior art, and the generated sound cannot be transformed into different styles according to user needs, the user of the present application can specify the singer's voice and retain the singing characteristics of the singer according to his own preferences and needs by providing the required singing style and the lyrics generated by the language model, thereby customizing the style of the music work with real human voice, and the automatic generation process of the present application greatly improves the efficiency of music creation and reduces the tediousness and time-consuming of manual creation. At the same time, users do not need to have professional music creation skills to obtain satisfactory music works through simple input. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a method for generating singing voice provided in one embodiment of the present application;

[0020] Figure 2 A structural block diagram of a device for generating singing voice provided in one embodiment of the present application;

[0021] Figure 3 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0022] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0023] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0024] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0025] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0026] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0028] Figure 1 A flow chart of a method for generating singing voice provided in an embodiment of the present application, the method comprising:

[0029] S110, obtaining lyrics to be voiced and required singing style, wherein the lyrics to be voiced are obtained through a language model.

[0030] Exemplarily, a large language model (LLM) of the generative pre-trained transformer (GPT) type that has been trained with a large database of lyrics is used to automatically generate lyrics S1 to be voiced according to user needs, and then the user provides the singing style S2 required for the lyrics S1 to be voiced, where the singing style S2 can be pop, rock, jazz, cheerful, sad, passionate, etc.

[0031] S120, based on the lyrics to be voiced and the required singing style, determine the lyrics sound vector and the style sound vector through a first vector model, and the first vector model is obtained through real human voice training.

[0032] Exemplarily, the lyrics S1 to be voiced and the required singing style S2 provided by the user are input into the first vector model m1, and the first vector model m1 generates the lyrics sound vector v1 and the style sound vector v1' in the Musical Instrument Digital Interface (MIDI) format according to the lyrics S1 to be voiced and the singing style S2 provided by the user. v1 mainly reflects the emotion and rhythm of the lyrics, while v1' reflects more style characteristics.

[0033] In one possible implementation, constructing a first vector model includes: acquiring singing data, the singing data including multiple real human voice singing dry sounds and original lyrics; preprocessing the singing dry sounds to determine preprocessed dry sounds; based on the preprocessed dry sounds, determining a MIDI dry sound file and a preprocessed dry sound style corresponding to the preprocessed dry sounds; creating a first mapping relationship among the MIDI dry sound file, the original lyrics and the preprocessed dry sound style; and training the first mapping relationship through a diffusion model to obtain a first vector model m1.

[0034] Specifically, the collected singing data includes the dry sounds (i.e., the vocal part excluding the accompaniment) of multiple specified singers and the original lyrics corresponding to the dry sounds. The collected singing dry sounds are subjected to a series of preprocessing, such as unifying the volume, removing noise, maintaining a consistent bit rate, etc., to ensure the consistency of the audio quality and the accuracy of subsequent processing, thereby determining the preprocessed dry sounds. The pitch information is extracted from the preprocessed dry sound data, and the pitch information is converted into the form of a MIDI file to obtain a MIDI dry sound file, in which each note is represented as a MIDI file. Then, based on the preprocessed dry sound, additional information tags are added, such as: dry sound style, so that the model can learn and imitate these specific style features.

[0035] Furthermore, a mapping relationship between the notes in the MIDI dry sound file and the original lyrics is established, so that the model can understand the correspondence between the lyrics and the tone. At the same time, the MIDI dry sound file is mapped to the pre-processed dry sound style to provide additional style information for the model. The two types of mappings have the same length vectors, so that the same model can be used for training. Then, through double-layer structure training, the embedded representation from lyrics to notes and from style to notes is obtained.

[0036] Furthermore, a diffusion model is used to train a first vector model m1 that can generate audio in the style of a specific singer. The first vector model m1 can infer the corresponding MIDI note sequence from the lyrics, and then generate audio with a specific style. In addition, the model can also support collective operations, that is, considering the style characteristics of multiple singers at the same time, and generating audio that combines multiple styles.

[0037] In order to make the results inferred by the first vector model m1 more accurate, all training singing data need to be additionally marked with descriptive words such as author, style, era, rhythm, etc. In this way, during inference, model m1 can infer the results more accurately according to the input requirements.

[0038] Through the above steps, the training process can construct a first vector model m1 that can generate audio with a specific singer's style based on lyrics and style.

[0039] S130, based on the required singing style, determine the background sound vector through a second vector model, and the second vector model is obtained through pure music training.

[0040] Exemplarily, the required singing style S2 is input into the second vector model m2 to determine the background sound vector, and the second vector model m2 is specifically used to generate the background sound vector v2 in MIDI format according to the style keyword. The background sound vector v2 will be completely based on the style features and will not be directly affected by the lyrics content.

[0041] In a possible implementation, constructing a second vector model includes: acquiring a plurality of pure music data; determining preprocessed pure music by preprocessing the plurality of pure music data; determining a MIDI pure music file and a pure music style corresponding to the preprocessed pure music based on the preprocessed pure music; creating a second mapping relationship between the MIDI pure music file and the pure music style; and obtaining a second vector model m2 by training the second mapping relationship through a diffusion model.

[0042] Exemplarily, multiple pure music data are obtained as the original data set. The collected pure music data is subjected to routine cleaning operations, including unifying the volume, removing noise, maintaining a consistent bit rate, etc., to determine the preprocessed pure music. Tone information is extracted from the preprocessed pure music data, and the tone information is converted into a MIDI pure music file. In a MIDI pure music file, each note is a representation of audio data. The MIDI pure music files are then labeled, and these labels are usually used to describe the pure music style, which will help the model learn the difference between different styles.

[0043] Furthermore, the MIDI music file and the music style are mapped, that is, a second mapping relationship between the style tag and the note sequence of the MIDI music file is established. Through the second mapping relationship, an embedded representation from the style to the note can be obtained.

[0044] Further, the diffusion model is used for training to obtain the second vector model m2. This model is an inference model from labels to MIDI note sequences, which can generate corresponding MIDI note sequences according to the input style labels.

[0045] During the training process, the second vector model m2 learns how to convert style labels into abstract representations of MIDI note sequences, and gradually generates MIDI sequences that are similar to real audio data.

[0046] S140, converting the background sound vector into MIDI format and determining the MIDI background sound.

[0047] Exemplarily, the background sound vector v2 is converted back to MIDI format to determine the MIDI background sound.

[0048] S150, determining a final background sound based on the MIDI background sound and the preset background audio library.

[0049] Exemplarily, the audio frequency and timbre corresponding to the background sound vector v2 are obtained based on the MIDI background sound and the preset background audio library d2, and finally the final background music o1 in WAV format is generated.

[0050] In one possible implementation, a method for constructing a preset background audio library includes: associating preprocessed pure music and MIDI pure music files to determine background sound units; and determining a preset background audio library based on the background sound units and the preprocessed pure music.

[0051] Exemplarily, the pre-processed pure music is associated with the notes in the corresponding MIDI pure music file, and equalization is performed to determine the background sound unit, wherein the equalization ensures the accurate correspondence between the pre-processed pure music and the MIDI pure music file, and eliminates possible differences or inconsistencies.

[0052] Furthermore, in order to increase the richness of background sound units and the generalization ability of the model, the preset background audio library d2 is finally determined by combining the audio features in the interval before and after the single tone in the pre-processed pure music (such as slur, transition, stop and other audio features). These additional audio features provide contextual information about how the notes are connected. The preset background audio library d2 contains the style tags of pure music audio and its corresponding MIDI pure music files.

[0053] As another example, based on the second vector model m2 and the preset background audio library d2, the method for determining the final background sound is:

[0054] The user provides an audio file o3 in WAV format, inputs o3 into the second vector model m2, and converts it into a MIDI sound vector v3;

[0055] Convert the MIDI sound vector v3 back to audio format, but do not output it directly at this time. Instead, obtain the original audio frequency similar to v3 based on the preset background audio library d2, and then generate a new WAV format audio file o4 based on these frequency information;

[0056] The generated audio o4 is reversely encoded into a MIDI sound vector v4. This step is to convert the audio signal back to the MIDI level for further processing;

[0057] Next, the two MIDI sound vectors v3 and v4 are superimposed based on the specified window to generate a new MIDI sound vector v5. This superposition process can be in many forms, such as 25% of v3, 75% of v3+v4 plus 25% of v3, 75% of v4, or a mixture of other proportions. This superposition allows the characteristics of the two audio sources to be fused and adjusted;

[0058] The superimposed MIDI sound vector v5 is converted back to MIDI format, and the corresponding original sound frequency is obtained again based on d2 to generate the final background sound o5 in the final WAV format.

[0059] As another example, another method for determining the final background sound based on the second vector model m2 and the preset background audio library d2 is:

[0060] The user provides a WAV format reference audio file o3 and keywords such as style s2. These keywords are used to describe the style characteristics of the background music that the user wants to generate, such as "jazz", "pop", "cheerful", etc.

[0061] The reference audio o3 is sent to the second vector model m2 for processing. The model recognizes the musical elements (such as notes, rhythms, chords, etc.) in the audio and converts them into a MIDI sound vector v3.

[0062] The singing style S2 is input into the second vector model m2 for processing, and the model generates a MIDI sound vector v2 that meets the user's expected style based on these keywords;

[0063] Next, v2 (style vector) and v3 (reference audio vector) are superimposed in a certain specified ratio to generate a new MIDI sound vector v4. This superposition process can be achieved through simple arithmetic operations or weighted superposition, which is not limited in this application, to ensure that v4 contains both the melodic characteristics of o3 and the style described by s2;

[0064] Convert the superimposed v4 back to MIDI format. In this process, it is necessary to obtain the original audio frequency based on the preset background audio library d2 to ensure that the generated audio is consistent or similar to the audio in d2 in timbre and frequency. Finally, the generated audio file is output as the final background sound o4 in WAV format.

[0065] S160, merging the lyrics sound vector and the style sound vector, converting them into MIDI format, and determining the MIDI vocals.

[0066] Exemplarily, the lyrics sound vector v1 and the style sound vector v1' are added together and then converted back to MIDI format to determine the MIDI vocals.

[0067] S170, determining a final vocal based on the MIDI vocal and the preset vocal audio library.

[0068] Exemplarily, based on the MIDI vocals and the preset vocal audio library d1, the original audio frequency and timbre corresponding to the fused MIDI vocals are determined, and the final vocals o2 in WAV format are generated.

[0069] In a possible implementation, a method for constructing a preset vocal audio library includes: associating preprocessed dry sound and MIDI dry sound files to determine vocal units; and determining a preset vocal audio library based on the vocal units and the preprocessed dry sound.

[0070] Exemplarily, for each note sequence in a MIDI dry sound file (such as CBEGD), find the audio clips on the corresponding timeline in the preprocessed dry sound and cut them accurately. Then associate these audio clips with the transitions between them (such as C to B, B to E) to form vocal units. Then, the same transitions (such as C to B) in the global scope are equalized to eliminate individual differences and obtain a more standardized audio representation. Finally, in order to improve the richness and generalization ability of the model, the audio features such as slur, transition, stop, etc. that may exist before and after each note of the preprocessed dry sound are also added to the vocal unit, and finally a preset vocal audio library d1 is formed.

[0071] S180, determining the singing voice based on the final background sound and the final vocals.

[0072] Exemplarily, the final background music o1 and the final vocals o2 are superimposed using audio editing software to generate a final audio work including the complete music style and vocals, namely, singing.

[0073] In another possible implementation, after the final background sound is determined based on the MIDI background sound and the preset background audio library, it also includes: based on the final background sound, reverse encoding into adjustment sound; accordingly, merging the lyrics sound vector and the style sound vector, and converting them into MIDI format, and determining the MIDI vocals, including: merging the adjustment sound, the lyrics sound vector and the style sound vector, and converting them into MIDI format, and determining the MIDI vocals.

[0074] Exemplarily, in order to increase the randomness and richness of the generated audio work, the final background sound o1 can be reverse-encoded into an adjustment sound v1" in MIDI format, and then the adjustment sound v1", lyrics sound vector v1 and style sound vector v1' are superimposed and converted into MIDI format to determine the MIDI vocals.

[0075] Furthermore, the singing voice is determined according to the determined MIDI vocals through steps S170 and S180. This can not only improve the randomness, but also improve the degree of integration between o1 and o2, making the final work more harmonious and unified.

[0076] In another possible implementation, the lyrics sound vector and the style sound vector are merged and converted into MIDI format to determine the MIDI vocals, including: merging the lyrics sound vector and the style sound vector through a preset influence coefficient to determine the merged sound vector, wherein the preset influence coefficient is used to characterize the proportion of the style sound vector in the merged sound vector; converting the merged sound vector into MIDI format to determine the MIDI vocals.

[0077] For example, when the lyrics sound vector v1 and the style sound vector v1' are merged, the influence coefficient of the style sound vector v1' can be adjusted, and the influence coefficient is used to characterize the proportion of the style sound vector in the combined sound vector (v1+v1'), and then the combined sound vector is converted into MIDI format to determine the MIDI vocals. Then, according to the determined MIDI vocals, the singing voice is determined through steps S170 and S180.

[0078] In another possible implementation, in addition to superimposing to generate a complete work, you can also choose to export the background music o1 and vocals o2 separately for further editing or separate use.

[0079] The technical solution provided by the present application is to obtain a preset vocal audio library d1 and a preset background audio library d2, as well as MIDI dry sound files and MIDI pure music files corresponding to multiple singing data and pure music files after processing multiple singing data and pure music files, and then train two types of diffusion models m1 and m2 based on the MIDI dry sound files and MIDI pure music files, and then generate a complete singing o2 and background sound o1 through the trained diffusion model in combination with the lyrics to be voiced and the required singing style provided by the user. Users of the present application can specify the singer and retain the singing characteristics of the singer according to their own preferences and needs by providing the required singing style and lyrics generated by the language model, so as to customize the style of music works with real human voices, and the automatic generation process of the present application greatly improves the efficiency of music creation and reduces the tediousness and time-consuming of manual creation. At the same time, users do not need to have professional music creation skills to obtain satisfactory music works through simple input.

[0080] Figure 2 This is a structural block diagram of a device for generating singing voice provided in an embodiment of the present application. For the sake of convenience, only the part related to the embodiment of the present application is shown. Figure 2 The device 200 for generating singing voice includes an acquisition module 201, a determination module 202, and a conversion module 203.

[0081] In one implementation, the apparatus 200 may be used to implement the above Figure 1 For example, the acquisition module 201 is used to implement S110, the determination module 202 is used to implement S120, S130, S150, S170 and S180, and the conversion module 203 is used to implement S140 and S160.

[0082] The technical solution provided by the present application is to obtain a preset vocal audio library d1 and a preset background audio library d2, as well as MIDI dry sound files and MIDI pure music files corresponding to multiple singing data and pure music files after processing multiple singing data and pure music files, and then train two types of diffusion models m1 and m2 based on the MIDI dry sound files and MIDI pure music files, and then generate a complete singing o2 and background sound o1 through the trained diffusion model in combination with the lyrics to be voiced and the required singing style provided by the user. Users of the present application can specify the singer and retain the singing characteristics of the singer according to their own preferences and needs by providing the required singing style and lyrics generated by the language model, so as to customize the style of music works with real human voices, and the automatic generation process of the present application greatly improves the efficiency of music creation and reduces the tediousness and time-consuming of manual creation. At the same time, users do not need to have professional music creation skills to obtain satisfactory music works through simple input.

[0083] Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: at least one processor 30 ( Figure 3 Only one is shown in the figure) a processor, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, and when the processor 30 executes the computer program 32, the steps in any of the above method embodiments are implemented.

[0084] The electronic device 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 3 and does not constitute a limitation on the electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0085] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0086] In some embodiments, the memory 31 may be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. In other embodiments, the memory 31 may also be an external storage device of the electronic device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the electronic device 3. The memory 31 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0087] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0088] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0089] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0090] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0091] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0092] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0093] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0094] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0096] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating singing voice, It is characterized in that The method comprises: Obtaining lyrics to be voiced and required singing style, wherein the lyrics to be voiced are obtained through a language model; Based on the lyrics to be voiced and the required singing style, determining the lyrics sound vector and the style sound vector through a first vector model, wherein the first vector model is obtained through real human voice training; Based on the required singing style, determining the background sound vector through a second vector model, where the second vector model is obtained through pure music training; Convert the background sound vector into MIDI format to determine the MIDI background sound; Determine the final background sound based on the MIDI background sound and the preset background audio library; The lyrics sound vector and the style sound vector are combined, converted into MIDI format, and the MIDI vocals are determined; Determine a final vocal based on the MIDI vocal and the preset vocal audio library; Based on the final background sound and the final vocal sound, the singing voice is determined.

2. The method according to claim 1, It is characterized in that Constructing the first vector model includes: Acquire singing data, wherein the singing data includes a plurality of real human voices singing dry sounds and original lyrics; The singing dry sound is subjected to preprocessing to determine the preprocessed dry sound; Based on the preprocessed dry sound, determining a MIDI dry sound file corresponding to the preprocessed dry sound and the preprocessed dry sound style; Creating a first mapping relationship between the MIDI dry sound file, the original lyrics and the pre-processed dry sound style; The first mapping relationship is trained through a diffusion model to obtain the first vector model.

3. The method according to claim 1, It is characterized in that Constructing the second vector model includes: Get multiple pure music data; Preprocessing the plurality of pure music data to determine preprocessed pure music; Based on the pre-processed instrumental music, determining a MIDI instrumental music file corresponding to the pre-processed instrumental music and the instrumental music style; Creating a second mapping relationship between the MIDI instrumental music file and the instrumental music style; The second mapping relationship is trained through a diffusion model to obtain the second vector model.

4. The method according to claim 2, It is characterized in that The method for constructing the preset human voice audio library comprises: Associating the preprocessed dry sound with the MIDI dry sound file to determine a vocal unit; The preset vocal audio library is determined based on the vocal unit and the pre-processed dry sound.

5. The method according to claim 3, It is characterized in that The method for constructing the preset background audio library comprises: Associating the pre-processed pure music with the MIDI pure music file to determine a background sound unit; Based on the background sound unit and the pre-processed pure music, the preset background audio library is determined.

6. The method according to claim 1, It is characterized in that After determining the final background sound based on the MIDI background sound and the preset background audio library, the method further includes: Based on the final background sound, reversely encode it into a modulation sound; Accordingly, the lyrics sound vector and the style sound vector are combined and converted into MIDI format to determine the MIDI vocals, including: The adjustment sound, the lyrics sound vector and the style sound vector are combined and converted into a MIDI format to determine the MIDI vocals.

7. The method according to claim 1, It is characterized in that The step of merging the lyrics sound vector and the style sound vector, converting the vector into a MIDI format, and determining the MIDI vocals includes: By using a preset influence coefficient, the lyrics sound vector and the style sound vector are combined to determine a combined sound vector, wherein the preset influence coefficient is used to characterize the proportion of the style sound vector in the combined sound vector; The combined sound vector is converted into a MIDI format to determine the MIDI vocal.

8. A device for generating singing voice, It is characterized in that The device comprises: An acquisition module, used to acquire lyrics to be voiced and required singing style, wherein the lyrics to be voiced are acquired through a language model; A determination module, configured to determine a lyrics sound vector and a style sound vector based on the lyrics to be voiced and the required singing style through a first vector model, wherein the first vector model is obtained through real human voice training; The determination module is further used to determine the background sound vector based on the required singing style through a second vector model, where the second vector model is obtained through pure music training; A conversion module, used for converting the background sound vector into a MIDI format to determine the MIDI background sound; The determining module is further used to determine the final background sound based on the MIDI background sound and the preset background audio library; The conversion module is further used to merge the lyrics sound vector and the style sound vector, convert them into MIDI format, and determine the MIDI vocals; The determination module is further used to determine the final vocal based on the MIDI vocal and the preset vocal audio library; The determination module is further used to determine the singing voice based on the final background sound and the final vocals.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images